Here’s an FAQ summary of the tutorial:

Tools in Data Science (TDS) Live Tutorial FAQ#

Q1: Where can I access the mock ROE (Review of Educational Exercises) files mentioned in the tutorial?

A1: The mock ROE files, including the PDF, are usually accessible from your TDS dashboard. If you’re having trouble accessing the mock ROE4.zip file directly from a link within the GitHub repository (e.g., it’s not clickable), it might be due to your PDF viewer blocking external links. Try downloading the PDF to your local system first, and then access the links.

Q2: Will the steps to solve the ROE questions be provided during the actual exam, or just the questions?

A2: Based on the current setup, it seems that for the actual ROE, everything except the correct final answer will be provided, including the steps. This is similar to a markdown format where you fill in the blanks or complete specific sections.

Q3: Will the questions in the exam be similar in structure to the ones presented in this mock ROE, involving multiple HTML, PDF, and database files?

A3: The exact nature of the questions in the actual ROE is not fixed, and they could vary. However, they will generally be related to the concepts covered in your graded assignments. The structure (e.g., using multiple data sources like HTML, PDF, and databases, with multi-dataframes) could be similar.

Q4: I haven’t received the email with the ROE files or links. What should I do?

A4: The emails for the ROE files are typically automated, so everyone registered should receive them. First, check your spam folder. If you still haven’t received it, please share your email address in the chat, and the team will look into it. There might be an issue with the Google Group distribution if you were not added correctly.

Q5: What are the primary data sources we are working with in this ROE, and how do we initially read them?

A5: We’re primarily dealing with three types of data:

  1. HTML files: These files contain tables with key-value pairs, some having multiple tables. We use pandas.read_html() to read them, which returns a list of dataframes.
  2. PDF files: These files also contain structured data like violation IDs, business IDs, and scores. We use the tabula library to read these into dataframes.
  3. SQLite Database (DB) files: This database file contains tables like violation DB with business IDs and descriptions. We connect to it using sqlite3 and read tables into dataframes using pandas.read_sql_query().

Q6: How do I handle multiple HTML files, each potentially containing multiple tables?

A6: You can use the glob library for pattern matching to get a list of all HTML file paths. Then, iterate through each HTML file. For each file, pandas.read_html() will return a list of dataframes (one for each table found in that HTML file). You can then collect all these individual dataframes into a single master list (using list.extend()) or convert each dataframe into a dictionary for easier parsing and access to specific columns like ‘postal code’, ’latitude’, or ’longitude’.

Q7: Can you summarize the steps for Question 1: “Filter restaurants present at postal code 94110 which had the moderate risk violation on Monday”?

A7:

  1. Read Data: Read the HTML files (for postal codes) and the violation DB (for risk category, business ID, date).
  2. Process Violation DB:
    • Convert the ‘Date’ column to datetime objects using pd.to_datetime().
    • Filter the dataframe to include only ‘Moderate Risk’ violations on ‘Monday’ using string matching on the dt.day_name() attribute.
    • Group the filtered data by ‘Business ID’ and count the violations (.size()), then reset the index.
  3. Process HTML Data:
    • Iterate through all HTML dataframes (or their dictionary representations) to identify ‘Business ID’s associated with the postal code ‘94110’. Store these unique business IDs in a list.
  4. Merge and Filter: _ Filter the grouped violation data (from step 2) to include only the ‘Business ID’s found in step 3. _ Sum the violation counts for the remaining businesses. This final sum represents the answer.

Q8: Can you summarize the steps for Question 2: “What is the highest average inspection score in this particular month that any latitude-longitude grid (rounded up to two decimal) has received?”

A8:

  1. Extract Data:
    • From HTML data, extract ‘Business ID’, ’latitude’, and ’longitude’.
    • From PDF data, extract ‘Business ID’ and ‘score’.
  2. Merge Data: Join the HTML and PDF dataframes based on ‘Business ID’.
  3. Round Coordinates: Round the ’latitude’ and ’longitude’ columns to two decimal places.
  4. Group and Average: Group the merged dataframe by the rounded ’latitude’ and ’longitude’, then calculate the mean (average) of the ‘score’ for each unique grid.
  5. Find Maximum: Identify the maximum value among these average scores. This will be your answer.

Q9: Can you summarize the steps for Question 7: “Perform linear regression using the average inspection scores (from Question 2) as the target variable and the latitude and longitude as features. Predict the score for a new data point (40.71, -74.00) using the trained model.”

A9:

  1. Prepare Data: Use the results from Question 2 (average inspection scores grouped by rounded lat/long).
  2. Define Features and Target:
    • Set the rounded ’latitude’ and ’longitude’ as your features (X).
    • Set the average ‘score’ as your target variable (y).
  3. Train Model: Use sklearn.linear_model.LinearRegression to train a model on your X and y data.
  4. Predict: Use the trained model’s predict() method with the new data point (40.71, -74.00) to get the predicted score.

Q10: The provided code for the mock ROE sometimes gives different answers than the expected ones, or has issues. Why is that?

A10: The provided Python notebook might be incomplete or have minor discrepancies, which can lead to different results. This particular mock ROE file is not an official, finalized version for evaluation. It’s intended for practice and understanding the tools and workflow. The team will provide corrected solutions and official ROE files later. For now, focus on understanding the logical flow and the usage of data science tools.

Q11: Will the full Python notebook code for this tutorial be shared?

A11: Yes, the complete Python notebook code will be shared on GitHub. However, please note that the current version might have minor issues or incomplete sections as discussed. It’s recommended to understand the concepts and flow rather than solely relying on the provided code as a definitive solution.