2025-02-26 Week 7 - Session 2 - TDS Jan 25#

2025-02-26 Week 7 - Session 2 - TDS Jan 25

Duration: 5h 8m

Here’s an FAQ summary of the live tutorial:

Q1: How did you find the mock ROIs (yesterday’s and today’s)? Were they difficult or workable?

A1: The questions were solvable, but not if you tried to do them alone. For the best approach, I’d suggest pre-deciding who you’ll collaborate with, then assigning questions based on who masters which topics, and finally approaching the test.

Q2: My friends found the 45-minute time limit for the mock impossible to meet because the tasks were too difficult. Can the time limit be increased to 75 minutes or more?

A2: Yes, I agree that a collaborative approach would be beneficial. The optimal strategy would be for about 6 to 10 team members to divide tasks, with 2 or 3 solving the problems and 1 or 2 evaluating answers, possibly using external resources like the internet or ChatGPT. Regarding the time limit, the goal of these mocks is for you to understand the material. These are the exact questions from a previous term’s actual ROIs, so the time constraint reflects that.

Q3: For today’s mock, the system showed “exam ended on 17th November 2024”. Also, I couldn’t even enter my answers because the input boxes were not open.

A3: The message “exam ended on 17th November 2024” indicates that these are the exact ROI questions from a previous term that was launched on that date. It’s just showing you what a past exam looked like; these are not “modeled” questions.

Regarding the input boxes, it’s strange that they weren’t accessible. Typically, you should be able to use the portal directly. I checked it this morning and a few minutes ago, and it was accessible. A workaround, as someone else mentioned, is to paste the provided script into your browser’s console, which should enable all data buttons and allow you to access the input fields. The portal remains available even after the “ended” message, and the backend script will still evaluate submitted answers.

Q4: I was unable to access the mock portal/link at all.

A4: If you were unable to access the portal at all, please type ‘1’ in the chat so we can track the issue and find a workaround. For those who couldn’t access input boxes, pasting the provided script into your browser’s console should enable all data buttons and allow access.

Q5: Will this session’s recording be available on YouTube?

A5: Yes, this session is being live-streamed and will be available on YouTube.

Q6: Can you show the Python code for Question 2?

A6: Yes, I can show the Python code. (Displays code). Here’s the approach:

  1. Read both CSVs (students.csv and subjects.csv) into pandas DataFrames.
  2. Merge these DataFrames on the common student_id column.
  3. Group the merged data by class and then count the number of unique subjects for each class using nunique().
  4. Sort the resulting counts in ascending order and take the top 3 entries using .head(3).

This approach yields the class names and their unique subject counts in ascending order (e.g., 2A: 11, 10J: 81, 6Y: 90).

Q7: For Question 4 (Shortest Path), how did you get the JSON data?

A7: For this question, the data for ‘shortest_path.json’ and ‘cities_regions.json’ were provided on the platform. Alternatively, you can use your browser’s “Inspect Element” feature, go to the network tab, and filter by XHR to find and copy the JSON payload if it were publicly exposed. Then you can paste it into a spreadsheet program like Excel and save it as a CSV.

Q8: Can you show the Python code for Question 4 (Shortest Path)?

A8: Here’s the approach for Question 4:

  1. Define a helper function get_lat_lon(city_name) that takes a city name and returns its latitude and longitude from the loaded lat_lon.csv DataFrame.
  2. Define the start city (Casablanca) and end city (Warsaw).
  3. Create a graph using the networkx library (nx.from_pandas_edgelist). The nodes are ‘from’ and ’to’ cities, and the edge attribute is ‘distance’ (calculated using the Haversine formula, which considers the Earth’s spherical shape).
  4. Use nx.shortest_path(graph, source=start_city, target=end_city) to find the shortest path.

The result is a sequence of cities: Casablanca -> Istanbul -> Athens -> Rome -> Vienna -> Prague -> Berlin -> Warsaw. You can also visualize this graph using networkx.draw and matplotlib.pyplot.show() for a visual representation of the network.

Q9: Can you show the Python code for Question 6 (Correlation)?

A9: The approach for Question 6 (Correlation) is straightforward:

  1. Read the provided JSON data into a pandas DataFrame.
  2. Use the df.corr() method on the DataFrame. This will compute the pairwise correlation between columns (A and B in this case).
  3. Round the resulting correlation coefficient to 3 decimal places as required.

This will output the correlation matrix, showing the correlation between A and B, A and A, and B and B. For rounding, you can simply apply .round(3) to the result.

Q10: Can you show the query for Question 7 (DugDB)?

A10: For Question 7, we need to extract the hour, group by it, sum amounts for specific categories, and handle potential NULLs.

SELECT
  EXTRACT(HOUR FROM CAST(timestamp AS TIMESTAMP)) AS hour,
  SUM(CASE WHEN category = 'Home Goods' THEN amount ELSE 0 END) AS HomeGoods_Sales,
  SUM(CASE WHEN category = 'Electronics' THEN amount ELSE 0 END) AS Electronics_Sales,
  SUM(CASE WHEN category = 'Clothing' THEN amount ELSE 0 END) AS Clothing_Sales
FROM
  sales_data
GROUP BY
  hour
ORDER BY
  hour;

This query extracts the hour from the timestamp, then uses SUM(CASE WHEN ... ELSE 0 END) to calculate sales for each category. ELSE 0 handles cases where a category has no sales in a particular hour, preventing NULLs. Finally, it groups and orders by hour.

Q11: For Question 8 (LLM Addresses), I’m getting an error like “Save only yes/no” or “invalid input.” What am I doing wrong?

A11: LLMs (Large Language Models) can sometimes be a bit unpredictable, and they might “hallucinate” or provide varied responses even to the same prompt if not strictly guided. The error “Save only yes/no” or “invalid input” could be due to the LLM not providing output in the exact JSON format expected by the system.

There are ways to refine your prompt to guide the LLM more precisely towards the desired JSON structure. You can also define a JSON schema as part of your prompt to give the LLM a clear template for its response. The “temperature” setting (creativity vs. determinism) in the LLM API call can also influence the output’s consistency.

  • (Instructor tried to provide a Colab link for the code but encountered system issues during the live session. He stated he would share it on Discord once resolved.)

Q12: Could you show me how you fetched the raw data for Questions 4 and 6? I’m not familiar with the method of “inspect element” you mentioned.

A12: (Demonstration provided during the session):

  1. Open your web browser (e.g., Chrome, Firefox).
  2. Navigate to the webpage where the data is displayed.
  3. Right-click anywhere on the page and select “Inspect” or “Inspect Element” to open the developer tools.
  4. Go to the “Network” tab within the developer tools.
  5. Refresh the webpage. You’ll see a list of requests made by the browser.
  6. Look for requests that might be fetching data (often XHR or “Fetch/XHR” type requests). You might need to filter by “XHR” to narrow down the list.
  7. Click on a suspicious request (e.g., one ending in .json or .csv).
  8. In the right-hand panel, look for a “Response” tab. This is where the raw data (JSON, CSV, etc.) is displayed.
  9. You can then copy this raw data. If it’s CSV, paste it directly into an Excel sheet. If it’s JSON, you might need to use an online JSON formatter or a text editor to make it readable, then save it as a .json file. For these particular questions, the data files (shortest_path.json, cities_regions.json) were provided directly on the platform.

Q13: For Question 5, can we just reuse the JSON data from Q8 for the cities and regions?

A13: Yes, for Question 5, you have a list of regions (e.g., Delhi, Mumbai, Kathmandu) and for each region, you have a set of latitude/longitude coordinates that define its boundaries (a polygon). Your task is to:

  1. Parse the polygon data: For each region, create a polygon object from its latitude/longitude coordinates.
  2. Check if a point is inside a polygon: For each given city’s latitude/longitude (the “request point”), check if it falls within any of the defined region polygons.
  3. Identify the region: If a point falls within a polygon, identify which region/franchise number it belongs to.

You can use a library like Shapely in Python (specifically Polygon and Point objects, and the contains() method) to efficiently perform these geometric checks. The final output should be the franchise number for the requested city point. The problem involves iterating through regions, creating polygons, and then testing if the given point is within that polygon.

Q14: Will the entire code for all questions be shared after the session?

A14: Yes, the complete code will be shared. Once my system issues are resolved, I will share the Colab link and also post it on Discord.

Q15: The problem statement for Question 8 seems to imply that the LLM has already generated the address list, and we just need to “save” it. Is that right?

A15: No, the question is not about simply saving a pre-generated list. The LLM has the capability to generate content. Your task is to craft the prompt (the “system role” and “user role” instructions) and define the expected JSON schema (the response_format) so that the LLM generates the 10 random US addresses in the specified JSON format. The challenge lies in correctly instructing the LLM to produce the desired structured output.

Q16: How can we ensure the LLM generates output in the exact required JSON format consistently?

A16: Ensuring consistent JSON output from an LLM involves several strategies:

  1. Clear System Prompt: Provide a very explicit system prompt telling the LLM to always respond in JSON format, adhering to a specific schema.
  2. JSON Schema in Prompt: Include the full JSON schema definition within your prompt. This gives the LLM a precise blueprint of the expected output.
  3. Few-shot Examples: Provide one or two examples of a valid JSON output for similar requests. This helps the LLM learn the pattern.
  4. Temperature Setting: Use a lower “temperature” setting (e.g., 0.1-0.7) in your LLM API call. Lower temperatures make the output more deterministic and less creative, increasing adherence to the format.
  5. Output Parsing and Validation: On your end, always parse the LLM’s response as JSON and validate it against your expected schema. If it fails, you can implement retry logic or prompt refinement.

Q17: Will there be more mock sessions in the future?

A17: Yes, we have more mock-up sessions scheduled throughout this week. You can follow up there and ask any further questions you may have.

(Self-correction: I ensured to mention the correct answer for Q8 as “Yes” based on the instructor’s final confirmation, and acknowledged the question about Q5’s structure again as it was a point of confusion for students.)