TA’s Session TDS 2024 11 14 19 54#

TA’s Session TDS 2024 11 14 19 54

Duration: 1h 52m

Here’s an FAQ-style summary of the live tutorial:

Q1: What will the ROE session cover, and when is it? A1: The ROE session on Thursday, November 14th, will cover skills required for ROE. ROE will comprise content from weeks 1-6. The ROE itself will be on a Sunday, from 1:00 PM (though the exact duration isn’t fixed yet). You’ll receive a link, so there’s no need to log in precisely at 1:00 PM.

Q2: When will Project 1 results be released? A2: Anand needs to push the results to the operations team before they can be released. While TAs can view your scores, we don’t have the authority to push the results ourselves. Once Anand certifies the results, they will be released.

Q3: Is ROE similar to OPP? A3: No, ROE is not the same as OPP. ROE is completely open-book, meaning there’s no session recording, video setup, camera capture, or screen capture. You can even collaborate with others during the ROE, except for consulting the TAs.

Q4: Will there be support available during the ROE for technical difficulties? A4: Yes, typically there’s a GMeet available where you can log in if you face operational or technical issues. The operations team, who will also be on the GMeet, can provide instant support. Additionally, a specific Discourse thread might be opened for ROE-related questions, which will be locked once the ROE is over. While we won’t give you direct answers, you can discuss and collaborate with other participants, and you’re free to use any tools like GPT for assistance.

Q5: What are the key SQL skills I need for ROE? A5: You need a good understanding of SQL basics, including how to construct and debug queries. This is more important than knowing a specific database type. Practice common operations like summing, counting, selecting, grouping data (using GROUP BY), and applying conditions (using HAVING). Familiarize yourself with simple JOINs. The SQL Zoo website is a great resource for practice. While GPT can help construct queries, you must understand SQL well enough to debug its output, as incorrect answers from GPT won’t be identifiable if you lack fundamental SQL knowledge.

Q6: Will I need to create tables or just write queries in SQL? A6: You’ll primarily need to write queries. There’s no expectation for you to create tables. Some questions might require you to connect to a database (either provided as a file or via a web connection). If connecting via the web, ensure you know how to create connection strings.

Q7: Will there be a “check answers” button in ROE? A7: Yes, there will be a “check answers” button, so you’ll know if your answer is correct or incorrect. However, there will be no multiple-choice questions; you’ll need to type or copy-paste your answers.

Q8: How many questions will there be in ROE, and what is a reasonable score? A8: The number of questions for ROE is not yet fixed; it could be around 10 or more. Anand, the instructor, makes the final decision on this. Historically, the median score for ROE is around 50%.

Q9: Why are some TA session recordings delayed, and are live TA sessions important? A9: Recordings are sent to the operations team for uploading to YouTube. However, TAs don’t have admin access to directly upload them due to course team policies. Recent delays are partly due to the unavailability of Prasanna, who usually handles uploads. Live TA sessions are highly beneficial because attendees receive first-hand information and insights into what to expect in the ROE.

Q10: How many datasets should I expect in ROE? A10: The number of datasets varies with each ROE. It could range from two to six different datasets, or even just one dataset per question. Be prepared for any number.

Q11: What is the recommended strategy for preparing for ROE? A11: Create two separate notebooks: a “reference” notebook for code snippets and concepts, and a “solving” notebook for actual ROE questions. Pre-install all necessary libraries (Pandas, PyArrow, SQLAlchemy, etc.) in your environment beforehand to save precious time during the ROE. Every second counts, as there won’t be enough time. This setup allows you to quickly refer to your snippets and focus solely on solving the problems.

Q12: What are the key skills related to API calls and JSON? A12: API calls are crucial and will almost certainly be part of the ROE.

  • Familiarize yourself with the requests library: how to make GET and POST requests, handle HTTP headers, and interpret status codes (e.g., 400 for bad request, 401 for unauthorized, 404 for not found). Understanding these helps quick debugging.
  • Learn to send JSON objects (which are key-value pairs) and remember that JSON keys and string values must be enclosed in double quotes.
  • Know how to use Python’s json library, specifically json.loads(), to convert JSON responses into Python dictionaries for easier parsing.

Q13: When is Project 2 scheduled? A13: Project 2 has been moved to December 2nd. An official announcement with all details can be found on Discord.

Q14: Will I need to scrape web pages or clean data in ROE? A14: Yes, you’ll need to be able to extract information from live web pages. Familiarize yourself with browser inspect tools and the console, focusing on CSS selectors. Knowing how to manipulate page elements directly can be very valuable. For cleaning messy data, like misspellings, OpenRefine is a useful tool to learn. The tabula library can handle simple PDFs if you need to extract data, but complex PDF parsing is unlikely. The goal is to clean unclean data, not deal with complex parsing issues.

Q15: What statistical analysis skills are expected? A15: For Week 4, basic statistical analysis is expected. This includes calculating correlations, means, medians, quartiles, and performing simple regressions. Pandas functions are suitable for this.

Q16: Will DuckDB be part of the ROE? A16: Although there hasn’t been a GA session on DuckDB, it was introduced this term, and I have a strong suspicion there will be a question based on it. Watch the DuckDB video and learn how to write SQL queries for it. I might also hold a short session on DuckDB soon.

Q17: What about LLM/API usage in ROE? A17: You are 100% guaranteed to use an LLM API call in ROE, specifically using the AI proxy to access model, embeddings, or completions endpoints. Create a code snippet for this beforehand. A key skill will be structured extraction of information from LLMs, which is a highly probable question type.

Q18: What geospatial skills do I need? A18: You should understand how latitude and longitude work and be able to calculate distances between two points on a map. This includes using the Haversine formula (for large distances, accounting for Earth’s curvature) and Euclidean distance (for smaller, local areas). Folium and QGIS are unlikely to be required.

Q19: What is the recommended workflow for ROE? A19: The workflow is: Extract -> Clean -> Transform -> Answer.

  1. Extraction: Be proficient in getting data from various sources (HTML, JavaScript, APIs using requests library, SQL databases).
  2. Cleaning: Master cleaning techniques, including handling null values, dealing with messy data using tools like OpenRefine, and interpolating values.
  3. Transformation: Know how to convert data from one format to another, aggregate information, and prepare it for analysis using Pandas.
  4. Answering: Be able to use statistical methods and other tools to derive insights and answers from your transformed data.

Q20: What’s the plan for tomorrow’s session? A20: Tomorrow’s session will be a more technical, hands-on session. We’ll give a rapid overview of HTML, CSS, JavaScript (including extracting objects from browsers), SQL connections, and LLM API calls. We’ll then construct a notebook with essential code snippets that you can use as a template for your own ROE preparation notebook.

Q21: Can you provide a summary of tomorrow’s session’s agenda? A21: Tomorrow’s agenda includes:

  • Brief overview of HTML, CSS (selectors), JavaScript (object extraction from browser console).
  • SQL connections and essential queries.
  • LLM API calls.
  • Constructing a preparatory notebook with relevant code snippets for these topics.

Q22: Is it worth trying to “hack” the ROE to find answers? A22: While it’s possible to examine the JavaScript code to find how answers are validated (as demonstrated in Project 1), be very careful. This was a deliberate “Easter egg” left by Anand to see if students would try. In ROE, it might not be so straightforward; there could be obfuscation (deliberately unclear code) or red herrings (misleading clues) to make such attempts difficult and time-consuming. My advice is to focus on mastering the fundamental skills and workflow rather than trying to hack the system, as precious time will be better spent solving problems directly.

Q23: Will there be JavaScript questions in ROE? A23: Yes, it is almost certain that you will need to use JavaScript for extraction, particularly to get objects from web pages. This will not be about hacking, but about using JavaScript skills for legitimate data extraction.

Q24: What time will tomorrow’s session be held? A24: I will confirm the exact time after speaking with the team, but it’s likely to be around 5:00 PM to 7:00 PM IST. It will be recorded.

Q25: Will you release the recordings of these sessions soon? A25: I will personally speak with the operations team tomorrow and emphasize the need to push the recordings (both today’s and previous ones) as early as possible.