TA’s Session TDS – 2024 10 13 19 49 IST – Recording#

Duration: 2h 11m

Here’s an FAQ summary of the live tutorial:

Q1: What’s the plan for today’s session and future topics?

A1: Today, we’ll go through GA1, most of Week 2, and cover GA2. Many of your questions about tool usage will be answered. We won’t cover web scraping extensively today, but there are videos available that cover it in depth. For upcoming topics like ROE (Return on Investment), we’ll have extra sessions, some led by me and some by other instructors.

Q2: Where can I find bonus questions and how do I get notified?

A2: Bonus questions are posted on the Tools for Data Science Discord channel. To receive them as soon as they’re released, I recommend subscribing to the relevant Discord posts. They’ll come straight to your inbox. The previous bonus question saw about 5% of registered TDS students attempt it, which the course creator, Anand, was pleased with.

Q3: What’s the deadline for bonus questions?

A3: Bonus questions are typically pinned on Discord and remain active for a couple of weeks. The deadline is Thursday, October 17th. “AOE” means “Any Time On Earth,” so even if it’s 23:59 on October 16th in your time zone, it will still be accepted. This means the deadline is effectively 23:59:59 on October 16th (before the 17th starts) anywhere on Earth.

Q4: What was the purpose of the first GA1 question, which asked to replace text in multiple files, and how can I do it efficiently?

A4: The main purpose was to familiarize you with using the terminal for performing operations, even complex ones like finding multiple files and executing commands on them. You could do it manually by opening each file, but for efficiency, a small Bash script using the find and sed commands is much better.

Q5: Can you explain the find and sed commands used to replace text?

A5:

The find . -type f command searches the current directory (.) for files (-type f).
The -exec option tells the shell to execute a command on each file found.
The sed command (Stream EDitor) is used for replacing text within files. The format is sed 's/pattern/replacement/flags'.
- s indicates substitution.
- pattern is the text to find (e.g., IITM).
- replacement is the text to insert (e.g., IIT Madras).
- g flag means “global,” replacing all occurrences within a line, not just the first.
- i flag means “case-insensitive,” ignoring the case of the text being searched.
The -i option for sed (e.g., sed -i '') performs the edit directly “in-place” on the file. For some shells like ZSH, an empty string ('') is needed after -i.
The {} symbol in the find -exec command is a placeholder where the name of each found file is inserted.

Q6: What is the purpose of a regular expression (regex) and why is it important for data science?

A6: Regular expressions are powerful patterns used to search for and manipulate text. The pattern used in sed (e.g., /IITM/) is a regex. Understanding regex is crucial, especially when working with AI tools like ChatGPT, as they often generate regex. If you can’t read or debug regex, you won’t be able to fix issues in your code.

Q7: Where can I learn more about regular expressions?

A7: The System Commands module (part of the BS program) has two dedicated lectures on regular expressions in Week 4 content. I highly recommend watching those. Additionally, I will host a separate session later in the course to walk through basic regex concepts and how they are constructed, but not today.

Q8: What is a hash code and how does it relate to the GA1 exercise?

A8: A hash code (like SHA-256) is a unique, fixed-size string that represents the content of a file or set of files. If even a single character is changed in any of the files, the hash code will completely change. In GA1, generating a hash code for all your changed files allows you to verify if your modifications were successful and if your answer is correct. You’ll also learn more about hash codes in MAD1.

Q9: What was the purpose of the multi-cursor editing question (GA1 Q2), and how can I do it in VS Code?

A9: This question aimed to introduce you to the powerful concept of multi-cursor editing, which is incredibly useful for applying the same change to multiple lines or locations simultaneously. In VS Code:

To create multiple cursors vertically, use Alt + Down Arrow (Windows) or Cmd + Alt + Down Arrow (Mac).
To select all occurrences of a highlighted string (and then edit them simultaneously), use Ctrl + Shift + L (Windows) or Cmd + Shift + L (Mac). This technique is extremely valuable for quick fixes, especially in time-sensitive scenarios like ROE.

Q10: How do I resolve the image loading error in GA1 Q4 (Google Colab)?

A10: The error occurs because the Python script expects a single image file directly, but it might be receiving a list containing the image.

First, ensure the image (test_img.jpeg) is uploaded to your Google Colab environment. You can do this by starting the runtime (connecting to a server), then using the file browser (folder icon on the left) to upload the file.
Next, modify the Python code to load the image directly, removing any list wrappers. For example, if it was data = [image], change it to data = image. This will provide the script with the single image it expects.

Q11: What was the purpose of the GA1 Q4 error resolution question?

A11: This exercise was designed to test your basic debugging skills within the Google Colab environment and to ensure you understand how to upload and manage files in Colab.

Q12: What was the purpose of the GA1 Q5 question (accessing specific dataset elements), and how can I solve it without complex scripting?

A12: This question teaches you that sometimes simple tools are more efficient than complex code. You can:

Run ls -l in the terminal to get file metadata (including size and date).
Copy this output and paste it into Google Sheets.
Use the “Text to Columns” feature in Google Sheets to parse the data into structured columns.
Apply filters (e.g., “greater than or equal to 1154 bytes”, “on or after 25th September 2002”). Be careful to correctly interpret ambiguous date formats (e.g., “23:59” might refer to a time in the current year, not an actual year, so convert it if necessary).
Finally, sum the filtered ‘size’ column. This approach is often quicker and avoids complex regex.

Q13: How can I compare files and find the number of differing lines (GA1 Q7)?

A13: You can use the comm command in the terminal. It compares two sorted files and outputs three columns: lines unique to file1, lines unique to file2, and lines common to both. To find the differing lines, you would count the lines in the first two columns of comm’s output.

Q14: What is the importance of character encodings like UTF-8 or UTF-16 when working with text files?

A14: Character encoding determines how characters are represented in a file. UTF-8 is a common and widely compatible encoding. However, if a file uses a different encoding (like UTF-16), you need to specify this when reading or processing the file to avoid errors. There’s a video by Anand Sir dedicated to explaining this.

Q15: How do I use the Pokedex API to retrieve Pokémon data (GA1 Q8)?

A15: This involves making HTTP requests and parsing JSON data:

Use the requests library in Python. requests.get() is used to send a GET request to the API endpoint (e.g., pokeapi.co).
The API returns a JSON object. Python automatically converts this into a dictionary or list, making it easy to navigate.
You then traverse the nested dictionary/list structure of the JSON response to extract specific information, such as Pokémon moves, versions, or other attributes.
The instructor demonstrated a function that iterates through all available moves for a Pokémon and filters them by a specific game version (e.g., “x-y”) to find common moves.

Q16: What’s a common pitfall when using APIs, and how can I avoid it?

A16: A common mistake is making too many unnecessary API calls, especially when debugging. Many APIs (like the Pokedex API) have rate limits. Repeatedly hitting the API for static information (e.g., debugging other parts of your code) can lead to your IP being banned.

Best practice: When debugging, hit the API once, store the response (e.g., as a variable), and then work with the stored data. Only make a new API call when you genuinely need updated information. This shows respect for the API provider’s resources.

Q17: Why is learning JavaScript important for web scraping, and how can I learn about it?

A17: JavaScript is essential for scraping dynamic websites. Many modern websites use JavaScript to load content dynamically after the initial page load. A standard requests call won’t execute JavaScript, so you’ll miss this content.

JavaScript allows you to interact with the browser’s Document Object Model (DOM), which represents the fully rendered page content. This enables you to extract data that isn’t present in the initial HTML.
The “Scraping with JavaScript” video under “Optional Live Recordings” is highly recommended. It demonstrates using JavaScript in the browser console to access and extract data from the DOM.

Q18: What are common mistakes when using the Nominatim API (GA1 Q8)?

A18: The Nominatim API (for geographical data) can be tricky:

User-Agent: Not using a proper User-Agent string in your request. You should identify your application.
Ambiguous Names: Locations can share names. A query like “Chongching, China” might return multiple results (e.g., a district instead of the city). You need to be specific (e.g., “Chongching City, China”) to get the desired result.
Filtering: Nominatim often returns various types of entities (cities, parks, buildings). You’ll need to filter the JSON response to find the specific type of entity you’re interested in (e.g., type: "city").

Q19: How do I access the Dev Tools command palette and disable JavaScript temporarily for a page?

A19:

Open the Developer Tools in your browser (usually by pressing F12 or Ctrl+Shift+I / Cmd+Option+I).
Press Ctrl+Shift+P (Windows) or Cmd+Shift+P (Mac) to open the Command Palette.
Type “disable JavaScript” and select the option. This temporarily disables JavaScript for the current tab, which can be useful for debugging and seeing the initial HTML of a dynamic page. Remember to re-enable it or the page might not function as expected (e.g., your microphone might not work if it relies on JavaScript).

Q20: What’s the main point about using CSS selectors for web scraping (GA2 Q2)?

A20: CSS selectors are a fundamental skill for web scraping. They allow you to precisely target specific elements within an HTML document. I will cover CSS selectors in detail in the next session, including combinators and understanding element hierarchy. The Beautiful Soup documentation also has excellent resources on this topic.

Q21: How do I extract data from PDFs (GA2 Q3)?

A21: The tabula-py library is designed for this:

You can use tabula.read_pdf() to directly extract tables from a PDF into a Pandas DataFrame.
Alternatively, tabula.convert_into() allows you to convert PDF tables into other formats like CSV or JSON.
Common mistake: If your PDF has multiple tables on a page (e.g., a header table and a data table), use the multiple_tables=True option in read_pdf to ensure all tables are extracted correctly. Otherwise, data might be wrongly aggregated.

Q22: How can I perform filtering in Google Sheets (GA2 Q4) correctly for filtered data?

A22: This question revisits filtering skills. When you filter data in Google Sheets, the standard SUM() function will sum all rows, including those hidden by the filter.

To sum only the visible (filtered) rows, use the SUBTOTAL() function. Specifically, SUBTOTAL(9, range) will calculate the sum of visible cells in the specified range (where 9 is the function code for SUM). This prevents incorrect calculations based on hidden data.

Q23: What are the key concepts related to K-Means Clustering (GA2 Q5 & Q6) and the Elbow Method?

A23: K-Means is an unsupervised learning algorithm for clustering data points. The instructor will cover:

Initialization: How to choose the initial cluster centroids.
Assignment: How each data point is assigned to the nearest centroid.
Update: How centroids are recalculated based on their assigned data points.
Convergence: When the algorithm stops iterating (centroids no longer move significantly).
Elbow Method: A heuristic technique used to find the optimal number of clusters (k) by plotting the within-cluster sum of squares against the number of clusters and looking for an “elbow” point.

Q24: What is DBSCAN Clustering (GA2 Q7)?

A24: DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is another clustering algorithm, distinct from K-Means. It focuses on identifying clusters based on the density of data points. The instructor will cover this in the next session.

Q25: What is Principal Component Analysis (PCA) and dimensionality reduction (GA2 Q8 & Q9)?

A25: PCA is a widely used dimensionality reduction technique. Dimensionality reduction aims to reduce the number of features (variables) in a dataset while retaining as much essential information as possible. PCA achieves this by transforming the data into a new set of uncorrelated variables called principal components. The instructor will explain PCA and general dimensionality reduction concepts in the next session.

Q26: What are the common mistakes when using the requests library for APIs (GA1 Q8)?

A26: A common mistake is not correctly encoding URL parameters or handling the API response.

requests.get(url, params=payload) is the recommended way to pass parameters, as requests handles URL encoding automatically.
The API response comes as a JSON object, which can be accessed as a Python dictionary using .json().
The actual answer to the question was found by inspecting the API response: the result you’re looking for is within the moves key, then iterate through version_group_details to find the correct version_group and extract the move name.

Q27: What was the point of the Nominatim API exercise (GA2 Q1)?

A27: The purpose of this exercise was to show that while you can use an API to get information (like an OSM ID for a location), the API response often contains much more detail than you need. You’ll have to parse the JSON response to extract the specific piece of information required. The OSM ID was nested within the JSON structure, under specific keys.

Q28: What was the point of the Pokedex API exercise (GA2 Q2)?

A28: This exercise, involving the Pokedex API, demonstrated how to interact with an API to retrieve structured data. The key points were:

requests library: Essential for making HTTP requests.
JSON parsing: Understanding how to navigate nested JSON objects (which become Python dictionaries/lists) to extract specific data.
Filtering: Applying logic to filter data based on specific criteria (e.g., finding moves available in a particular game version).
Common mistake: Not specifying the correct version or filtering criteria. The question required finding moves common to specific Pokémon in the “x-y” version.

Q29: How can I extract data from dynamically loaded websites using JavaScript, and where can I learn about it?

A29: For dynamic websites, a simple requests call might not capture all content loaded by JavaScript.

JavaScript in Dev Tools: You can use JavaScript directly in your browser’s Developer Tools console (F12) to interact with the Document Object Model (DOM). This allows you to access and extract data from the fully rendered page.
Recommended Resource: The “Scraping with JavaScript” video under “Optional Live Recordings” (which I consider essential) demonstrates this process in detail.
CSS Selectors (for JavaScript): To effectively extract specific elements from the DOM using JavaScript, you’ll need to know CSS selectors.
- Learn about CSS selectors using the excellent “Beautiful Soup” documentation.
- Learn the basics of CSS (selectors, combinators, hierarchy) from the MDN (Mozilla Developer Network) website. Only learn up to “combinators” – that’s all you need for this course.
Cheat Sheet: I have created a cheat sheet with essential JavaScript code snippets for common DOM manipulation tasks. I’ll share this next time.
Key takeaway: Focus on understanding how to extract information from the DOM using JavaScript, not necessarily becoming a JavaScript expert.

Q30: What was the point of the PDF extraction exercise (GA2 Q3)?

A30: This exercise highlighted two main things:

tabula-py: How to use the tabula-py library to extract tabular data from PDFs into DataFrames. You can read all pages (pages='all') or specify a range (pages='9-80').
Common Mistakes:
- Not using multiple_tables=True when a PDF page contains multiple tables (e.g., a header and a main table). This can lead to incorrect data aggregation.
- Failing to do a “sanity check”: Always visually inspect a small sample of the extracted data to confirm it looks as expected before processing the entire dataset.

Q31: What’s the best way to calculate sums on filtered data in Google Sheets (GA2 Q4)?

A31: When working with filtered data in Google Sheets, it’s crucial to use SUBTOTAL(9, range) instead of SUM(range). The SUM() function includes hidden rows, giving an incorrect total for filtered data, while SUBTOTAL(9, range) correctly sums only the visible cells.

Q32: What’s the strategy for solving K-Means, DBSCAN, PCA, and Dimensionality Reduction questions (GA2 Q5-Q9)?

A32: I will thoroughly explain these concepts in the next session. This includes covering K-Means from scratch (initialization, assignment, update, convergence), the Elbow Method for optimal k, DBSCAN, PCA, and general dimensionality reduction principles.

Q33: How should I approach solving GA assignments for ROE, especially concerning domain knowledge and technical complexity?

A33:

Domain Knowledge: For ROE, you won’t be expected to have prior domain knowledge. Questions will be designed for straightforward understanding.
Technical Complexity: While complexity might exist in the data or task, you’ll have been shown how to quickly overcome it (e.g., using specific libraries or techniques). The goal is to see if you can apply what you’ve learned to solve problems.
Learning: The key battle in ROE is how to get the information you need, not necessarily having complex code. My aim is to teach you how to quickly get that information.
Focus: Understand the thought process behind obtaining values, using tools like Google Sheets or specific Python libraries. Practice makes perfect.

Q34: What are “emerging tools” mentioned in the course, and are they important?

A34: Anand Sir has added “emerging tools” to the course. I haven’t personally looked at them yet, so I can’t comment directly. However, Anand Sir generally doesn’t include useless things in the course, so if they’re there, they’re probably valuable. I encourage you to explore them if you have the opportunity.

Q35: Do you have any general advice or final thoughts for us?

A35:

I hope this session was useful, and that you’ve picked up something valuable, especially for ROE.
The goal is to empower you to handle challenges efficiently. Don’t be afraid of complexity; focus on breaking down problems and using the right tools.
I hope I’ve imparted something of value to you.

Q36: Can you give us some details about the upcoming GA3 session?

A36: GA3 will be very interesting. I will thoroughly explore Pandas in that session, showing how to solve complex issues. I’m looking forward to it.