TA session 13 June : Introduction and week 1,2 recap#

Duration: 2h 2m

Here’s an FAQ based on the live tutorial:

Q1: What are your names and roles?

A1: I’m Amit, and I’m a Teaching Assistant (TA) for the TDS course, this being my third term. Carton is also a TA; he was a student last term. Please call us by our names, Amit and Carton, not “sir,” as we are all essentially students here.

Q2: Have there been any changes to the course portal this term?

A2: Yes, our course portal looks a bit different from other subjects, as you might have noticed. One key change is that all the videos for the first two weeks are now available on a single page, which is a new update for the TDS course.

Q3: What exactly is “Tools in Data Science” (TDS) about?

A3: TDS is not just a three-credit subject; it’s a journey about problem-solving using data. It’s more like a skill-based course that covers several steps:

Data Discovery: Identifying what kind of data you need to answer a problem.
Data Sourcing: Obtaining that data.
Data Cleaning/Preparation: Processing the raw data, which might be messy, to make it usable.
Modeling: Applying mathematical or machine learning models to the prepared data.
Output Design: Creating clear, visual outputs from your analysis, as your audience (bosses, managers) may not be data scientists.
Storytelling: Presenting your entire data journey, from problem to solution, in an understandable narrative.

Q4: Do I need prior knowledge in Mad1 or MLPD to do well in TDS?

A4: While it’s good if you already have some knowledge from courses like Mad1 or MLPD, it’s not strictly mandatory. We teach you the basics, and the course is designed as a journey. For those without prior experience, we’ve provided extensive resources to help you catch up.

Q5: Why can’t I see all the course content on the portal?

A5: The course content is rolled out gradually, usually two weeks at a time. This is because the course is structured as a journey, and you’re meant to progress through it step-by-step.

Q6: I’m new to the diploma program. Is TDS a good course to start with?

A6: It seems many of you are new. About 10-12 students in this session are either in their first or second term. TDS can be taken early, but be prepared for its skill-based nature.

Q7: I’m having issues with background noise during the session. Can this be fixed?

A7: I apologize for the background noise. It’s because other TAs are in the same room, also conducting sessions. We’ll try our best to minimize it. Carton also apologized for the noise.

Q8: I have a doubt about Week 1 and Week 2 content, specifically a script error in Excel when scraping data. Can you address that?

A8: Yes, we will definitely go through that specific issue during the content review.

Q9: I also posted on Discord about getting the location URL from weather apps. Will that be covered?

A9: Yes, I’ve seen your post on Discord. We will definitely cover that during the session.

Q10: I can’t find the supplementary content (like slides) for TDS. Also, Week 1 doesn’t show sub-modules or a clear list of videos. Is this a known issue?

A10: Yes, this is a known issue.

Supplementary Content: All supplementary content, including slides and other materials, is currently embedded underneath each video on the single-page layout. This isn’t the ideal way, but that’s how it’s structured for now.
Video Visibility/Sub-modules: This is a problem that was identified early on and raised with the IT team and instructors. They are hoping to correct it at some point. Currently, all videos for a week are presented on one page without clear sub-modules (e.g., 1.1, 1.2). You’ll need to scroll down to find them. The important tools like DBF opener are also embedded under videos, especially in Week 2.

Q11: What do you mean by “notebooks,” and how should I use them?

A11: When I say “notebooks,” I’m referring to Colab notebooks. They are extremely important for this course.

Value: They often contain pre-written code that demonstrates concepts, saving you initial setup and thinking time.
Usage: Don’t just “plug and play.” Try to understand what the code does.
Saving: Always make a copy of the notebook in your own Google Drive before you start working on it. This prevents issues with shared notebooks and ensures your changes are saved correctly.
Learning: Play around with the code in your copied notebook. Don’t rely solely on the live sessions; spend time studying independently. I will also go through these notebooks in detail during the sessions.

Q12: How many ROEs (Review of Essential Skills) will we have in this course, and what will they entail?

A12: You will have exactly one ROE in this course, likely scheduled for the third or fourth week of July. We will point out important topics for the ROE during our content review sessions. The “Optional Live Session Recordings” from Week 2 are particularly crucial, as 50% of your ROE might come from the concepts covered there (web scraping, URLlib, Beautiful Soup).

Q13: What are the take-home projects about? Will they be difficult, and how are they graded?

A13: The take-home projects will be assigned before the ROE. They are not designed to be overly difficult.

Purpose: They assess your fundamental data manipulation skills on very clean datasets that we have carefully prepared. You’ll use tools like Pandas or Excel to extract valuable information and answer a set of randomized questions.
Grading: A significant component of the projects will be peer-reviewed. This is to simulate real-world scenarios where colleagues evaluate your work and to broaden your learning by exposing you to different approaches. Detailed grading criteria are provided in the course’s grading document or course introduction resources.

Q14: I’m a novice with Excel. How much Excel proficiency is required for TDS, and will you cover it?

A14: No worries if you’re new to Excel. We will cover Excel during the course. Our goal is to emphasize open-source tools like Google Sheets as much as possible, as some students may not have Excel. However, Excel is widely used in industry, so familiarity with it is beneficial. We will have dedicated sessions for both Excel and Google Sheets.

Q15: Can we share useful resources, like NPTEL courses or YouTube tutorials, on Discord?

A15: Absolutely, please do! Sharing good resources is highly encouraged. In data science, continuous learning and finding solutions online are key skills. If you find a great resource, share it on Discord so everyone can benefit. (One student mentioned Chan.org for Excel tutorials, and I encouraged them to share the link on Discord).

**Q16: Can the live sessions be shifted from Sunday evenings to Monday? Sunday evenings coincide with assignment deadlines. **

A16: I understand your concern, and others have raised this too. However, scheduling these live sessions is a complex process managed by the NPTEL and BS Degree teams, not by us directly. They try to accommodate everyone, but with a large student body, it’s difficult to find a time that suits all. We did relay your feedback. The good news is that all live sessions are recorded. If you can’t attend, you can always watch the recordings at your convenience.

Q17: What are the types of data, and what’s the difference between structured, semi-structured, and unstructured data?

A17: We covered this in Week 1.

Data Sources: Data can come from various sources:
- Public Data: Freely available (e.g., Kaggle, government websites). Often needs cleaning.
- Private Data: Requires payment or specific access (e.g., proprietary company data).
- Personal Data: Collected from your own devices (e.g., phone data).
Data Types:
- Structured Data: Organized with clear fields and attributes (like columns in a spreadsheet), defined data types, and is easy to manipulate (e.g., relational databases, CSVs).
- Semi-structured Data: Has some organizational properties but isn’t strictly tabular, often with flexible schemas (e.g., JSON, HTML, XML, emails). It might not easily fit into a flat file.
- Unstructured Data: Has no predefined structure, making it difficult to categorize and analyze (e.g., images, audio, video, free-text documents). Its size and format can vary widely.

Q18: What are shape files, and how are they used in data analysis?

A18: Shape files contain geographical coordinates that define shapes, which are typically used for representing maps or spatial data. For example, they can be used to precisely map property boundaries (like in India) or administrative districts. This has great benefits for urban planning, property management, and even monitoring agricultural activities (as demonstrated by satellite imagery used by governments). It’s a very powerful tool in data analytics.

Q19: What are the different types of data values, like categorical, numerical, and composite?

A19: This topic is usually covered in statistics.

Categorical Values: Represent qualities or categories and cannot be subjected to mathematical operations. Examples include names, colors, or images. Some categorical data can have an inherent order (ordinal data, like “good,” “better,” “best,” or ratings like 1-5 stars), but you can’t perform mathematical calculations like “better is three times good.”
Numerical Values: Represent quantities and can be subjected to mathematical operations (e.g., age, height, price).
Composite Values: A combination of different data types clustered together to form a specific structure (e.g., a customer record containing name, age, and purchase history).

Q20: What’s the best way to approach Week 2, especially with the notebooks, and what kind of tools will be introduced for scraping?

A20: Week 2 is where the “real meat” of the course begins.

Notebooks: The notebooks are crucial. Don’t just run the code; try to understand what each line does. Remember to make a copy to your drive before editing.
Developer Tools: For web scraping, a fundamental tool you’ll use is your browser’s Developer Tools (right-click -> Inspect, or F12). This helps you examine the underlying structure of a webpage.
API Calls: We’ll learn how to identify and use API calls. An API (Application Programming Interface) is like a waiter in a restaurant; it’s an interface that allows different software applications to communicate and exchange data. Instead of manually navigating a website, you can use the API to directly request and receive specific data.
Scraping Techniques: We’ll cover libraries like URLlib and BeautifulSoup for Python, which are essential for web scraping. These will be especially important for your ROE.
Practical Application: The goal is to move beyond manually extracting data from websites and programmatically request data using APIs or scraping tools.

Q21: I’m still encountering script errors in Excel when trying to scrape data, even after trying to fix them. What should I do?

A21: We’ll address your specific Excel script error. Additionally, sometimes issues arise from the dynamic nature of websites or browser extensions. We’ll look into using the Developer Tools to inspect network traffic and identify the correct API calls.

Q22: Why did my screen show hundreds of requests in the Developer Tools when I only typed one city, and then only one request when I typed slowly?

A22: This is an important observation about how web applications work:

Full Page Reload: When you press Enter, your browser typically requests an entirely new page, which involves fetching all its components (images, scripts, ads, etc.). This can generate hundreds of background requests, even for a simple search.
Asynchronous Requests (AJAX): When you type slowly, the website often uses “asynchronous” requests (like JavaScript’s AJAX). Instead of reloading the whole page, it sends a small request with your partial input (e.g., “New Yo”) and the server responds with suggestions (e.g., “New York”). This generates fewer, more targeted network activities, making it easier to pinpoint the specific API call for location data.
Filtering: To find the relevant API call, you can use the “Fetch/XHR” filter in your browser’s Developer Tools network tab. This filters out most extraneous requests, leaving only the data-related API calls.

Q23: Why should I bother finding these API calls and not just scrape data directly from the website or simply get the ID from the URL?

A23: Understanding API calls is crucial because:

Efficiency: Directly calling an API is much more efficient than scraping an entire webpage, especially for repetitive tasks or large datasets. It’s like ordering food with a button instead of going to the restaurant, sitting down, and asking a waiter.
Reliability: Websites can change their visual layout frequently, which breaks scraping scripts. APIs are more stable endpoints for data access.
Programmatic Access: Once you have the API call’s structure (its “JSON body”), you can programmatically fetch data using Python (e.g., requests library) or other tools, allowing for automation and integration into applications.
“Swiggy API” Analogy: Think of it like a food delivery app. You’re not physically going to the restaurant. Instead, you’re interacting with their “API” to get the food delivered directly. This is what we aim to do: get data directly, not manually.

Q24: Do I need to memorize all the Colab code for the ROE, or will resources be available?

A24: No, you absolutely do not need to memorize all the Colab code for the ROE.

Open Internet Exam: The ROE is an open-internet exam. This means you can use any resources you like, including your notes, Colab notebooks, Google searches, and even asking friends (though obviously not during the exam itself for direct answers).
Skill Testing: What they are testing is not your ability to recall code, but your intuition to quickly find and apply solutions.
Preparation: The best way to prepare is to practice. Make a bookmark folder for all your Colab notebooks from the course. If you need to quickly grab a code snippet for something you’ve done before, you can easily find it.

Q25: What tools and Python packages will be introduced in TDS?

A25: We’ll be working with a variety of tools and Python packages relevant to data science.

General: Expect to use tools like Excel, Google Sheets, Pandas (Python library for data manipulation), and various Python scripts.
Web Scraping: URLlib, BeautifulSoup, and JavaScript for browser-based scraping (demonstrated using Developer Tools). We’ll also cover tools like NominaTim and Tabula for specific scraping tasks.
Flexibility: We give you some foundational tools, but our goal is to make you aware of what’s available. You’re free to use any tool you’re comfortable with to get the job done. The course simulates a real-world scenario where you’d use any available tool to solve a problem.

Q26: I have an older version of Excel (2016) and Windows. Will this be a problem, and how can I get the latest versions?

A26: Having older software can sometimes cause compatibility issues.

Recommendation: It’s best to install the latest version of Microsoft Office (which includes Excel) and upgrade your Windows operating system to version 10 or 11. These recent Windows versions are more likely to support all the necessary tools and packages without issues.
Installation: You can usually download the latest versions from the Microsoft website. If your system has hardware limitations preventing an upgrade, you might need to consider an SSD extension for better performance.

Q27: My question is about the ROE being an open-internet exam. Does that mean it won’t be proctored? And will we be tested on dynamic websites like Justdial or on older technologies like Flash/Silverlight?

A27:

Proctoring: The ROE is not proctored. You can use any resources, phone a friend, or do whatever you need to find the answers. However, you’ll be on a strict time limit (maybe 30-45 minutes) with many questions, so quick intuition is key.
Dynamic Websites: For the ROE, we will not ask you to scrape dynamic websites (like Justdial) or websites that rely on old technologies like Flash or Silverlight. The reason is logistical: we cannot create a static answer key for data that constantly changes. You will be tested on static websites where the data remains consistent.
Testing Skills: The ROE is designed to test your skills in data science, not your ability to handle complex, real-world website quirks or outdated tech. It focuses on your intuition and ability to apply learned methods efficiently.

Q28: Will you be providing solutions or explanations for the graded assignments after the deadlines, so we can learn from them?

A28: Yes, after the graded assignment deadlines, we will definitely go through the solutions and provide explanations. This is crucial for your learning, and we’ll ensure you understand the correct approach and how to tackle such problems in the future.

Q29: Can you please guide us on what specific practical aspects from Week 1 and Week 2 we should focus on for professional development, beyond just scoring points?

A29: Yes, that’s a great question, and it aligns with the course’s goal of professional development. I’ll inform Amit about this, and for the next session, we can dedicate time to specifically highlight these practical takeaways from each week.

Framework: Our aim is to provide you with a framework. We’ll give you foundational tools and practical examples.
Self-Learning: Your journey involves using this framework to explore further. If you’re learning something like Pandas, try to become proficient in it by practicing beyond the course material. Don’t just learn to score; learn to apply.
Hands-on: We’ll be doing more hands-on sessions in the future, including challenges and assignments that will require you to apply your skills and critically think. This will help you identify areas where you need more practice or learning.

Q30: What specific tools and Python packages will be covered in Week 2, for example, for web scraping using JavaScript?

A30: In Week 2, we dive into practical web scraping techniques.

JavaScript Scraping: We’ll cover scraping using JavaScript directly from the browser’s console (Developer Tools). This is a newer method compared to traditional Python libraries. I myself learned this recently! It’s very straightforward, but it’s important to go through the provided video for it.
NominaTim and Tabula: We’ll also look at tools like NominaTim (for geographical data) and Tabula (for extracting tables from PDFs).
General Approach: Remember, we’re not just giving you tools; we’re teaching you how to get the job done. You can use any tool you’re comfortable with, but we’ll show you the core ones.

Q31: What about scraping data from PDFs using Tabula?

A31: Scraping PDFs using Tabula is very straightforward, typically requiring just three or four lines of code. It’s an efficient way to extract tabular data from PDF documents. In fact, you’ll need to use PDF scraping in one of your graded assignments.

Installation: When installing Tabula, remember to use pip install tabula-py. The documentation mentions this, but it’s a common point of error.
Process: You read the PDF, specify the page you want to extract data from, and Tabula creates a data structure (like a Pandas DataFrame) that you can then manipulate, transform, or export (e.g., to a CSV file) using just a few lines of code.

Q32: Will we be expected to be JavaScript experts for the ROE or final exam?

A32: No, you are not expected to be JavaScript experts.

Focus: The course is not designed to make you a JavaScript master. At most, you might get questions in the end-term final exam related to basic concepts or specific code snippets we’ve covered.
Understanding: The goal is to understand how JavaScript interactions work on a webpage in the context of data scraping, not to write complex JavaScript code. We won’t test you on advanced JavaScript programming skills.

Q33: How can I get copies of the session recordings?

A33: I will share the link to our YouTube channel in the chat. All our live session recordings will be posted there. Subscribe to the TDS YouTube channel, and you’ll be able to access them.