TA Session 22 August: Mock and End Term, Revision: Modules 1 to 3#

TA Session 22 August: Mock and End Term, Revision: Modules 1 to 3

Duration: 2h 6m

Here’s an FAQ-style summary of the live tutorial:

TDS Live Tutorial: End Term Preparation FAQ#

Q1: What is the main agenda for today’s session?

A1: Today, I’ll be giving you a mock quiz for the final exam. After you attempt it, I’ll explain how it was designed, which might be useful for your other courses as well. If time permits, I’ll also provide a quick overview of the important topics and terms from the first three modules of TDS.

Q2: How will the mock quiz be structured, and how much time do we have?

A2: The mock quiz will have 40 questions, and I’ll give you 40 minutes to complete it, which is about one minute per question. I’ll share the link in the message window. Since it’s a practice, feel free to open your mic and ask questions. You’re allowed one submission, so please submit at the end.

Q3: Is this mock quiz from previous years, or is it newly generated?

A3: This mock quiz is not from previous years. It’s newly generated using all the course material, following the exact same process we used for the actual end term. There might be some slight variations from the main instructor’s personal approach.

Q4: Will there be bonus mark questions in the final exam?

A4: No, there are no more bonus mark questions. That was a feature in previous instances only.

Q5: The questions in this mock seem subjective (e.g., “What is the first step in data cleaning?”). Can we expect such subjective questions in the final exam?

A5: No, a question like that would never appear in the final exam. The exam questions will be much more robust and were carefully framed by the instructors. This mock was created at very short notice from a smaller, unfiltered pool, so some questions might seem inappropriate or subjective. It’s primarily intended to give you an idea of the difficulty level and the types of questions you might encounter. However, there will be non-subjective questions around topics like data cleaning and data sourcing.

Q6: If it’s an MSQ (Multiple Select Question) in the mock, and I select only some correct options, it marks it completely wrong. Will the final exam also do this?

A6: No, that’s just a peculiarity of this mock quiz due to its quick setup. The final exam will award partial marks for partially correct MSQs, as usual.

Q7: Should I be concerned about questions that require specific syntax (e.g., streamlit.text, GeoPandas, Nominatim)?

A7: You should only expect syntax-related questions at a very basic level. In the remaining three sessions, I’ll go over the absolute essentials you must know. If you focus on what’s covered in these review sessions, you should be well-prepared to score high.

Q8: I’ve forgotten a lot of the material. If I follow these three review sessions, will it be enough to score well, considering I have other subjects too?

A8: Definitely. Our primary goal with these review sessions is to prepare you as best as we can for the final exam. While your individual performance depends on you, we will cover all crucial material to help you succeed, just like we did for the RV.

Q9: Will you be discussing the answers to this mock quiz?

A9: Not extensively. The main purpose of the mock was to give you a feel for the question distribution, difficulty, and types. Many questions are self-explanatory. I will, however, explain the process we used to generate these questions, as a similar method was employed for the actual end-term exam. If you have a specific question about why an answer is right or wrong, I can address that individually, but we won’t go through every single question.

Q10: This mock quiz has 40 questions in 40 minutes, but the final exam is 1.5 hours for 40-60 questions. What’s the best strategy for the final?

A10: This mock gives you an idea of the pace you’ll need. My advice for the final end term remains the same: if you have multiple subjects, answer the TDS section first. TDS questions are generally quick (you either know it or you don’t), allowing you more time for other subjects.

Q11: What are the key topics and takeaways from Module 1 (Data Discovery)?

A11:

  • Three sources of datasets: Personal (devices, usage), Public (GDLET, GADM, government portals, Kaggle, Google, Data Meet), Private (corporate, most common/restricted, detailed/accurate).
  • Three types of data: Structured (schema, databases, spreadsheets, shapefiles), Semi-structured (documents, XML, messages, may have partial schema), Unstructured (text, images, video, audio, needs processing). Data exists on a continuum.
  • Three types of values: Categorical (labels, e.g., True/False, colors, ordered like Low/Medium/High, cyclical like days of week), Numerical (integers/reals, full math ops, but watch units), Composite (made of categorical & numerical, e.g., currency, dates, spatial data, IP/URL, phone numbers).

Q12: What are the key topics and takeaways from Module 2 (Data Sourcing)?

A12:

  • Three ways to get data: Downloading, Querying databases, API/Web scraping.
  • Excel for Web Scraping: Use Data ribbon -> New Query -> From Other Sources -> From Web. (Watch the video for a demo).
  • Google Sheets for Data Sourcing: Functions like IMPORTXML, IMPORTFEED, IMPORTRANGE, IMPORTDATA, IMPORTHTML.
  • Python Libraries:
    • Requests: For making HTTP requests (GET, POST). Remember r.status_code, r.headers, r.encoding, r.text, r.json(). Responses can have cookies (r.cookies) for logins.
    • BeautifulSoup: For parsing HTML/XML (extensively covered in recordings).
    • Nominatim (GeoPy): For geocoding (lat/long from location name). Use geolocator.geocode("location", user_agent="user_agent").
    • Wikipedia: For searching and getting summaries from Wikipedia (wikipedia.search(), wikipedia.summary()).
    • Tabula: For extracting tables from PDFs (tabula.read_pdf(), tabula.convert_into()).
    • URL Encoding: urllib.parse.urlencode() to prepare URL parameters from a dictionary.

Q13: What are the key topics and takeaways from Module 3 (Data Preparation)?

A13:

  • Excel for Data Preparation:
    • TRIM(): Removes excess spaces.
    • Deleting blank rows: Find & Select -> Go To Special -> Blanks.
    • Removing duplicates: Data ribbon -> Remove Duplicates.
    • CONVERT TEXT TO COLUMNS
    • (Basic functions like Find & Replace, Change Date Formats, Data Transformation/Aggregation via Pivot tables, Filtering are also important).
  • Shell for Data Preparation:
    • Know common commands and their purpose (e.g., head, tail, grep, sort, uniq, wc, cat, cut, sed, awk).
  • OpenRefine for Data Cleaning:
    • Know terms like “faceted browsing” and “clustering methodologies”.
  • Pillow/PIL (Python):
    • Image manipulation library. Know basic functions for opening, resizing, cropping, and converting image formats.
  • Airflow:
    • Know its purpose: Connecting data processing tasks into pipelines.

Q14: Will there be specific questions about CSS and HTML, for example, related to web scraping selectors?

A14: Yes, there will likely be questions specific to CSS and HTML selectors, especially for web scraping purposes. This will be similar to what was seen in GA2, where you’re given a bit of code and asked which selector would extract specific information. I haven’t included these in the mock because they are harder to construct properly, but I want to let you know in advance to prepare.

Q15: Will the GPT-based TA (virtual assistant) trained on course material be available to students?

A15: I can inquire with Anand (the main instructor). The idea was to roll it out for all students, potentially for next term. This mock and the final exam questions were partly generated using this trained GPT, which was fed all the course materials and video transcriptions.

Q16: My project 1 marks are not visible on the dashboard. Where can I find them?

A16: If your project has been reviewed by all three assigned peer reviewers, you should be able to see your marks. You might need to check the peer review section within the portal, not necessarily the main dashboard. If you’ve had issues with reviews not being completed, the TA team will manually review those projects after the end term.

Q17: You mentioned “data lies on a continuum.” What does that mean, and how does it relate to structured, semi-structured, and unstructured data?

A17: It means that the distinction isn’t always black and white. For example, a spreadsheet is generally considered structured, but it might contain a chart or a block of text (unstructured data) within it. So, while it’s predominantly structured, it can have elements from other types. This concept helps understand that data often blends these categories.

Q18: What is meant by “data extraction” and converting unstructured data to structured data?

A18: Data extraction is about getting specific information out of raw data. A major part of this involves converting unstructured data (like text, audio, video) into a structured format that can be easily analyzed. For example, transcribing an audio file converts unstructured audio into semi-structured text. This text can then be further processed (e.g., extracting keywords or sentiment scores) to become fully structured data.