05 ROE revision session jan 24#

05 ROE revision session jan 24

Duration: 2h 56m

Here’s an FAQ-style transcription of the live tutorial, addressing common questions and providing concise answers.

ROE Exam Preparation FAQ#

Q1: What are the general rules and format of the ROE exam?

A1: The ROE exam is not proctored, meaning you are allowed to use any resources, including friends, ChatGPT, or even data scientists in your family. It’s completely open book. The exam duration is 45 minutes, as specified in your email. It’s structured like a timed graded assignment; the portal opens at a specific time (e.g., 8:00 AM) and closes precisely at the end (e.g., 8:45 AM). Questions will be multiple-choice (MCQ) or Numerical Answer Type (NAT). You will not be required to upload files, but you may need to download several files via provided links during the exam.

Q2: What types of questions will be on the ROE exam, and what are their approximate weightings?

A2: There are three main types of questions:

  1. Scraping (approx. 50%): This includes web scraping (HTML, XML) and PDF scraping.
  2. Python Libraries (approx. 20%): These questions will specifically ask you to use certain Python libraries, primarily those discussed in Weeks 2 & 3 of the course. They won’t ask you to apply complex machine learning models.
  3. Excel Tables (approx. 30%): This section involves using pivot tables and filters, along with basic calculations like sum, average, and percentage.

Q3: What kind of websites will I encounter in web scraping questions?

A3: You will be given simple, static websites, not dynamic ones like YouTube that constantly update. Examples include Wikipedia-like pages. Even if a website has a lot of content, its underlying structure will be straightforward. You might also encounter websites that require handling cookies, especially if login information is involved.

Q4: What types of files might be provided for scraping tasks?

A4: For web scraping, you might get a live website or an HTML file located on your local drive. For XML scraping, they could provide a government data website (like data.gov) or an XML file directly. For PDF scraping, it will either be a website containing PDFs or a direct PDF file.

Q5: How should I prepare for and approach the Python libraries section?

A5: The trick is to create a personalized Jupyter or Colab notebook containing all the relevant code snippets from the Weeks 2 and 3 lectures (e.g., for Tabula, Nominatim, Wikimedia, BBC weather/lat/long data). Just before the exam, open and run this notebook. This way, when a Python library question appears, you can quickly find the relevant code, make minor adjustments (like changing a city name), and get the answer within seconds. While it takes 2-3 hours to create this notebook, it’s a significant time-saver during the exam. Avoid relying on shared notebooks, as you might not understand the specific variables or logic used by others.

Q6: What is the best strategy for handling Excel table questions?

A6: For Excel table questions (which can involve CSV, TSV, or Excel formats), the most important tools are pivot tables and filters. Most problems can be solved using pivot tables, and the rest with filters. You can use Google Sheets, MS Excel, LibreOffice, or Apache Office. To optimize time, download the Excel file right at the start of the exam, and let it upload in the background while you read other questions. This prevents losing valuable time if the file is large. When working with filtered data, always copy the filtered subset and use “Paste Special” -> “Values Only” into a new sheet before performing calculations. This ensures you’re working with actual values, not potentially broken formulas from the original data.

Q7: Will I need to submit my Excel or Python code files?

A7: No, you are only expected to provide the final answers (e.g., for MCQ or NAT questions). There is no requirement to upload your working files.

Q8: Are recorded sessions and previous year’s papers available?

A8: Today’s session is recorded and will be uploaded to the TDS YouTube channel. Other Colab notebooks from previous lectures are available on the portal, just below each lecture. However, some are older (e.g., 2 years) and may require minor modifications to run correctly. Previous year’s papers are generally not shared by IT Madras, though you might find some discussions on platforms like Discord. The instructor advises focusing on the current topics rather than past papers.

Q9: What are some general tips and tricks for the ROE exam?

A9:

  • Time Management: Be quick and strategic. Fight against the clock.
  • Utilize Resources: Since it’s open book, use everything at your disposal: friends (e.g., have a data scientist family member nearby), ChatGPT, open-source software, and your prepared notebooks.
  • Problem-Solving: Don’t get stuck on one method. If a website is complex, don’t just rely on Python code; use browser tools like Ctrl+F to manually search for information if you’re short on time.
  • ChatGPT: Use chat.openai.com. Create an account (it’s free and easy). For basic questions (like “how to find max value in pivot table?”), ChatGPT provides direct, pinpoint answers, saving time compared to searching multiple websites.
  • Data Handling: Always use “Paste Special” as “Values Only” when copying filtered Excel data to prevent formula issues.

Q10: Should I aim for 50% in ROE?

A10: No, aim for at least 90%. While previous students might have found it challenging, with the right strategy and preparation, a higher score is achievable.

Q11: How can I access recorded sessions or additional resources for specific topics like XML scraping?

A11: Recorded sessions, including this one, will be uploaded to the TDS YouTube channel. For specific topics like XML scraping, it’s covered in the lectures, and you can find related materials in the designated playlists on the YouTube channel or below the lectures on the portal.

Q12: What’s the best way to extract data from an XML file using Python?

A12: You can use the requests library to fetch the XML content from a URL or use Python’s file handling (open('file.xml', 'r')) if the file is local. Then, use BeautifulSoup (or lxml for more advanced parsing) to parse the XML content. For example, soup = BeautifulSoup(file_content, 'lxml') would parse it. You can then use soup.find_all('tag_name') to extract specific elements. To handle initial stylesheet tags, you might need to clean the raw XML content before parsing, or directly target the relevant data tags.

Q13: How do I convert the extracted XML data into a CSV file and download it?

A13: After extracting the XML data into a Python list of lists (e.g., using BeautifulSoup), you can convert it into a Pandas DataFrame using pd.DataFrame(your_list_variable). Once it’s a DataFrame, you can save it as a CSV file using your_dataframe.to_csv('your_file_name.csv', index=False). To download it from a Colab notebook, the file will appear in the left-hand file browser. You can then right-click and download it.

Q14: How can I scrape PDF tables into Excel using Python or directly?

A14:

  • Using Python: You can use libraries like Tabula-py (after installing Java on your system or confirming its availability in your Colab environment) to scrape PDF tables into a Pandas DataFrame. Once in a DataFrame, you can convert it to CSV/Excel.
  • Directly in Excel/Sheets: You can open a PDF directly in Excel or Google Sheets (via the ‘Data’ tab, then ‘From PDF’ or ‘From Web’ if it’s a URL). Excel/Sheets will attempt to recognize and import tables. However, this method might not be as accurate for complex or visually inconsistent PDFs compared to programmatic tools.

Q15: What if some of the provided code from the lectures doesn’t work?

A15: Some of the provided Colab notebooks and codes from previous lectures might be old (e.g., 2 years) and may no longer work as intended due to library updates or changes in website structures. You might need to perform minor modifications or debug them to get them running. If you encounter issues, you can search for updated solutions or alternative methods (e.g., using ChatGPT for help).

Q16: Is it possible to collaborate or discuss answers during the exam?

A16: While the exam is open book and technically allows collaboration, the instructor advises against relying on group discussions during the exam. In previous experiences, group chats become chaotic with everyone asking questions and few providing answers. It’s best to prepare independently. If you have a friend who has already completed TDS, having them nearby to offer guidance (not direct answers) might be helpful.

Q17: Will the questions be randomized for each student?

A17: Yes, the questions will be randomized for each student. Even if multiple students receive the same base dataset, the specific questions asked about that data will differ.

Q18: What is the recommended approach for working with an XML file, especially if it contains styling information?

A18: When working with XML files, especially those with styling information (like CSS stylesheets), you’ll first need to clean the raw XML content. Before parsing, remove any non-data-related tags or instructions that might interfere with extraction. Once cleaned, you can proceed to parse the relevant data using libraries like BeautifulSoup in Python. The key is to focus on extracting the structured data blocks rather than the presentation elements.

Q19: How do I open and edit an XML file locally before uploading it to Colab?

A19: First, download the XML file to your local computer. Then, right-click on the downloaded XML file and select “Open with” -> “Notepad” (or any other plain text editor). This allows you to view and edit the raw XML content. Make any necessary deletions (e.g., removing the stylesheet line at the top) and save the file. After saving, ensure the file extension remains .xml (you might need to manually rename it if it saves as .txt). Then, upload this cleaned XML file to your Colab environment.

Q20: What is the best way to handle large datasets in Colab, particularly when downloading or uploading?

A20: When dealing with large datasets, it’s always best to download them directly from the source to your local computer first. Then, upload the file from your local disk to your Colab environment. Avoid using direct connections (like drive.mount for Google Drive) as the connection can be unstable, causing your code to crash. Uploading locally ensures a smoother workflow.

Q21: Can I use beautiful soup for an HTML file instead of XML?

A21: Yes, BeautifulSoup is versatile and can parse both HTML and XML content. When initializing BeautifulSoup, you specify the parser, e.g., BeautifulSoup(content, 'html.parser') for HTML or BeautifulSoup(content, 'lxml') for XML.

Q22: Can I use BeautifulSoup to scrape tables from HTML content directly into a Pandas DataFrame?

A22: Yes, once you have the HTML content (either from a file or a URL) parsed by BeautifulSoup, you can convert the entire content (as a string) into a Pandas DataFrame using pd.read_html(str(soup_content)). This function will automatically identify and extract tables from the HTML and return them as a list of DataFrames. You can then select the specific DataFrame (table) you need (e.g., df = list_of_dataframes[index]).