TA Session 30 June : More complex webscraping#

Duration: 2h 5m

Here’s an FAQ based on the provided transcript:

Q1: What is the main problem we are trying to solve with web scraping today?

A1: We’ll be given a web page, which acts as an index. From this page, you’ll find some initial information, like user names. These names are actually links to other individual pages. Your task is to visit each of those linked pages, scrape more specific information from them, and then consolidate all this extracted data into a single file for potential visualization. It’s a slightly more complex, real-world web scraping scenario.

Q2: I heard that the recent bonus project on Twitter scraping is difficult. How can I do it?

A2: If you can successfully complete the complex web scraping problem we are tackling today, you’ll have the skills needed to approach that Twitter scraping task.

Q3: Can you explain the specific task we need to accomplish for today’s session?

A3: You need to scrape information about approximately 1200 students listed on a main “autobiographer” page on our Discourse website. For each student, you must retrieve: their name (or username), the total number of badges they have earned, and the names of all those badges. All this data should then be compiled into a single structured format, like a CSV file.

Q4: Do I need to use cookies for this scraping task?

A4: Yes, using cookies is essential for this task, along with providing other appropriate headers and parameters in your requests.

Q5: Can I use the Curl to Python method for handling cookies?

A5: While it might technically work, I generally don’t recommend or use Curl methods as they can be overly complex for this kind of task. I’m looking for a simpler, more direct Python-based web scraping approach today.

Q6: What’s the best approach to start solving this multi-page scraping problem?

A6: The best way to figure out the pattern is to first try scraping data for a single student. Once you get that working, try it for two, then three students. You’ll start to see a pattern emerge in how the data is structured and how to navigate. Then, you can encapsulate that pattern into a loop to process all 1200+ students automatically. Remember to always use your browser’s developer tools to inspect the network tab and elements.

Q7: I previously worked on this type of problem and scraped the autobiographer page. I could get the student’s name, when they got a badge, and their username. Is this the same data you’re asking for?

A7: My requirements are slightly different. I need the total number of badges and the names of all badges, not just when they received one. And while you could get some data, you mentioned you couldn’t directly get the anchor links for each user’s profile. We need those links to visit their individual pages and gather the full badge information.

Q8: So, to get all the badge information, I need to go to each of the 1200+ student profiles and then navigate to their specific badge page?

A8: Yes, that’s correct. You’ll need to automate navigating from the main list page to each individual student’s profile page, and then from their profile page to their dedicated “badges” page. This allows you to extract the required badge count and names for every single student. You could do this manually, but for 1200 students, that would take over 20 hours! The goal is to find a clever, automated way to do it in about 1 to 1.5 hours.

Q9: I noticed when inspecting the main page that not all user information loads initially. It appears as I scroll down. How do I handle this dynamic loading?

A9: This is a common issue with dynamically loading websites. You need to ensure you scroll down the entire page (either manually or programmatically) until all 1200+ student entries have loaded into the HTML. Only then will you have all the necessary usernames and their respective profile links available in the page’s source code for scraping.

Q10: I’m having trouble extracting the direct URL to each user’s profile page (the anchor tag link). It doesn’t seem to be reliably available.

A10: We found that the anchor tag link is available within the HTML once all content has loaded. You can use BeautifulSoup’s find_all method to target specific div elements (like user-info-media), and then within each of those, find the span with the username class to get the name, and the a tag (anchor) to extract its href attribute. This href will be the direct link to the user’s profile page.

Q11: What about providing headers for the request?

A11: Yes, it’s crucial to provide appropriate headers (including User-Agent) with your requests to avoid being blocked by the website. Many sites have policies against excessive requests from the same IP address or generic user agents. This is a common practice learned from experience in web scraping.

Q12: Can you summarize the entire working solution and the steps involved?

A12:

Load Full Page: First, ensure the entire main “autobiographer” page (with all 1200+ users) is loaded by scrolling down until all dynamic content appears.
Save HTML: Save the complete HTML of this main page as an .html file (e.g., auto.html).
Read into BeautifulSoup: Read this .html file into a BeautifulSoup object.
Extract User Info: Use BeautifulSoup to find all the relevant HTML elements (e.g., div with class user-info-media) containing student details. Loop through these elements:
- Extract the student’s username (from a span with class username).
- Extract the direct profile URL (from the href attribute of an a tag).
Iterate & Scrape Profiles: For each extracted profile URL:
- Construct the full URL to the user’s badge page (e.g., by replacing “/summary” with “/badges” in the profile URL).
- Make a requests.get() call to this badge page URL, making sure to include necessary headers (like User-Agent) and cookies.
- Parse the HTML content of the badge page using BeautifulSoup.
- Extract the total number of badges and the specific names of each badge.
- Store all this information (username, badge count, badge names) into a list of dictionaries.
Final Output: Convert the list of dictionaries into a CSV file.

Q13: Can I share the code I developed during the bonus project?

A13: We appreciate the offer, but during these live sessions, we encourage everyone to figure out the solutions independently.

Q14: Is there a library that can make this process faster?

A14: Yes, there are libraries specifically designed to simplify and potentially speed up web scraping tasks, though we focused on a more fundamental approach today.

Q15: Why are we not using JavaScript for this, especially since I’m good at it?

A15: The TDS course is part of a Diploma in Data Science, and we aim to keep the technical prerequisites to a minimum. Therefore, we prioritize Python-based solutions that are more universally accessible to all students in the program, rather than relying on JavaScript-specific methods.

Q16: What are your final thoughts on this session?

A16: This was a challenging but rewarding session! We went through a lot of trial-and-error, which is very typical in real-world web scraping. We successfully managed to extract the required information for each user by combining techniques like HTML parsing, URL manipulation, and handling dynamically loaded content. My appreciation goes to all of you for your patience and active participation!Here’s an FAQ based on the provided transcript: