TA Session 16 June : Introduction to HTML and web scraping#

Duration: 2h 7m

Here’s a summary of the TDS live tutorial structured as an FAQ:

TDS Live Tutorial FAQ: Web Scraping Basics#

Q1: What will we cover in this session?

A1: We’ll start from scratch with web scraping. This includes creating a very simple website, understanding how to read and interpret the structure of a website (even complex ones like Wikipedia), and the fundamental concepts of web scraping. We’ll also cover a special topic: how to scrape websites that require authentication or login using “cookies.” We’ll practice on real-world websites like Wikipedia. If time permits, we’ll delve into more advanced topics.

Q2: Why is understanding website structure important for web scraping?

A2: All websites are built on HTML, which acts as their “skeleton.” Even if you have prior programming experience, understanding HTML is crucial because when you scrape a website, you’re essentially reading and extracting information based on the HTML tags and structure. Knowing this allows you to effectively navigate and extract the specific data you need from any website.

Q3: How do I create a basic HTML website?

A3: You can start by opening Notepad (which comes pre-installed on most computers). In Notepad, type <html>YourName</html>. Then, save this file with a .html extension (e.g., first.html). It’s crucial to use .html instead of the default .txt extension.

Q4: My HTML file opened, but the line breaks I typed in Notepad aren’t showing on the webpage. Why?

A4: HTML doesn’t automatically render line breaks from your text editor. To create a new line on your webpage, you need to use the <br> tag (for “break”). For example, Hello<br>World. The <br> tag is also a self-closing tag, meaning you don’t need a </br> closing tag.

Q5: What are HTML “tags” and what is the basic rule for using them?

A5: HTML tags are keywords enclosed in angular brackets (e.g., <html>, <head>, <body>, <p>). They define different parts of a webpage’s structure. The most fundamental rule is: whenever you open a tag, you must close it. Closing tags use a forward slash (/) before the tag name (e.g., </html>, </head>).

Q6: What is the basic structure of an HTML page?

A6: Every HTML page generally starts and ends with the <html> tag. Inside <html>, there are typically two main sections:

<head>: Contains metadata about the page, such as the page title (which appears in the browser tab).
<body>: Contains all the visible content of the webpage, like text, images, and links.

Q7: How do I add a title to my webpage and text to the body?

A7: To add a title, place the <title> tag inside the <head> section: <head><title>My Page Title</title></head>. For visible content, place text directly within the <body> tags. For example, <body>Hello World</body>.

Q8: How do I create headings of different sizes in HTML?

A8: HTML provides heading tags from <h1> (largest) to <h6> (smallest). For example, <h1>Main Heading</h1> or <h2>Sub-heading</h2>. Any tag beyond <h6> (like <h6>) will typically be displayed as normal text without special heading formatting.

Q9: How do I add a paragraph to my HTML page?

A9: You use the <p> tag to define paragraphs. This helps structure longer blocks of text, as each <p> tag typically creates its own distinct paragraph block.

Q10: How do I add a link to another webpage?

A10: You use the anchor tag <a>. It requires an href attribute, which specifies the URL of the page you want to link to. For example: <a href="https://www.wikipedia.org">Click here</a>.

Q11: How do I display an image on my webpage?

A11: You use the <img> tag, which is a self-closing tag. It requires a src (source) attribute that points to the image file. You have two main methods:

Online Image: Provide the URL of an image hosted online: <img src="https://example.com/image.jpg">.
Local Image: If the image is on your computer, it must be saved in the same folder as your HTML file. Then, you can simply use its filename: <img src="my_picture.jpg">.

Q12: I’m new to HTML. Should I follow along by writing code myself, or just watch?

A12: It’s highly recommended to do it side-by-side with me. This is a hands-on session, and the best way to learn is by practicing. Don’t hesitate to ask questions if you get stuck.

Q13: Is HTML knowledge mandatory for web scraping?

A13: Yes, it is very important. HTML forms the foundation of web pages, and web scraping heavily relies on understanding and navigating this structure.

Q14: How can I “inspect” a website to see its HTML code?

A14: On most browsers, you can right-click anywhere on a webpage and select “Inspect” or “Inspect Element” (sometimes “Developer Tools”). This will open a panel displaying the website’s underlying HTML code, along with other elements like CSS.

Q15: My website’s HTML looks very messy compared to the structured Wikipedia code. What’s the deal?

A15: You’re looking at the raw HTML content. To make it more readable and organized, you can use the BeautifulSoup library in Python, which helps “beautify” and parse the HTML into a more structured format, making it easier to navigate and extract information.

Q16: How do I bring website content into my Python script for scraping?

A16: You’ll use the requests library in Python. First, import requests. Then, specify the URL of the website you want to scrape. You can fetch the HTML content using requests.get(URL), which returns a response object. The actual HTML content is in response.content.

Q17: How do I make the raw HTML content more readable and navigable in Python?

A17: After getting the raw HTML content using requests, you use the BeautifulSoup library. First, from bs4 import BeautifulSoup. Then, create a BeautifulSoup object: soup = BeautifulSoup(html_content, 'html.parser'). This will parse the HTML into a tree structure that you can easily navigate.

Q18: What is html.parser used for in BeautifulSoup?

A18: html.parser is a built-in Python parser that BeautifulSoup uses to interpret the HTML string into a Python object tree. There are other parsers available, like lxml and html5lib, which might offer different performance or error-handling characteristics.

Q19: How do I find specific HTML tags within the BeautifulSoup object?

A19: BeautifulSoup provides methods like find() and find_all():

soup.find('tag_name'): Returns the first occurrence of the specified tag.
soup.find_all('tag_name'): Returns a list of all occurrences of the specified tag.

Q20: How do I extract just the text content from a found HTML element?

A20: After finding an element (e.g., first_paragraph = soup.find('p')), you can access its text content using the .text attribute: first_paragraph.text.

Q21: How do I find elements based on their attributes (like class or id)?

A21: You can pass attribute-value pairs as arguments to find() or find_all(). For example:

soup.find('div', class_='my-class') (note class_ because class is a Python keyword).
soup.find('p', id='unique-id').
You can also specify just the attribute without the tag: soup.find(class_='my-class') (though this is less common).

Q22: How can I get a list of all the words (text content) from specific links on a webpage?

A22: First, use soup.find_all('a') to get all anchor (<a>) tags (links). This will return a list. Then, you can iterate through this list using a for loop. Inside the loop, for each link element, you can extract its text content using .text and add it to another list.

Q23: How do I get the number of found elements (e.g., how many links are on a page)?

A23: Since find_all() returns a list of elements, you can simply use the len() function on that list: len(soup.find_all('a')).

Q24: What is the parser argument in BeautifulSoup(html_content, 'parser')?

A24: parser tells BeautifulSoup how to interpret the HTML. html.parser is a common built-in Python parser. Other options include lxml and html5lib, which might handle malformed HTML differently or offer performance benefits.

Q25: Why did you use class_ instead of class when searching by class name?

A25: In Python, class is a reserved keyword. To avoid conflicts, BeautifulSoup uses class_ when you want to specify an HTML class attribute in your search.

Q26: What’s the purpose of the find and find_all methods in BeautifulSoup?

A26: They allow you to locate specific HTML tags or elements within the parsed HTML document. find returns the first matching element, while find_all returns a list of all matching elements. This is fundamental for navigating the HTML tree and extracting desired data.

Q27: Can I search for elements only by their class name, without specifying a tag?

A27: Yes, you can. For example, soup.find(class_='my-class') would find the first element with that class, regardless of its tag type. However, it’s often more specific to combine tag and class (e.g., soup.find('div', class_='my-class')).

Q28: If I want to search for a tag and then, within that tag, search for a specific class, how would I do that?

A28: You first find the desired tag (or a list of tags), and then apply find() or find_all() on the result. For example, to find all <span> tags with the class “MW-headline” within a document:

soup.find_all('span', class_='MW-headline') (combining tag and class in one go).
Alternatively, you could do it in steps: all_spans = soup.find_all('span'), then loop through all_spans and check span.get('class') or span.has_attr('class') for the specific class. The first method is generally more efficient.

Q29: How do I iterate through a list of found elements to extract their text or attributes?

A29: find_all() returns a list-like object. You can use a standard Python for loop to go through each item in this list. Inside the loop, you can access attributes like .text for text content, or element.get('attribute_name') for attribute values.

Q30: What is the content attribute of the requests.get(URL) response object?

A30: response.content holds the raw, unparsed HTML content of the webpage as bytes. You typically pass this directly to BeautifulSoup for parsing.

Q31: How do I know if my requests.get(URL) call was successful?

A31: The response object has a .status_code attribute. A 200 status code (e.g., response.status_code == 200) indicates a successful request. Other codes (like 404 for “Not Found” or 500 for “Server Error”) indicate a problem.

Q32: What’s the difference between find() and find_all() in BeautifulSoup?

A32: find() returns the first matching tag or element it encounters, while find_all() returns a list containing all matching tags or elements. When you only need one specific element, find() is more efficient. When you need multiple (or all) occurrences, find_all() is necessary.

Q33: How do I extract an element’s attribute value (e.g., the href from an <a> tag)?

A33: You can use the .get() method. For example, if link_tag = soup.find('a'), then link_tag.get('href') would return the value of its href attribute.

Q34: What are “parsers” in the context of BeautifulSoup, and why are they needed?

A34: Parsers are software components that analyze and convert raw HTML or XML code into a structured, readable format (a parse tree) that Python can easily work with. BeautifulSoup needs a parser to understand the structure of the HTML content you provide. Common parsers include html.parser (Python’s built-in), lxml (very fast, requires installation), and html5lib (very robust, handles malformed HTML well, requires installation).

Q35: Can I modify the content of a webpage through BeautifulSoup?

A35: BeautifulSoup is primarily a parsing and extraction library. While it allows you to manipulate the parsed representation of the HTML in Python, these changes are not reflected on the live webpage itself. To make changes to a live website, you would typically need to interact with its backend or use browser automation tools.

Q36: What is a “BeautifulSoup object” (e.g., soup = BeautifulSoup(...))?

A36: The BeautifulSoup object is the main entry point for interacting with your parsed HTML. It represents the entire HTML document as a nested data structure (a parse tree), allowing you to easily navigate, search, and extract specific elements.

Q37: What is the text attribute of a BeautifulSoup tag object?

A37: The .text attribute returns all the text content contained within a specific HTML tag and its children, stripped of any HTML tags themselves. It’s useful for getting just the visible words from an element.

Q38: Can BeautifulSoup handle malformed HTML code?

A38: The ability to handle malformed HTML depends on the parser you choose. html.parser is decent, but html5lib is known for being extremely robust with messy HTML, as it tries to mimic how web browsers handle imperfect code. lxml is also good but might be stricter.

Q39: What is requests.get().text vs. requests.get().content?

A39: response.text gives you the content as a Unicode string, inferring the character encoding. response.content gives you the raw content as bytes. For BeautifulSoup, response.content is often preferred as it allows BeautifulSoup to handle the encoding detection itself.

Q40: How can I debug my BeautifulSoup code if it’s not finding elements correctly?

A40:

Print the soup object: print(soup.prettify()) will print the HTML in a nicely indented format, making it easier to see the structure.
Inspect the live page: Use your browser’s “Inspect Element” feature to verify the exact tags, classes, and IDs you are trying to target. HTML can be tricky, and what you see might not be exactly what the underlying code suggests.
Check for typos: Small errors in tag names, class names, or attribute spellings can prevent find() or find_all() from working.
Gradual search: Start by finding a broader parent element, then narrow your search down within that parent.

Q41: What are “attributes” in HTML tags (e.g., href, src, class, id)?

A41: Attributes provide additional information about an HTML element. They appear as name="value" pairs inside the opening tag (e.g., <a href="url">).

href: Specifies the URL for a link.
src: Specifies the source (URL) for an image.
class: Assigns a class name (or names) to an element, often used for styling with CSS.
id: Provides a unique identifier for an element within the page.

Q42: Can I get the value of an attribute (e.g., href) directly from a BeautifulSoup tag object?

A42: Yes, you can. If you have a tag object (e.g., link_tag), you can access its attributes like a dictionary: link_tag['href']. A safer way is to use link_tag.get('href'), which won’t raise an error if the attribute doesn’t exist (it will return None instead).