A session 23 June : Web scraping with cookies and understanding JSON#

A session 23 June : Web scraping with cookies and understanding JSON

Duration: 1h 57m

Here’s an FAQ based on the live tutorial:

Q1: I’m trying to open a specific badge page on the Discourse platform. It works fine in my regular browser, but when I open it in an Incognito window, it just shows a login page. Why is this happening?

A1: When you try to access the badge page in Incognito mode, you’re not logged in, which means you’re not authenticated. The website requires you to be logged in to view that specific page, so it redirects you to the login screen instead of showing the content.

Q2: How does the website “know” if I’m logged in, and where can I find this information in my browser?

A2: Websites use “cookies” to manage your login session. When you first log in, the server sends your browser a cookie, which is a small piece of information containing your authentication details. Your browser then sends this cookie back with every subsequent request, authenticating you without needing to log in repeatedly.

You can inspect these cookies in your browser’s developer tools:

  1. Right-click on the page and select “Inspect” (or “Inspect Element” / “Developer Tools”).
  2. Go to the “Network” tab.
  3. Refresh the page. You’ll see a list of network requests.
  4. Click on any of the requests (e.g., the main page request).
  5. In the details panel, look for the “Headers” section and scroll down. You’ll find “Cookies” listed there.
  6. Alternatively, some browsers also list them under the “Application” tab (look for “Cookies” on the left panel).

Q3: What exactly are cookies, and why are they important for scraping?

A3: Cookies are small pieces of data that a server sends to a user’s web browser. The browser may store it and send it back with later requests. They’re essential for maintaining “state” on the web, like remembering your login session. For scraping, if you need to access personalized or restricted content on a website (like your profile page on Movielens or the Discourse badge page), you need to provide the correct authentication. Cookies contain this authentication information, allowing your scraping script to access content as a logged-in user.

Q4: Can you explain APIs using an analogy, and how they relate to cookies?

A4: Think of a restaurant. You (your application) want a specific dish (data). The chef (the server/database) knows all the recipes (all the data) but won’t give you the secret recipe. Instead, you interact with a waiter (the API). You tell the waiter what you want from the menu (a specific type of data), and they bring you the dish. The waiter acts as an intermediary, giving you the prepared data without exposing the full backend or sensitive information. Cookies are like your ID or reservation that you give to the waiter to prove you’re a valid customer allowed to make requests.

Q5: Are cookies stored permanently, and why do websites ask for cookie consent nowadays?

A5: Cookies are not always stored permanently; sometimes they are, sometimes they are session-based and expire when you close your browser. Generally, they are stored in your browser. Websites ask for cookie consent due to European regulations (like GDPR) that require explicit user permission for data collection and tracking. It’s good practice not to accept cookies if they’re not necessary for the website’s core functionality.

Q6: I’m trying to open the personalized Movielens homepage. It only shows a generic page without cookies. After logging in, the page shows customized content. I tried to use the original Movielens URL in my Python script without cookies, and it still got the generic page. How do I get the personalized content?

A6: The personalized content (e.g., your Movielens homepage after login) is only accessible when your request is authenticated. When you try to fetch movielens.org/home without providing cookies, the server treats you as an unauthenticated user and sends the generic page. To get the personalized content, you need to include the authentication cookies in your Python requests.get() call.

Q7: How do I include cookies in my Python requests.get() call, and how can I extract data from the JSON response?

A7:

  1. Extract Cookies from Browser:

    • Log in to the Movielens website.
    • Open Developer Tools (Inspect Element).
    • Go to the “Network” tab, refresh the page, and click on the main page request.
    • Find the “Cookies” section in the headers. You’ll see a list of cookie_name: cookie_value pairs.
    • Copy all these key-value pairs.

  2. Use Cookies in Python:

    import requests
import json  # For JSON responses
from bs4 import BeautifulSoup  # For HTML parsing

URL = "https://movielens.org/home"  # Or the specific API URL

# Create a Python dictionary of your cookies
my_cookies = {
    "cookie_name_1": "cookie_value_1",
    "cookie_name_2": "cookie_value_2",
    # ... and so on for all cookies
}

# Make the request including the cookies
response = requests.get(URL, cookies=my_cookies)
print(response.status_code)  # Should be 200 for success

# To extract data from a JSON response (like for API calls):
data = json.loads(response.text)  # Converts JSON string to Python dict
# Now you can navigate the dictionary using keys, e.g.,
# print(data['data']['email']) # To get your email from the 'me' API call

# To extract data from an HTML response (like the full page):
# soup = BeautifulSoup(response.text, 'html.parser')
# print(soup.prettify())

    You navigate the JSON dictionary by continuously accessing nested keys until you reach the desired value.

Q8: I’m trying to scrape specific tags from a movie page on Movielens (e.g., Dark Knight). How can I do this, especially considering there are many tags, and I need to compare them across movies?

A8:

  1. Use API Calls for Tags:
    • Navigate to the specific movie page on Movielens while logged in.
    • Open Developer Tools and go to the “Network” tab.
    • Refresh the page and look for an API call related to “tags” (you might see URLs like /api/movies/{movie_id}/tags).
    • Copy this API URL. This API call usually returns a clean JSON response containing the tags and their associated scores/details.
  2. Extract Tags from JSON:
    • Use the requests.get() method with your cookies to call this tags API URL.
    • Use json.loads() to convert the JSON response into a Python dictionary.
    • Parse the resulting dictionary to extract the tag names and their associated counts/scores (e.g., data['tags'][0]['tag'], data['tags'][0]['score']).
  3. Compare Tags (Bonus Question):
    • Perform steps 1 & 2 for both movies you want to compare.
    • You’ll have two lists of tags (or dictionaries of tags and scores).
    • Use Python list/dictionary comprehensions or loops to find common tags and identify the top 5 common ones based on their scores. This method is much cleaner than parsing complex HTML for tags.

Q9: What if directly fetching the API URL with cookies still gives me an “Outdated browser” error, or the JSON is not structured as expected?

A9:

  1. User-Agent Header: If you get an “Outdated browser” error, the server might be rejecting your Python requests default User-Agent. You can find your browser’s User-Agent in Inspect Element -> Network -> Request Headers. Copy it and add it to your requests.get() call:
    headers = {"User-Agent": "Your_Browser_User_Agent_String"}
response = requests.get(URL, cookies=my_cookies, headers=headers)
  2. Downloaded HTML as Fallback: If API calls are consistently problematic (e.g., blocked or very complex authentication), a last resort is to download the entire HTML page as a static file:
    • In your browser, go to the page.
    • Right-click and select “Save page as…” (save as HTML complete).
    • In Python, read this local HTML file into BeautifulSoup:
    with open("downloaded_page.html", "r", encoding="utf-8") as f:
    html_content = f.read()
soup = BeautifulSoup(html_content, "html.parser")
    Then, you can use BeautifulSoup methods (like find_all()) to extract data from this static HTML. Note that for dynamic content that loads after the initial page, this method won’t capture everything.

Q10: I found the tags using BeautifulSoup on the Movielens home page, but the output was very small and didn’t contain all the tags. Why is this?

A10: The Movielens home page’s HTML, when fetched directly or downloaded, might not contain the full dynamic content, including all tags. Many websites load content dynamically using JavaScript after the initial HTML is served. This means the BeautifulSoup object created from the initial HTML won’t have the full picture. For tags, it’s better to use the specific API calls as described in Q8.

Q11: I’m still facing issues with the API calls not working in my Python script, even with cookies. What could be wrong?

A11: The main issue is likely related to the User-Agent header or other specific headers the server expects. You can copy the entire Request Headers dictionary (not just the cookies) from your browser’s developer tools (Network tab, click on the specific API request, scroll down to Request Headers). Then, pass this entire dictionary in your Python requests.get() call:

headers = {
    # Copy all key-value pairs from your browser's Request Headers here
    "Accept": "application/json, text/plain, */*",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
    # ... etc.
}
response = requests.get(API_URL, cookies=my_cookies, headers=headers)

This ensures your Python request closely mimics a request coming from a legitimate browser, which often resolves authentication issues. Remember to be very careful with selectors.

Q12: I need to scrape the tags for a movie, but the website is very dynamic, and I’m having trouble. What’s the best approach?

A12:

  1. API Calls (Preferred): The most effective way for dynamic content is usually through API calls, as they provide clean JSON data. Find the specific API URL for movie tags (as discussed in Q8).
  2. Authentication: Ensure you’re including your cookies (and potentially full headers) to authenticate your API requests (refer to Q7 and Q11).
  3. JSON Parsing: Once you get a successful JSON response, use json.loads() to convert it to a Python dictionary and then access the tag information using the appropriate keys.
  4. HTML Download (Fallback): If API calls remain problematic, download the full HTML of the movie page (File -> Save page as… in your browser). Then, use BeautifulSoup to parse this static HTML file. You’ll need to carefully inspect the HTML structure to find the specific div or span elements containing the tags.
    with open("movie_page.html", "r", encoding="utf-8") as f:
    html_content = f.read()
soup = BeautifulSoup(html_content, "html.parser")
tags = soup.find_all("a", class_="tag")  # Example selector, inspect your HTML
for tag in tags:
    print(tag.text)