Here’s an FAQ based on the live tutorial:

Q1: What issues did I face initially with the AI agent?

A1: You faced several challenges:

• Token Exhaustion: Your AI agent quickly ran out of tokens, sometimes within 3-4 attempts.
• Code Execution Failures: The agent would generate Python code, but instead of executing it, it would often just return the code itself or pipe back errors.
• Gemini-Specific Problems: Gemini frequently generated incomplete code or code with syntax errors (e.g., missing triple quotes), making it unreliable.
• Ineffective Error Feedback: Attempting to correct the agent by feeding back the errors didn’t work; it would just make different mistakes in subsequent attempts.
• Lack of Generalizability: You expressed concern that without custom tooling, the agent struggles with diverse requests, requiring a lot of tokens for even simple tasks.

Q2: What is the proposed general strategy for web scraping and data extraction for analysis?

A2: The proposed strategy involves several steps:

1. Fetch Full HTML: Use Playwright to scrape the full HTML content of a webpage. This is crucial for dynamically loaded content that might not be available via simple HTTP requests.
2. Store HTML: Save the fetched HTML to a local file.
3. Identify Relevant DOM Structure: Use a dedicated tool (like dom_structure.py) to parse the HTML and identify the specific DOM structure (e.g., IDs, classes, or XPath selectors) of the content you need. This reduces the amount of information the AI needs to process.
4. Extract and Clean Data: Based on the identified DOM structure, extract the raw data and perform any necessary cleaning.
5. Answer Question: Finally, use the cleaned, relevant data to answer the user’s question.

Q3: Why is Playwright used for web scraping, and what are the considerations for its installation and deployment?

A3: Playwright is used because it can handle dynamic web content that loads in the browser’s Document Object Model (DOM), allowing it to effectively scrape pages that traditional requests modules might miss. It also supports running in headless mode, meaning it won’t pop up a visual browser window during execution.

• Installation: Playwright can be installed in a virtual environment (which is recommended for dependency management). I personally prefer Docker for even better isolation, though installing it directly in your system’s Python environment (e.g., WSL) is also possible.
• Deployment: While Playwright can be dockerized, it’s uncertain if deploying it with a headless browser on platforms like Vercel would cause specific issues. For now, my Docker setup is primarily for local testing.

Q4: Will the tutorial code be shared, and how were the files for the demonstration created?

A4: The code isn’t being shared yet because it’s currently in a “broken” (non-working) state (at the time of the Q&A). However, I will push it to GitHub once it’s fixed. Regarding the files used in the demonstration, some were pre-existing from previous sessions (indicated by no green status in Git), while others were created live during the tutorial (indicated by a green symbol for untracked files).

Q5: What is the dom_structure.py tool, and how does it help optimize AI agent performance?

A5: The dom_structure.py tool is designed to parse the HTML of a webpage and extract only the relevant structural information, such as element IDs, classes, or XPath selectors.

• Optimization: This significantly reduces the amount of data (tokens) that needs to be sent to the Language Model (LLM). Instead of feeding the entire, often verbose, raw HTML (which can be 26,000 tokens), the LLM receives a concise, structured representation (around 900 tokens). This helps the LLM focus on the specific elements needed for the task, improving efficiency and reducing token costs.

Q6: What IDE do you use, and do you recommend using Copilot?

A6: I primarily use VS Code. Yes, I do use Copilot; it helps a lot. I previously used Codium but uninstalled it because it caused too many issues.

Q7: How can I resolve an ImportError for ‘Gemini’ from ‘Google’ when using generativeai?

A7: The ImportError: cannot import name 'Gemini' from 'Google' is a known issue. A common solution is to adjust your import statement. You can often resolve this by pasting the specific error message into ChatGPT, which will likely suggest the correct package to import or the precise syntax for accessing the Gemini model via generativeai. The instructor confirmed that using generativeai directly should work.

Q8: What is the overall workflow for an AI agent performing web scraping and data analysis, and how does the code execution fit in?

A8: The workflow involves:

1. Breaking Down Tasks: The agent first breaks down complex requests into simpler, programmable steps.
2. Tool Selection: For each step, the agent decides which pre-defined “tool” (Python function) to use.
3. Code Execution:
   • If the task involves web scraping, it uses Playwright. The scraped HTML is processed by dom_structure.py to identify relevant parts.
   • If the task requires generating Python code (e.g., for data analysis or answering questions), the LLM generates the code. This generated code is then written to a temporary file, executed, and its output is captured.
4. Error Handling (Dynamic Code): If the dynamically generated code results in errors, that error feedback can be provided back to the LLM to help it correct its approach (though this has proven challenging).
5. Data Processing: Any relevant data is extracted and processed.
6. Final Response: After executing all necessary steps and tools, the agent compiles the information to provide the final response.

Q9: Can the code be run from other Python scripts, and how can I integrate the main.py functionality into another script?

A9: Yes, you can execute main.py from another Python script. You can use the subprocess module to run main.py as a separate process. For example:

import subprocess

subprocess.run(["uv", "run", "main.py"])

This allows you to execute the script and capture its output. You can also import specific functions from main.py directly if they are defined to be reusable.

Q10: Can you explain the dom_structure.py code in more detail, especially regarding its implementation and purpose?

A10: The dom_structure.py code is designed to create a structured, textual representation of a webpage’s DOM.

• Purpose: The goal is to provide the LLM with a concise summary of the webpage’s structure, allowing it to understand the layout and identify relevant elements without processing the entire raw HTML. This helps in targeting specific parts for data extraction.
• Implementation:
  1. Parsing HTML: It takes HTML content as input (e.g., from Playwright).
  2. Identifying Elements: It focuses on extracting key elements and their attributes like id, class, name, type, and potentially others that uniquely identify them. This includes HTML elements like <a>, <div>, <img>, <input>, <button>, etc.
  3. Structuring Output: The output is a formatted string (or JSON) representing the hierarchy and key attributes of these elements. For example, instead of a full HTML table, it might return something like: Table with ID 'data_table' and class 'results-grid' containing columns 'Name', 'Age', 'City'.
  • Optimization: By providing this filtered and structured information, the LLM receives less noise and more contextually relevant data, which significantly reduces token usage compared to processing the raw HTML. For instance, the token count dropped from ~26,000 to ~900 tokens in one example.

Q11: What is the current status of Gated Assignment 7, and why can’t I access it?

A11: The link for Gated Assignment 7 is currently not active because the assignment has not yet been officially released. It will become accessible after the Project 2 deadline.

Q12: I am having trouble with Python code execution due to the large amount of code. How can I manage this?

A12: If your Python code is very large (e.g., 26,000 lines) and causing execution issues, it often points to a need for modularization or a more efficient execution strategy.

• Modularization: Break down your large script into smaller, manageable functions or modules.
• Dynamic Code Execution: For code generated by the LLM, the current approach involves writing it to a temporary file and then executing that file. This is better than trying to run huge blocks of code directly in memory.
• Environment Management: Ensure your environment (especially when using Docker or virtual environments) is correctly set up with all necessary dependencies.

Q13: Why is the response.choices message appearing, and how can it be resolved?

A13: The response.choices error indicates a syntax issue with the code generated by the LLM, likely when trying to access parts of the response object.

• Verification: You should verify that the generated Python syntax for accessing the LLM’s response (e.g., response.choices[0].message.content) is correct according to the specific LLM API you are using.
• Correction: The LLM’s prompt can be refined to ensure it generates the correct syntax for accessing its own output. If response.choices is not part of the expected output structure, it means the LLM is misinterpreting how to extract information from its own response.