2025 06 11 Project 1 Session 2 - TDS May 2025#

Duration: 2h 59m

Here’s an FAQ based on the provided transcript of the TDS live tutorial:

Q1: I’m experiencing audio issues, and it sounds like yours might be breaking up too. Is my audio audible, and can I share my screen if needed?

A1: Yes, I can hear you, though there might be some audio issues on both our ends. For now, let’s proceed with the session.

Q2: Can you give me an overview of today’s project agenda?

A2: Today, we’ll focus on creating a RAG (Retrieval-Augmented Generation) system. This involves converting data into Markdown, processing images into text descriptions using LLMs, performing chunking, storing embeddings efficiently, and finally, deploying and testing the application.

Q3: Where does the data for this project come from?

A3: We’re using the Tools in Data Science (TDS) guide content, which includes 12 weeks of material and images. You can get this data directly from the public GitHub repository.

Q4: When extracting data using the API, should I create one JSON file per thread (topic)?

A4: Yes, your understanding is correct. You should create one JSON file for each thread or topic.

Q5: My scraped data contains mixed content (text, image URLs, etc.). Should I separate these, or is it okay to keep them mixed when creating JSON files?

A5: While it’s okay to have mixed content initially, the best approach for performance and accuracy is to convert all data (text, URLs, etc.) into a clean Markdown format. Then, specifically for images, you should extract them and replace their URLs with an LLM-generated text description within the Markdown file.

Q6: When dealing with images, I shouldn’t directly embed them, but rather ask an LLM to describe the image and then embed that description. Is that right?

A6: Absolutely, that’s correct and a crucial “hot tip” we haven’t shared elsewhere. Directly embedding images is inefficient and costly. Instead, use an LLM to generate a detailed text description of the image, and then embed that description. This significantly improves accuracy and reduces embedding costs.

Q7: When you say “ask the LLM to read the image,” do you mean using OCR, or is it a different process?

A7: No, not OCR. We’re asking the LLM to describe the image carefully. This requires careful prompt engineering to get a high-quality, meaningful description. Test your prompts independently to ensure they accurately describe the images.

Q8: What specific Python libraries or tools are you using in the demo for processing HTML and images?

A8: We’re using html2text for converting HTML to Markdown, BeautifulSoup for parsing HTML documents and locating elements, and the Gemini API for generating image descriptions. numpy.savez is used for storing embeddings.

Q9: How do I get and use the Gemini API key, and is it free?

A9: You can obtain your Gemini API key from AI Studio. It’s free, but comes with rate limits: 500 requests per day and 10 requests per minute for the Flash 2.0 model, which is what we’re using. If you pay, you get more.

Q10: How do you handle API rate limiting when making many calls to Gemini, especially when generating embeddings at high speed?

A10: We implement an exponential backoff strategy using a RateLimiter function. If a 429 “Too Many Requests” error occurs, it waits for 1 second on the first retry, 4 seconds on the second, and 8 seconds on the third. We allow a maximum of three retries before stopping.

Q11: Why is chunking necessary, and what’s the best strategy for it?

A11: Chunking is crucial because LLMs have token limits for embeddings. You can’t embed an entire document at once. There’s no single “best” strategy; it’s an “art” that requires experimentation. A good approach is to have slight overlaps between chunks (e.g., include a sentence or two from the previous chunk in the next one) to maintain context. You’ll need to empirically test different chunking strategies to find what works best for your project.

Q12: How are the generated embeddings stored?

A12: We store all embeddings in a compressed NumPy array archive (.npz file) using numpy.savez. This keeps the file size small (e.g., 15MB for the entire project) and is very efficient for our use case.

Q13: How will we access these stored embeddings for retrieval?

A13: We will retrieve the embeddings from the NumPy archive. When a user sends a query, we’ll convert that query into an embedding, then use cosine similarity to find the most relevant embeddings (e.g., top 5 or 10) from our stored archive. These relevant embeddings, along with their associated content, will form the context for the LLM’s response.

Q14: What are the recommended options for deploying our application?

A14: We’ll demonstrate three deployment methods:

1. Vercel: Always-on, but requires a serverless function setup which can be slightly complex.
2. Render: Easy setup, similar to local development, but servers go inactive after a few minutes, leading to a slow first response.
3. Hugging Face: High-performance and free, using Docker containers. It’s a bit more complex to set up due to Dockerization, but very powerful and popular for LLM applications.

Q15: What are the requirements for the final application, and how will it be tested?

A15: Your application should be accessible worldwide via a URL endpoint (which you’ll provide to us). We’ll test it using PromptFoo with a set of 10 questions. Each question has two assertion types: an LLM rubric test (checking contextual correctness) and a “contains” test (checking for exact text matches). Your application needs to respond within 30 seconds per question.

Q16: What tools should I use for testing, and how should I create test cases?

A16: We’ll be using PromptFoo for testing. You can refer to the “LLM Evals” video in the Tools in Data Science course (Week 3) for guidance. Create your own test cases with relevant questions and expected assertions. We strongly recommend watching that video to understand how to build robust test cases.

Q17: How can I access the necessary Python files (.py) for the demo and the Markdown files from the repository?

A17: You can use git clone with the GitHub CLI to clone the entire repository. This will recreate the folder structure on your local machine. Once cloned, you can use Python’s os.walk to iterate through the directories and access the Markdown and other .py files.

Q18: I’m not from a coding background, and I’m worried about completing this project. Is it truly possible, and what should I focus on?

A18: Yes, it is absolutely possible! We’ve designed these sessions to provide all the knowledge and tools you need. Focus on understanding the core concepts – data acquisition, chunking, embedding, and application deployment. These sessions string together the individual components you’ve learned. You have all the ingredients; just apply them. Your non-coding background is not a barrier; it’s an opportunity to learn a valuable skill. We’re confident you can do it.

Q19: What’s the bonus question about, regarding the final submission?

A19: The bonus question is for a script that scrapes Discourse posts, not just uses the API. If your GitHub repository includes a script that successfully scrapes Discourse posts (rather than just fetching JSON via API), you can earn a bonus mark.

Q20: When will the recordings of these sessions be available?

A20: The availability of recordings depends on Google’s processing time. For example, yesterday’s recording was available around 3 AM. We then do our own processing and upload it. So, recordings are typically available the morning after the session.

Q21: How will you test my deployed application?

A21: We’ll use our own API keys for testing. You need to provide us with the URL endpoint for your deployed application. We will then send PromptFoo tests (using NPX promptfoo run -f <YAML_FILE> -u <YOUR_APP_URL>) to your application’s endpoint. We will be using our own YAML test files, not yours.

Q22: I’m worried about the project deadline. Is it possible to get an extension until November 20th?

A22: At the moment, we cannot make any promises regarding an extension. The assumption is that by the end of day 14 (November 14th), most students, especially if they have watched all three sessions, should be able to complete the project. We still feel it’s doable within the current timeframe, and we expect you to have all the necessary knowledge and tools. However, we understand challenges can arise, and we appreciate your dedication.

Q23: I’m facing a hard deadline for the project. Is it really possible to finish in two days, especially since I’m lagging on LLMs?

A23: Yes, it is absolutely doable. We expect you to have already done the data scraping from yesterday’s session. Today, we showed you how to chunk and embed that data. Tomorrow, we will create the application itself. If you follow tomorrow’s session, you will be able to complete the project. Your focus should be on getting the data, chunking, and embedding it. The rest is application development, which we’ll cover tomorrow.

Q24: For the testing on PromptFoo, can I just show you my test setup directly from my system?

A24: No, your application needs to be accessible worldwide. You will provide us with the URL endpoint of your deployed application (accessible over the internet), and we will run our tests against that URL. We won’t test directly on your local system.

Q25: I’m just getting a little stressed about how to complete this project in two days. How can I manage?

A25: I understand it can feel stressful, but remember these sessions are designed to guide you. Try to absorb what we’ve covered today, and you’ll have something ready for tomorrow. You’ll have time to ask questions, and we’ll show you the application development. It’s a marathon, not a sprint. We’re here to help.

Q26: What’s the best way to get all the required project files from the GitHub repository?

A26: The most efficient way is to use git clone with the GitHub CLI. This command (git clone <repo_url>) will replicate the entire repository structure, including all files and subdirectories, directly to your local machine.

Q27: Can I use any platform for deployment, like Railway, or do I have to stick to the ones you mentioned?

A27: You can use any platform you prefer (Railway, Vercel, Render, Hugging Face, or even your own on-premise server) as long as your application is accessible worldwide via a URL endpoint. However, if you use a platform not explicitly covered, make sure to include all necessary configuration files (like vercel.json or Railway’s configuration) in your GitHub repository, so we can replicate your deployment setup if needed. We need to be able to access and test your application without issues.

Q28: Is it better to test my deployed application locally, or should I involve others?

A28: It’s crucial to test your application from outside your local network. Don’t just test it locally. Ask a friend (even someone in a different city or country) to test it using the PromptFoo command you provide. This ensures your application is truly accessible globally and performs as expected from different locations.

Q29: When testing, you mentioned using a YML file. What does that file contain, and do I need to create it?

A29: The YML file contains the test cases that PromptFoo will run against your application. It specifies the questions, expected responses, and the type of assertions (e.g., LLM rubric, ‘contains’) for each test. We will provide our own YML test files for evaluation, so you don’t need to create them.

Q30: I’m confused about the URL in the YML file. Do I need to put my local host URL, or a deployed one?

A30: You need to put the URL of your deployed application in the YML file, not your local host. This is the URL that makes your application accessible worldwide. We will use this URL to send our test requests.

Q31: For the test results, it shows a duration of 13 seconds. Is there a time limit per question?

A31: Yes, there is a time limit. Each question in the PromptFoo tests must respond within 30 seconds. Your application successfully completed all three questions in 13 seconds, which is excellent. If a response takes longer than 30 seconds, that test will fail.