2025 06 12 Project 1 Session 3 - TDS May 2025#

Here’s an FAQ based on the provided transcription:

Q1: What is this session about?

A1: This session is a live tutorial part of the “Tools in Data Science” (TDS) course. We’re creating an API endpoint that takes a question and an image as input and responds with an answer and its sources.

Q2: How do I ask questions during the session?

A2: If you have any questions at any point, just stop me and ask. I won’t be able to see if you raise your hand.

Q3: What Python libraries are we using for this project?

A3: We’re using markdownify to convert markdown to HTML, html2text to convert HTML to text, and pydantic for verifying the incoming image and question data.

Q4: Why are we reusing functions from yesterday’s session?

A4: We’re reusing functions from yesterday’s session (when we created the embedding store) because when you send a question to the server, the server needs to embed that question. Similarly, for the image, we convert it to a description and then embed that description. This allows us to find similarities between the question/image and our RAG content.

Q5: How do we handle image inputs that are base64 strings from the API, rather than file paths?

A5: Our embedding functions initially expected local file paths. Since images will come as base64 strings via the API, we need to convert these strings into actual image files (e.g., JPG, PNG) first. We’ll assume JPG for now, but you can identify the correct format using the image’s mimetype.

Q6: How do we use the image description in the question-answering process?

A6: The description we get for the image is appended directly to the user’s question. This means the combined “question” now contains both the actual user query and the image description. We then generate embeddings for this combined text.

Q7: How do we find relevant information to answer the question?

A7: After embedding the combined question and image description, we calculate the cosine similarity between this query embedding and all the stored text chunks. We then retrieve the top 10 most similar chunks, which will serve as the context for generating the answer.

Q8: How does the system generate the final answer?

A8: We create another prompt using the top 10 similar chunks and the user’s question. This prompt is sent to an LLM (specifically gemini-pro, using 2.0/flash which allows 30 requests per minute) to generate a concise answer. We also include a system_prompt that can be optimized for better results. The LLM’s text response is what we’ll return.

Q9: Will the response include the source of the information?

A9: Currently, the npz file that stores the embeddings doesn’t include the source URLs. For your project, it’s crucial to store the URL of the original document for each chunk. When retrieving a chunk, you should also retrieve its corresponding URL. This way, the API response can include both the concise answer and a reference link to the source.

Q10: What’s the best practice for chunking documents to avoid breaking down content (like URLs)?

A10: Chunking is indeed an art. A good practice is to make sure your chunks have a few lines of overlap with the previous chunk. This helps ensure continuity if a URL or important piece of information is split. Another approach is to ensure that one markdown file contains an entire post/document, and its reference URL is kept at the top of that markdown file. When you retrieve the chunk, you also retrieve that reference. You’ll have to figure out a method that works best for your specific use case.

Q11: How do we perform prompt engineering for this project?

A11: Prompt engineering is very relevant and impacts performance significantly. It’s an art, but a big tip is to use the LLM itself as a tool for prompt engineering. Explain to the LLM what you want to achieve and ask it to generate a good prompt for your specific task (e.g., “Construct me a good prompt using good prompt engineering practices. This is what I am trying to achieve. Please give me a good prompt for this project.”). You can even provide existing good prompt engineering practices as context.

Q12: How do we deploy the application to Vercel?

A12: To deploy to Vercel, you need three key things:

vercel.json file: This file configures how Vercel builds and serves your application. Make sure the src path correctly points to your main application file (e.g., api/answer.py).
requirements.txt file: This lists all the Python dependencies for your project.
Environment Variables: You must export your API keys (e.g., Google API Key) as environment variables on Vercel, not hardcode them directly in your code or GitHub repo. Use the command vercel env add <KEY_NAME> and provide the value off-stream.
Vercel operates as a serverless platform, so it doesn’t run your application top-to-bottom like a traditional Python script. When an endpoint is hit, it directly triggers the specified function in your vercel.json (e.g., answer.py) and runs only that block of code. It ignores other parts of your code outside that function.

Q13: How do we deploy the application to Render?

A13: Deploying to Render is very straightforward, similar to running it on your local machine:

Create a New Web Service: Connect Render to your GitHub repository.
Build Command: Set the build command to pip install uv (UV is a fast Python package installer) or pip install -r requirements.txt. UV will handle all dependencies.
Start Command: Set the start command to uv run answer.py (assuming answer.py is your main application file).
Environment Variables: Add your necessary environment variables (like your Google API Key) directly in Render’s configuration.
Port Configuration: Vercel by default uses a specific port for deployment, so in answer.py, make sure your application listens on a dynamic port, like port=os.environ.get("PORT", 8000).

Render works exactly like your local machine, so if your code runs locally, it will run on Render without further modification. However, Render has some disadvantages: it’s slower to start up (it goes into an inactive state after a period of inactivity, taking about a minute or two to “spin up” when a request comes in) and its compute power is lower than other platforms.

Q14: How do we test the deployed application?

A14: We can test the deployed application using curl commands. We’ll use a POST request to the deployed endpoint (e.g., https://your-vercel-app.vercel.app/api/answer) and provide the question and image data in the request body. If successful, it should return a valid response from the LLM.

Q15: How do we configure Promptfo to use Gemini Pro instead of OpenAI?

A15: By default, Promptfo uses OpenAI’s API. To switch to Gemini Pro, you need to configure your promptfo YAML file. You must explicitly set the provider to google/gemini-pro. Additionally, ensure your Google API key is correctly configured as an environment variable, not directly in your code. You can find detailed instructions and code examples in Gemini’s documentation or by asking ChatGPT/other LLMs for the correct YAML configuration.

Q16: I’m getting a 401 Unauthorized API error even after setting my Google API key. What could be wrong?

A16: If you’re getting a 401 error, it indicates an issue with your API key or how it’s being used. It’s possible your Promptfo configuration is still trying to use OpenAI by default, despite your Google API key being set. You need to explicitly configure Promptfo to use google/gemini-pro as the provider in your YAML file. Also, ensure the API key environment variable is correctly exported and named as expected by Promptfo.

Q17: Will the deployed application on Vercel/Render include the full GitHub repository, or just the necessary files?

A17: For Vercel/Render, you only need to push the essential files for your application to run. This typically includes:

answer.py (your main application file)
embeddings.npz (your vector database)
requirements.txt (your dependencies)
vercel.json (for Vercel configuration)
promptfo.yaml (for Promptfo testing, if applicable)

Other development files or markdown folders are not strictly necessary for the deployment itself.

Q18: What is vercel.json and requirements.txt used for?

A18:

vercel.json: This file is a configuration file for Vercel that tells it how to build and deploy your application. It specifies the source files, output directory, and any other build-related settings.
requirements.txt: This file lists all the Python packages your application depends on. Vercel (and Render) will automatically install these packages during the deployment process to ensure your application has all the necessary libraries to run.

Q19: How should I put my API keys for Vercel/Render? Should I use .env files?

A19: It’s best practice not to commit .env files directly to your GitHub repository or embed API keys in your code. For Vercel/Render, you should export your API keys as environment variables directly in their respective platforms’ settings. For Vercel, use the vercel env add <KEY_NAME> command. This keeps your sensitive information secure and separate from your codebase.

Q20: What’s the main advantage of using Vercel or Render for deployment?

A20:

Vercel: Offers 100% (or nearly 100%) uptime and immediate response once “spun up”. It’s excellent for production applications requiring low latency. It’s a serverless application, so it only runs the specific function triggered by an endpoint.
Render: Works very similarly to a local machine, making deployment super easy if your code already runs locally. It also supports Docker for containerization. Both platforms simplify the deployment process significantly compared to traditional methods.

Q21: Are there any disadvantages to using Render?

A21: Yes, Render has a couple of downsides:

Spin-up Time: After a period of inactivity, your application on Render might go into an “inactive” state. The first request to the endpoint will then take some time (typically around 1-2 minutes for a lightweight app like this) to “spin up” the application again before responding.
Compute Power: Render’s free tier offers relatively low compute power, which might not be suitable for very heavy applications, although it works fine for this project.

Q22: Will we be covering testing the application using Promptfo?

A22: Yes, we will cover how to test the application using Promptfo, including configuring it correctly for different LLM providers and setting up YAML test files. This will happen towards the end of this session or in the next one.

Q23: (Follow-up) I’m using Gemini, and my Promptfo test gives an OpenAI error (401 Unauthorized). What does this mean?

A23: This indicates that your Promptfo configuration is still trying to use OpenAI’s API by default, even if you’ve set up your Google API key. You need to explicitly configure your promptfo.yaml file to use google/gemini-pro as the LLM provider. Promptfo’s documentation has specific instructions on how to override the default provider.

Q24: What is the purpose of using UV (uv run answer.py) in the deployment commands instead of just python answer.py?

A24: UV is a new, very fast Python package installer and executor. When used in deployment commands, uv run answer.py will automatically handle installing all your project dependencies (as listed in requirements.txt) and then run your application, similar to how pip install -r requirements.txt && python answer.py would work, but often much faster.