2025-03-04 Week 8 - Session 3 - TDS Jan 25#

2025-03-04 Week 8 - Session 3 - TDS Jan 25

Duration: 1h 52m

Here’s an FAQ-style summary of the TDS live tutorial:

Q1: Many GA questions (like GA1 Q13, GA2 Q7 for GitHub CLI; GA2 Q8 for local LLM/Docker; GA2 Q6 for Python API deployment; FastAPI servers) require tools/privileges (sudo access, local execution) that Vercel doesn’t provide. How can I complete these tasks? A1: I acknowledge your concerns, which are genuine and have been raised on Discourse. We will discuss these internally with Anand Sir and provide clarification. For now, please proceed with other questions that do not require these specific functionalities.

Q2: What about questions requiring specific external tools like Excel (Office 365 only), DevTools, or Google Colab-specific Python libraries (e.g., google.colab) that can’t be installed locally? A2: All tasks requiring specific tools like Excel, DevTools, or Colab can be solved using Python as well. You don’t need to use those specific platforms; just use Python.

Q3: For GA1 Q5, there’s an Excel query that can’t be run in Python, and LLM gives incorrect answers. How should I approach this? A3: This entire function (the Excel query) can be implemented using Pandas or basic Python libraries. You can also use a function-calling approach, sending the entire query to an LLM. Another student confirmed they successfully used the function-calling method.

Q4: For GA1 Q6, I need to use DevTools to find hidden elements in a post request where the question is passed as a parameter. How do I do this? A4: For questions requiring DevTools or console tab usage, you will receive an HTML file as input in the request. You need to apply your logic to identify the hidden element from this HTML file and extract the required value. You can then pass this HTML file to an LLM and ask it to find the hidden elements.

Q5: Will the dataset always be included in the request for questions that require it? A5: Yes, if a question requires a dataset, it will be provided within the request itself.

Q6: For GA2 Q4, I need to run code that requires Google Colab. How can I do this without local installation? A6: I’ve noted this specific question (GA2 Q4). I don’t have an immediate solution but will look into it and get back to you during our next session.

Q7: Is there a maximum latency time for Project 2’s responses, similar to Project 1’s 20-second limit? A7: That’s a good point. We will add this information regarding latency limits to the project documentation.

Q8: Will the question format in the request change, or will they be asked exactly as presented? A8: The core question will be presented in the exact same wording and language. What might change are the parameters within the question (e.g., “Wednesday” might become “Friday,” or a value like “61” might become “something else”). You can identify the specific question based on unique keywords.

Q9: Could you provide an example of how the request would be formatted for a question that includes a specific code block or extraneous information, like the one you just showed? A9: In the request, you won’t receive the entire verbose question, including extra details or redundant code blocks (like videos or extraneous text). You will only receive the actual, concise question text. Any necessary data files (e.g., a PDF) will be sent as a separate part of the request.

Q10: What approaches can I use to identify questions and handle varying parameters, especially given the conciseness of the request format? A10: You can use two main approaches: 1. Function Calling: If you have specific functions designed for each question, you can call the appropriate one. This is potentially useful for up to 50 questions (10 questions per GA x 5 GAs). 2. String Searching: Match unique identifiers or keywords within the question. For example, “marks” or “student who scored” could be unique identifiers. Parameters like subjects (e.g., “Math” vs. “English”) or values (e.g., “61”) might change, so your solution should be able to identify these parameters. You can combine both approaches. If string searching isn’t sufficient or there’s nothing unique, fall back to function calling.

Q11: Will the question parameter be a plain text or JSON format? A11: The question parameter will be in plain text format.

Q12: For GA2 Q2, which involves images: 1. How will the image (containing shapes) be sent in the request? 2. The response must be a JSON object. Can I embed the image as Base64, a URL, or a data URI within the JSON, or should I send a binary image? A12: 1. The image will be sent as an attachment (a binary file) in the request. 2. You will need to respond with a binary image file itself, not a Base64 encoded string, URL, or data URI within the JSON.

Q13: How will deployment questions (e.g., deploying a Python API to Vercel) be handled? A13: Deployment questions will be discussed in an internal meeting. After that, I can provide clarity on whether deployment to Vercel is actually possible and how the application would work.

Q14: If deployment questions are possible, should the answer be a link to the deployed application? A14: Yes, if it’s a Vercel deployment question, your answer should return a link to the deployed application, as that’s what you’re submitting for evaluation.

Q15: For deployment questions, will the evaluation check if the provided link is functional, or just its format? A15: The evaluation will send a request to the link you provide to verify it. If the expected data isn’t retrieved, it will indicate that the data doesn’t exist or isn’t correct.

Q16: Can I temporarily skip questions related to GitHub, deployment, external installations (like Docker, Ngrok, Google Colab), or running multiple FastAPI servers? A16: Yes, for now, you can skip questions related to GitHub, deployment, external installations (like Docker, Ngrok, Google Colab), or running multiple FastAPI servers. We will address how these will be handled later.

Q17: If certain tasks (like running multiple FastAPI servers or using a vector database) are not feasible on Vercel, will an alternative solution be provided? A17: Yes, if something isn’t possible on Vercel, we will definitely come up with an alternative solution and discuss it.

Q18: Project 2’s post requests don’t have an API token parameter, so I’m using a team-provided API token. What happens if my API token credits run out during evaluation? A18: We have the ability to increase API token limits for individual students. If such a situation arises, it will be addressed. Also, your API tokens are renewed every month, so you shouldn’t run out easily. We will take care of it during evaluation if needed.

Q19: I’m planning to use OpenAI’s text embeddings, store them in a vector database, and then use similarity search to match incoming questions with the most similar pre-embedded questions for function calling. Is this a good approach, and is it feasible on Vercel (e.g., running a vector database)? A19: Yes, that’s a really good approach. You can keep embeddings ready for each question, perform matrix multiplications, and identify the question. You can store embeddings in CSVs or NumPy arrays, so a dedicated vector database might not be strictly necessary. I’m not sure if Vercel supports running a vector database directly, but storing in CSVs or NumPy arrays are viable alternatives.

Additional points discussed:

  • Next Session: The next session is tentatively scheduled for Tuesday.
  • Instructor’s Posts: The instructor confirmed they have read all posts on Discourse regarding these issues and will address them in an internal meeting.
  • Python Visualization (by Sharansh): The session transitions to a demonstration of creating visualizations in Python using libraries like Seaborn and Matplotlib. It covers scatter plots, correlation matrices, and heatmaps.
    • Seaborn/Matplotlib: Seaborn is an upgraded version of Matplotlib, offering more attractive, modern-looking charts. Matplotlib creates generic graphs. Seaborn can use Matplotlib as a backend.
    • Data: The iris dataset is used, which contains 150 rows of information about three types of flowers, with four features (sepal length, sepal width, petal length, petal width).
    • Scatter Plot (sns.scatterplot): Demonstrated using sepal_length vs. petal_length. The hue parameter can be used to color data points based on a categorical variable (e.g., flower type).
    • Color Palettes: Seaborn offers various color palettes. The palette (for scatter plots) or cmap (for heatmaps) parameter can be used to change colors (e.g., ‘reds’, ‘magma’). Specific palettes like ‘magma’ are good for contrasting colors.
    • Correlation Matrix/Heatmap (sns.heatmap): Demonstrated creating a correlation matrix to show relationships between features (e.g., negative correlation between sepal length and sepal width, suggesting longer sepals tend to be thinner). A heatmap visually represents this, with darker colors for lower correlation and lighter for higher (or vice versa depending on cmap). The annot=True parameter displays correlation values on the heatmap.
  • Database Integration (by Jivraj): The session then covers database integration, specifically using Python to connect to MySQL and execute SQL queries.
    • Date Time Module: Python’s datetime module (e.g., datetime.strftime) can be used to parse and format dates.
    • Pandas Integration: Once data is fetched from a database or file, Pandas DataFrames are used for manipulation.
    • DuckDB: A fast, open-source analytical database. It’s faster than Pandas for certain operations, especially when dealing with large datasets or complex queries, and can handle various file formats (CSV, JSON, Parquet).
    • SQLAlchemy/PyMySQL: These libraries facilitate connecting Python to SQL databases (e.g., MySQL, PostgreSQL). You can define connection strings (username, password, host, port, database name) and execute SQL queries.
    • Folium: A Python library for creating interactive geographical maps. You can add markers, customize icons, and display data on maps (e.g., showing shop locations and potential exploration areas).
  • NetworkX: A Python library for creating and analyzing network graphs.
    • Graph Creation: You can add nodes (elements) and edges (connections) between them.
    • Customization: Nodes and edges can be customized (e.g., colors, labels, icons).
    • Applications: Useful for visualizing relationships between items (e.g., product bundles in market basket analysis).
  • Clustering: A machine learning technique to group similar data points together.
    • Classification vs. Clustering: Classification aims to categorize data into predefined classes, while clustering groups data based on inherent similarities without prior labels.
    • Applications: Used to analyze natural clusters in data (e.g., identifying distinct customer segments).

(Note: The recording abruptly ends during the discussion on NetworkX and Clustering.)