2025 05 24 Week 3 Session 1 - TDS May 2025#

Duration: 6430.0Here’s an FAQ summary of the live tutorial:

Q1: What is the main focus of Module 3 in this course?

A1: Module 3 is about interacting with a specialized API. If you’ve completed weeks 1 and 2, you’ll find it manageable as it builds upon those API concepts. Today’s session will focus more on conceptual understanding than extensive coding demonstrations.

Q2: What is the first thing I should do for Module 3, and what’s a common mistake to avoid?

A2: First, ensure you get your API key from ai-pipe.org. A common mistake is using Open AI’s API key or URL directly. This course uses a proxy, so you need to use the specific API key and URLs provided in the ai-pipe.org documentation. Using the wrong one will result in errors like “ran out of credits” or “wrong key.”

Q3: How do Large Language Models (LLMs) fundamentally work?

A3: LLMs are not sentient; they are complex mathematical models leveraging “transformers.” When you provide a prompt, it’s broken down into “tokens” (which might not be full words). Each token is represented as a high-dimensional vector (e.g., 1600 features). These vectors are stored in a vast matrix. LLMs then perform matrix operations, analyzing the distances between these vectors to understand semantic relationships between tokens and generate responses.

Q4: What is “self-attention” in the context of LLMs?

A4: Self-attention allows an LLM to understand what specific words (or tokens) refer to within a given sentence or text. For example, in the sentence “This animal was too tired to cross the road because it was tired,” self-attention helps the LLM determine whether “it” refers to “this animal” or “the road” by calculating probabilities based on context. This is crucial for coherent and contextually relevant responses.

Q5: What is a major limitation of LLMs, and how does it manifest?

A5: A significant limitation of LLMs is their limited “context window.” While humans can maintain context for decades, hours, or even years, LLMs can only remember a relatively small amount of prior interaction (e.g., initially 128,000 tokens, now up to 1 million tokens in newer models). This means after prolonged interaction, LLMs can “forget” the earlier parts of a conversation, leading to “hallucinations” or nonsensical suggestions because they’ve lost the necessary context.

Q6: How can we overcome the limited context window of LLMs and enhance their knowledge with custom data?

A6:

1. Embeddings: Text (and even images/audio in “multimodal embeddings”) can be converted into numerical vectors. You can generate embeddings for your own company’s specific data, bringing it into the LLM’s understanding. Tools like Hugging Face offer models for this, or you can use Open AI’s API.
2. Vector Databases: For large amounts of embeddings, you store them in a vector database (e.g., DuckDB, PostgreSQL, SQLite, LanceDB, ChromaDB), as storing them all in memory is impractical.
3. Retrieval Augmented Generation (RAG): This technique involves breaking your custom data into smaller “chunks,” generating embeddings for them, and storing them in a vector database. When a user asks a question, the system first retrieves the most relevant chunks from your database based on the query. These retrieved chunks are then fed along with the original query to the LLM. This “augments” the LLM’s prompt with relevant, specific context, forcing it to provide answers referenced from your data and significantly reducing hallucinations.

Q7: Can you give an example of hybrid RAG, which combines contextual understanding with keyword precision?

A7: Hybrid RAG aims to get the best of both worlds: the contextual relevance of embeddings and the precision of keyword matching. For example, if you’re searching for a specific coding library, a purely contextual search might provide related but not exact results. Hybrid RAG (using tools like TypeSense) ensures that your specific keywords are present in the results while still leveraging the LLM’s contextual understanding. This provides both the accuracy of exact matches and the broader relevance of semantic search.

Q8: How can non-textual data like images or audio be sent to an LLM?

A8: You can use Base64 encoding. This process converts binary data (like images, videos, or audio files) into a textual string. This “text” (which will look like a jumble of characters) can then be passed to the LLM. The LLM (or a connected system) can decode it back into the original binary data without any loss of information. This enables LLMs to process and understand multimodal inputs, allowing applications like image search (e.g., “show me pictures of cats”) or even analyzing video footage for specific events (e.g., detecting cheating in an exam recording).

Q9: What is Prompt Engineering, and why is it so important for LLM applications?

A9: Prompt Engineering is the art and science of crafting effective prompts to get the best results from LLMs. It’s crucial because the quality, consistency, and cost-effectiveness of an LLM application heavily depend on the prompts it uses. Good prompt engineering ensures:

1. Consistency: Reliable and repeatable outputs.
2. Cost-effectiveness: Efficient use of tokens.
3. Scalability: Applications that perform well under high demand.
4. Accuracy: Reduces hallucinations and improves relevance. Key techniques include making prompts detailed, providing examples, using structured formats (like XML tags or JSON schemas), and continuous documentation of prompts that work (and don’t work) as models evolve.

Q10: How can LLMs be used to bridge the gap between natural language and backend functions, enabling intelligent applications?

A10: This is achieved through Function Calling. You inform the LLM about the available functions in your backend system (e.g., schedule_meeting(date, time, room)). When a user inputs a natural language query like “Schedule a meeting for tomorrow at 10 AM in Room 301,” the LLM understands the user’s intent, extracts the necessary parameters (date, time, room), and generates a call to your schedule_meeting function. This allows users to interact with complex systems using intuitive language, making applications more intelligent and user-friendly, like a personal travel assistant or a natural language interface to a database.

Q11: What is the significance of “text extraction” using LLMs?

A11: Text extraction is a highly valuable application where LLMs convert unstructured or semi-structured text (like invoices, contracts, or scanned documents) into structured data (e.g., JSON). This automates tedious and error-prone tasks like data entry, auditing, and compliance. By providing the LLM with a specific JSON schema (a defined structure of fields you want to extract), it can accurately pull out relevant information (e.g., invoice number, supplier, total amount) from complex documents. This capability can form the basis of entire businesses focused on automating document processing.

Q12: How do LLM evaluations (Evals) help in developing robust LLM applications?

A12: LLM Evals (like PromptFoo) are essential for thoroughly testing LLM applications. They involve creating test cases (prompts) and defining expected outcomes or success criteria (assertions). Evals help you:

1. Measure performance: Track cost (tokens), speed, and response quality.
2. Ensure consistency: Verify that your application reliably produces desired results.
3. Prevent regression: Test if new model updates or prompt changes inadvertently degrade performance or introduce errors. This systematic testing is vital for building commercially viable LLM applications, as it allows you to continuously monitor and improve their reliability, accuracy, and efficiency.

Q13: I’m having trouble with an image compression task in Assignment 2 (GA2). Can you provide some hints or advice?

A13: This is a challenging problem, and the instructor even struggled with it initially. A key hint is to be careful about feature removal before compression. Sometimes, in an effort to reduce file size, you might inadvertently discard features that a compression algorithm like WebP (designed by Google) relies on. Ensure you preserve all original image features and then apply the compression algorithm. While tools like GPT can generate code, sometimes understanding the underlying principles and carefully managing preprocessing steps are more crucial. The problem is solvable, and the instructor is confident you can figure it out.

Q14: Could you provide the name of the AI startup you mentioned that uses LLMs for workplace safety/exam proctoring?

A14: The startup is called Hyperbots. They are actively recruiting AI/ML engineers and offer internships. The instructor can connect interested students with them.

Q15: Can you solve my specific GA2 problem after the deadline, or share resources about your exam proctoring project?

A15: Yes, after the assignment deadline, the instructor can discuss specific solutions or help with any remaining questions. Regarding the exam proctoring project, the instructor can connect you with the co-founder of Hyperbots (Anand) to share more details about how they use LLMs for exam integrity, including analyzing student behavior via video frames to detect cheating patterns.

Q16: What is Function Calling, and how is it used in LLM applications?

A16: Function Calling is a powerful capability that allows an LLM to call external functions or tools defined by you. You provide the LLM with a “schema” (a structured description) of your available backend functions (e.g., schedule_meeting(date, time, room)). When the LLM receives a user prompt, it can understand if the user’s intent maps to one of your defined functions. It then extracts the necessary parameters from the natural language input and calls that function. This bridges the gap between natural language and executable code, allowing LLMs to perform complex tasks, interact with APIs, or query databases directly. This is a crucial concept for building truly intelligent and interactive AI applications.