“TA session 12 July : week 5 , LLM API and embedding”#

Duration: 1h 54m

Here’s an FAQ based on the TDS tutorial:

Tools in Data Science (TDS) Live Tutorial: LLM Techniques#

This FAQ summarizes a live tutorial on advanced Large Language Model (LLM) techniques, including aspect-based sentiment, LLM extraction, JSON schema, and function calling.

Q1: What’s the main goal of today’s session compared to yesterday’s?

A1: Yesterday, we learned to get basic prompts from ChatGPT using JSON POST requests. Today, we’re adding extra “training” to those prompts to get more precise and “calibrated” responses from ChatGPT for specific tasks.

Q2: How do I provide this additional training to ChatGPT?

A2: The API call components (URL, header with your API key) remain the same. The key difference is in the JSON object you send in the messages list. Instead of just a system role and a user prompt, you’ll add more details to guide ChatGPT.

Q3: Can you give an example of this “additional training”?

A3: Yes. For “aspect-based sentiment analysis,” you can:

Define Metrics: Tell ChatGPT the specific aspects you want it to evaluate (e.g., “acting,” “storyline,” “direction”).
Provide an Example: Include a sample review (e.g., from a CSV file).
Label the Example: Critically, you then explicitly label your interpretation of that example for each metric (e.g., “acting: good,” “storyline: good,” “direction: good”). This teaches ChatGPT your specific interpretation. When you then send a new, unlabeled review, it uses this previous training as a benchmark. This technique is also known as “one-shot learning” because you’re providing a single training example.

Q4: What are the different types of “LLM learning” based on how much training data I provide?

A4: There are three main types:

Zero-shot learning: You provide no specific training examples in your prompt. ChatGPT relies solely on its vast pre-existing knowledge base to generate a response.
One-shot learning: You provide one example and its corresponding labeled output within the prompt. ChatGPT uses this single example to guide its responses to subsequent queries in the same conversation.
Multi-shot learning: You provide multiple examples and their labeled outputs. This gives ChatGPT even more context and helps it produce more accurate and calibrated responses.
Note: Multi-shot learning uses a few examples within the prompt, which is different from the immense datasets used to train the base LLM models themselves.

Q5: Can I use these techniques to build my own chatbot?

A5: Yes, absolutely. Chatbots often leverage LLM APIs like ChatGPT. When a user types into your chatbot, in the background, your application can send requests to an LLM API (like OpenAI’s). The techniques shown (one-shot, multi-shot training) allow you to specialize your chatbot’s responses. For example, an Airbnb chatbot could be trained to only answer questions related to accommodations, preventing it from wasting tokens (and money) on general knowledge queries. This training helps control its scope and efficiency.

Q6: Can I run LLM models locally on my computer for training and use, instead of relying on a cloud API?

A6: Yes, you can. While the example uses OpenAI’s cloud API, many pre-trained LLM models are available for local download (e.g., from platforms like Hugging Face). Tools like “LLM Studio” can help you download and run these models.

Benefits: You get unlimited local usage (no token costs) and greater privacy.
Considerations: Local LLMs typically require a reasonably powerful computer, though models are becoming increasingly compressed, with some even capable of running on mobile phones.
Note: While this is possible for experimentation, extensive local LLM training is generally beyond the scope of this course.

Q7: When I provide training data (one-shot or multi-shot), does it permanently update the LLM’s base model?

A7: No, it does not. The training you provide within your prompt (e.g., the one-shot example) calibrates the LLM’s behavior specifically for your current session or a particular sequence of interactions. It influences how the LLM responds to you based on the context you’ve given, but it does not permanently alter the underlying base model for all users.

Q8: What is “LLM Extraction” and why is it useful?

A8: LLM Extraction is the process of getting structured data out of unstructured or semi-structured text using LLMs.

Why it’s useful: Unstructured text (like a block of address text) is difficult for computers to process for analysis. Structured data (like a CSV with separate columns for street, city, zip code) is easily usable for tasks like project analysis. LLMs can intelligently parse and extract specific pieces of information from complex text.
Process: You instruct ChatGPT to output a JSON response ("response_format": {"type": "json_object"}) and, crucially, you explicitly define the desired fields/keys you want in the JSON (e.g., “state_name,” “zip_code,” “country”). You then provide the unstructured text.
Result: ChatGPT uses its understanding to parse the text and output a JSON object, extracting and sometimes inferring values for the requested fields in a perfectly structured way.

Q9: What is a “JSON Schema” and how is it used with LLMs?

A9: A JSON Schema is a “blueprint” that defines a very specific and consistent structure for JSON data. While basic LLM Extraction can provide structured data, the output isn’t always strictly predictable. A schema guarantees a consistent output, which is vital for automated processing and validation.

How to use: You can ask ChatGPT to generate a JSON schema based on your desired output structure.
Components: A schema defines the type of data (e.g., “object”), its properties (fields like “state”, “country”), nesting of objects (e.g., “state” containing “name” and “code”), and required fields.
Note on description fields: While description fields exist in a schema for human readability and documentation, experiments suggest the LLM can often infer meaning from key names (like “state_name”) even without detailed descriptions. Their primary benefit is for developers, not necessarily critical for the LLM’s functional parsing.

Q10: What is “Function Calling” and why is it so powerful for LLMs?

A10: Function Calling is a powerful capability where an LLM can execute external code (functions) based on your natural language prompt.

How it works:
1. You define your custom functions (e.g., reverse_string, trim_string) within your prompt, specifying their name, description, and parameters (using JSON Schema).
2. You tell ChatGPT which tools (functions) are available.
3. You make a normal prompt (e.g., “Reverse the string ‘hello world’ and then trim the first and last letters.”).
4. ChatGPT understands the intent, decides which function(s) to call, extracts the arguments, and outputs the function call (e.g., trim_string(reverse_string("hello world"))).
5. Your application then intercepts this output, executes the actual Python function, and optionally feeds the result back to ChatGPT.
Benefits:
- Cost Savings: Function definitions are processed once. Subsequent calls just send the function name and arguments, drastically reducing token usage.
- Nesting/Chaining: Functions can be combined (e.g., trim(reverse(string))) to perform complex, multi-step operations.
- External Control: Allows LLMs to interact with and control external systems, process data, or generate content by running code outside their “brain.”
Real-world Example: We successfully used Function Calling to parse a complex, unstructured election PDF document. We asked ChatGPT to generate a CSV with specific columns (state, constituency, candidate, votes). It returned a perfect CSV in minutes, demonstrating its ability to run code (outside its own internal processing) to extract and structure data from messy sources. This can revolutionize data processing.

Q11: What are “Embeddings” in the context of LLMs?

A11: Embeddings are the numerical representations of words, phrases, or concepts. LLMs convert language into high-dimensional vectors (arrays of numbers).

How they work: These numbers capture a word’s meaning and its relationships with other words in a “semantic space.” For instance, “Apple” might have thousands of numbers associated with it, representing its ties to “fruit,” “tree,” “company,” or even idioms like “apple of my eye.”
Similarity: Words with similar meanings will have “closer” embedding vectors, which can be measured mathematically.
Why they’re important: They form the mathematical basis for how LLMs understand context and meaning. By comparing embeddings, LLMs can identify semantic similarities, categorize information (like sentiment), and model topics.
Topic Modeling (K-Clustering): Embeddings are used in algorithms like K-clustering to group similar words or documents into “clusters” based on the proximity of their embedding vectors. This helps identify underlying themes or categories in large datasets.

Q12: Do the embedding numbers change frequently, or are they fixed?

A12: The core embedding numbers for a base LLM model are generally fixed until the model itself is updated to a new version (e.g., from GPT-3.5 to GPT-4). However, when you provide specific training (like one-shot or multi-shot learning) within a prompt, the LLM temporarily calibrates or “fine-tunes” its understanding (and thus its internal use of embeddings) for your specific session or task. This doesn’t change the underlying global model but adapts its behavior to your immediate context.

Q13: Which LLM models are best to download and use locally?

A13: The “best” model depends entirely on your specific use case.

Factors: Are you focused on coding, general knowledge, a specific language, image generation, or a niche task like medical analysis?
Resources: Platforms like Hugging Face offer a vast array of pre-trained models. You can often filter by task, size, and language. Examples include “Codium” for coding, or various models for image-to-text conversion.
Recommendation: Search on Hugging Face using keywords relevant to your task (e.g., “model for coding,” “image generation model”). You’ll find options you can download and run locally.

Q14: What is the benefit of running LLMs locally, even if cloud APIs like ChatGPT are available?

A14: Running LLMs locally (e.g., using LLM Studio and models from Hugging Face) offers several key advantages:

Unlimited Usage: You avoid token limits and associated costs that come with cloud-based APIs.
Privacy: Your data remains on your local machine, offering greater control over sensitive information.
Experimentation: It allows for extensive experimentation and development without worrying about usage fees.

Q15: What about LLMs and the concept of “self-awareness”?

A15: It’s important to clarify that current LLMs are essentially sophisticated matrix-multiplying machines. Their “intelligence” stems from complex mathematical operations performed on immense datasets, allowing them to generate human-like text and understand context. They do not possess self-awareness, consciousness, or true understanding in the human sense. Claims of LLMs being “self-aware” or leading to “killer robots” are not based on their current technological capabilities; they are powerful tools for pattern recognition and generation, but nothing more.