2025 05 11 Week 1 Session 2 Setup & Debugging TDS May 2025#

Duration: 11005.0Here’s an FAQ based on the provided transcript:

Q1: VirtualBox didn’t work when I tried to install it yesterday. What went wrong, and why can’t I follow yesterday’s installation instructions today?

A1: The issue likely stems from your computer’s architecture. I was using a Mac with an ARM architecture (M-series chip). VirtualBox doesn’t currently support ARM architecture, so it failed. This also applies to Windows ARM-based machines (like some Windows Surface models). If your machine is ARM-based, VirtualBox won’t work because it hasn’t been compiled for ARM architecture, and it’s missing necessary drivers.

Q2: What was discussed in yesterday’s session, and where can I find the recording? I missed it.

A2: Yesterday’s session was an introductory session about Tools in Data Science (TDS). It covered many valuable things unique to TDS that I can’t summarize quickly here. I highly recommend watching it, as it’s crucial for understanding how this subject runs, which is different from other diploma subjects. The recording will be available on the main TDS portal later today.

Q3: Where exactly on the portal can I find the recording for yesterday’s session?

A3: You need to go to the main TDS portal (not the Seek portal). On the main page, at the bottom, you’ll see “Live Sessions Streams” and then a “Playlist.” The recording will be uploaded there under the playlist for this term (e.g., “May 2025”).

Q4: What tools should I install for TDS, and do I need a virtual machine (VM)?

A4: We highly recommend using a Linux environment for TDS, as it’s the industry standard.

If you’re on Windows: You should use either Windows Subsystem for Linux (WSL) or a VirtualBox VM running Linux. WSL is generally the better option.
If you’re on Mac: You don’t need a VM, as macOS is Unix-based, and most Linux commands and tools will work directly.
If you already have Ubuntu: You don’t need to install WSL or a VM.

Q5: What are the initial steps after installing WSL, and how do I troubleshoot common issues like drives not appearing?

A5: After installing WSL, the very first thing you should do is open it and run sudo apt update, followed by sudo apt upgrade. This ensures all your packages are up-to-date.

Community Tip: If you encounter issues like your drives not appearing in WSL, try running wsl --shutdown in PowerShell, then restart your computer. This has resolved the issue for some users.

Q6: Which specific packages should I install for Week 1?

A6: You should refer to the Week 1 content on the main TDS portal page. It lists prerequisites for the rest of the term. Some essential packages include:

VS Code (highly recommended)
GitHub Copilot (highly recommended)
UV (for Python virtual environments)
Node.js Package Manager (NPM) and NPX
LLM at the CLI (Command Line Interface)

Q7: Why do I need to install Unicode, and what is it?

A7: Unicode is an encoding mechanism. Its video explains how it works and why it’s important. When processing data, especially text files, different encodings (like UTF-8, ASCII) can cause issues even if the file looks normal. The Unicode topic helps you identify file encodings and understand why it matters for smooth data processing workflows.

Q8: Can I use cloud-based VMs like AWS EC2 for TDS?

A8: Yes, you can use cloud-based instances. However, ensure that any projects or services you run on them are accessible to us for evaluation. If we cannot access your work due to an unconventional hosting setup, we cannot properly assess it.

Q9: What is JQ, and do I need to install it?

A9: JQ is a command-line JSON processor. It’s listed as a useful tool for working with JSON data. While not strictly mandatory, it can be very helpful. If you encounter installation issues (e.g., Apple blocking it as an unverified developer tool), you can try using Homebrew (a package manager) to install it, as it often provides solutions for such cases.

Q10: What is Homebrew, and why is it recommended?

A10: Homebrew is a package manager for macOS and Linux. It’s highly recommended because it simplifies installing various tools and packages. Many tools not directly available in standard Linux apt repositories can be easily installed via Homebrew. It also allows installing packages without needing sudo permissions.

Q11: How do I get and configure the AI Pipe/tokens for LLM tools?

A11: You’ll need to obtain an API key from OpenAI (you can do this via platform.openai.com). Once you have your key:

Export the API Key: Set an environment variable for your OpenAI API key using a command like LLM key set open-ai YOUR_API_KEY.
Export the Base URL: Set another environment variable for the AI Pipe’s base URL using a command like export OPEN_AI_BASE_URL=https://api.toolsindatascience.com/v1/. (This URL is specifically for the TDS AI Pipe setup, which acts as a proxy to OpenAI).

Q12: Is there a limit or expiration for the AI Pipe tokens provided by TDS?

A12: Yes, the tokens have a daily usage limit. It’s set fairly low initially but will be increased when projects come around to accommodate higher usage needs. The tokens last for the duration of the term.

Q13: Do I need to install VS Code separately within WSL, or will my Windows installation suffice?

A13: You should install VS Code within your WSL instance. This ensures that any extensions you install and the project files you work on are directly compatible with and accessible from your Linux environment, preventing potential conflicts or setup complexities.

Q14: I’m new to Linux and the command line. Where can I learn the basics?

A14: While this course doesn’t specifically cover Linux basics, it’s highly recommended to familiarize yourself with them. In Week 1, there’s a video under “Terminal/Bash” (around 75 minutes long) that explains the fundamentals of Bash, which is crucial for understanding how to interact with the Linux command line. Watching this video will significantly help you. Also, you can use the virtual TDS TA (a GPT-based bot available on the portal) to ask basic questions and get explanations in simple language.

Q15: Can I use my own paid OpenAI API key instead of the TDS-provided tokens?

A15: Yes, you can use your own OpenAI API key if you prefer. You would need to configure the LLM tool to use your key directly instead of the TDS AI Pipe’s proxy.

Q16: How has the TDS course structure adapted to the rise of AI tools in the last year?

A16: We will discuss this topic in more detail after this session, focusing on setup and tools first.

Q17: Is it mandatory to use my student email for GitHub, or can I use my personal email?

A17: We don’t have strict stipulations for GitHub email in TDS. You can use either. However, some other courses in the program might require your student email. During submission, we will ask for the GitHub URL you’ve used.

Q18: What is the deadline for GA1 (Graded Assignment 1)?

A18: The deadline for GA1 is May 18th. GA1 contributes 15% to your final grade.

Q19: I’m getting an “invalid token” error when trying to use LLM after setting up the AI Pipe.

A19: This indicates an issue with how your API key is being passed or configured.

If you’re using a key from platform.openai.com (not TDS-provided): Make sure you’re using the method to directly set your OpenAI API key and URL, not the TDS AI Pipe URL. The AI Pipe specifically uses TDS-generated tokens, not external OpenAI keys.
If you’re using a TDS-provided token: Double-check that you copied the key correctly and that the variable is set with export OPEN_AI_API_KEY=your_key_here and export OPEN_AI_BASE_URL=https://api.toolsindatascience.com/v1/. Ensure there are no extra spaces or characters.

Q20: The LLM embed command is showing a “404 Not Found” error even after following instructions.

A20: The “404 Not Found” error suggests that the LLM tool is trying to reach an endpoint that doesn’t exist or is incorrect. Make sure the OPEN_AI_BASE_URL environment variable is correctly set to https://api.toolsindatascience.com/v1/. Also, ensure you are providing the correct model name (e.g., text-embedding-ada-002, 3-small). You can use LLM embed models to see all available embedding models.

Q21: How do I get a clear explanation of an error message or command that I don’t understand?

A21: You can use the virtual TDS TA (a GPT-based bot available on the portal). Copy the command you ran and the error message you received, then paste it into the TDS TA. Follow up with a prompt like “Please explain this command and response in simple language, assuming I’m new to coding.” The TA is designed to break down complex information and can even explain things in different ways if you still don’t understand. This is a crucial tool for learning, especially for new coders.

Q22: Some packages like UV aren’t working even after following the steps. What should I do?

A22: There might be a configuration issue.

First, try restarting your computer.
If that doesn’t work, try reinstalling the package.
Ensure your network connection is stable (sometimes Wi-Fi or DNS issues can block installations). You might try using a mobile hotspot to rule out Wi-Fi problems.
If the issue persists, try sudo apt update and sudo apt upgrade before reinstalling.

Q23: Why should I install LLM packages in WSL instead of directly on Windows?

A23: You should install LLM (and other TDS-related packages) within your WSL environment because all course content, commands, and project evaluations are designed for a Linux/Bash environment. Installing them directly on Windows can lead to compatibility issues, especially with certain Python packages or command-line functionalities. Even Docker, when used on Windows, often runs a Linux subsystem in the background. Using WSL ensures a consistent and supported environment.

Q24: Why is it recommended to use Linux-based systems over Windows for TDS, especially since I’m already familiar with Windows?

A24: The primary reason is that Linux and command-line interfaces are industry standards for data science and development work. While you can do some things on Windows, our sessions and projects will be geared towards a Bash/Linux environment. Familiarity with Linux is a valuable skill in the industry. We encourage you to use WSL on Windows to get this experience. If you stick with Windows exclusively (e.g., using PowerShell), you might encounter limitations or lack direct support for specific tools required in the course. Also, some packages simply do not run on Windows. Even Microsoft itself uses Linux for many of its critical systems.

Q25: What is the expected behavior if LLM embed works correctly?

A25: If LLM embed works correctly, it will produce a large JSON object as a response, representing the embedding vector for the text you provided. You can direct this output to a text file for easier viewing (e.g., LLM embed ... > answer.txt) and then open the file in a text editor like Notepad.

Q26: I’m having trouble understanding how to use specific commands. How can I get help?

A26: You should use the virtual TDS TA (a GPT-based bot available on the portal). Copy the command and any error message you receive. Then, paste it into the TDS TA and ask it to explain what the command does, what the error means, or how to fix it, in simple language. The TA is designed to be a patient teacher and can help you understand the basics as you go along.

Q27: Can I use the virtual TDS TA (GPT) in my native language, or if I find English explanations unclear?

A27: Yes! The virtual TDS TA supports multiple languages. If you find the English explanations unclear, or if you prefer to learn in your native language, you can ask the TA to explain concepts in your mother tongue. It can translate your questions and provide explanations in your preferred language, which can be very helpful for understanding complex concepts.

Q28: What is the typical workflow for solving problems and learning in this course?

A28: The recommended workflow is:

Watch the videos: Understand the concepts presented in the video lectures.
Practice hands-on: Try out the commands and exercises shown in the videos yourself. Don’t just passively watch.
Identify problems: You will inevitably make mistakes or encounter errors. This is part of the learning process.
Use the TDS TA: For any command or error you don’t understand, use the virtual TDS TA to get step-by-step explanations in simple language.
Ask in Discord: If the TA can’t resolve your issue, or if you prefer human interaction, post your problem (with relevant code and error messages) in the Discord channel. Other students or TAs can assist.

Q29: Are there any fundamental Linux commands I should be aware of to start with?

A29: Yes, for anyone new to Linux, I strongly recommend watching the “Terminal/Bash” video in Week 1. It explains the basics of using the Bash command line. Understanding these fundamentals will greatly help you navigate and perform tasks within your Linux environment, as it’s essentially the “programming language” you’ll use to communicate with the system.

Q30: I’m having trouble getting the LLM embed command to work after setting up everything. I’m seeing “404 Not Found”.

A30: This “404 Not Found” error suggests an issue with the base URL that the LLM tool is trying to access for embeddings. You likely need to explicitly set the base URL for the embedding endpoint.

Use the command export OPEN_AI_BASE_URL=https://api.toolsindatascience.com/v1/
Then, retry your LLM embed command.

Q31: I get an “invalid host” error when trying LLM embed. What am I doing wrong?

A31: The “invalid host” error usually means there’s a problem with the URL you’re providing for the embedding service. Ensure the OPEN_AI_BASE_URL environment variable is correctly set. The correct URL for the TDS AI Pipe (which handles embeddings) is https://api.toolsindatascience.com/v1/.

Q32: I’m getting an “Insufficient Quota” error. What does this mean?

A32: This means you have run out of usage quota for your AI Pipe token for the day. Each token has a daily limit on how many requests or how much data it can process. This limit resets every day. For project submissions, the daily limits will be increased.

Q33: I’m encountering a “HTTP 401 Unauthorized” error after setting the token.

A33: A 401 Unauthorized error typically means your API key (token) is incorrect, expired, or hasn’t been properly recognized by the system.

Double-check that you copied the key exactly as provided by TDS or OpenAI.
Ensure the key is correctly set as an environment variable using the export command (e.g., export OPEN_AI_API_KEY=your_key_here).
Confirm you’re using the correct base URL for the AI Pipe or OpenAI directly, depending on which key you’re using.

Q34: The LLM embed command still isn’t working even after trying all these suggestions.

A34: This is unusual, as I’ve tested it myself, and it’s working. There might be a deeper configuration issue unique to your setup. Please post your specific problem, including the exact command you ran and the full error message, in the Discord channel. I will investigate it further with our team.

Q35: Is there a way to quickly copy all the output from the terminal in WSL?

A35: Yes, in your WSL terminal, you can press Ctrl+Shift+C to copy selected text. If you want to copy the entire output of a command to a file, you can use the redirection operator > followed by a filename. For example, LLM embed ... > output.txt will save the output to output.txt, which you can then open with a text editor.

Q36: After running LLM embed, I’m getting a “ValueError: Expected a JSON array” error, and the output just shows my input string.

A36: This error means the LLM embed command is not receiving a valid JSON array as input, even though it expects one. The issue is likely with how you’re formatting the input string itself. The command expects the text to be embedded as a JSON array.

Solution: When providing the input text, you need to wrap it in square brackets ([]) to make it a JSON array, even if it’s just one string. For example, LLM embed -c "['my email is this']" ...

Q37: I’m getting a “NameError: name ‘open_ai_api_key’ is not defined” error.

A37: This error indicates that the environment variable for your API key, OPEN_AI_API_KEY, is not correctly recognized in your current shell session. Even though you may have used export previously, new terminal sessions might not automatically load it, or there might have been a typo.

Solution: You need to explicitly set the OPEN_AI_API_KEY environment variable in each new terminal session before running LLM commands. Make sure the variable name is OPEN_AI_API_KEY (all caps, underscores) and you use the export command. You can also add it to your .bashrc file for persistent loading.

Q38: Can I access my WSL files from VS Code?

A38: Yes! If you’ve installed VS Code within your WSL instance (as recommended), you can launch it directly from your WSL terminal by typing code .. This will open VS Code, already connected to your WSL file system, allowing you to seamlessly edit files within your Linux environment.

Q39: How do I select all text in a file within the terminal (e.g., in vi/vim)?

A39: In vim (or vi), you can type ggVG to select all text. Then, to copy it, you would type y. You can then paste it into Notepad or another editor. For general terminal output, you can select and copy using Ctrl+Shift+C.

Q40: I’m seeing a “ModuleNotFoundError: No module named ’tiktoken’” error.

A40: This error means the tiktoken Python module is not installed in your current Python environment within WSL.

Solution: You need to install it using pip (or uv). Run pip install tiktoken in your WSL terminal. If you are using a virtual environment (which is recommended), ensure you activate it first.

Q41: How do I know which version of Python is installed in WSL?

A41: You can check your Python version by simply typing python --version or python3 --version in your WSL terminal.

Q42: What is pip?

A42: pip is the package installer for Python. It allows you to install and manage additional libraries and modules that are not part of the Python standard library.

Q43: What is uv?

A43: uv is a faster and more modern Python package installer and virtual environment manager, often recommended as an alternative to pip and venv. It helps manage project dependencies efficiently.

Q44: What is tiktoken?

A44: tiktoken is a tokenizer library developed by OpenAI. It’s used to count and manage “tokens” (pieces of words or characters) when interacting with OpenAI’s language models.

Q45: After LLM embed, I’m still getting errors related to JSON or embedding even after using a correct JSON array. It says “invalid value”.

A45: The “invalid value” error, even with correct JSON formatting, can happen if the base URL or API key is still incorrect or not active.

Double-check: Re-verify both your OPEN_AI_API_KEY and OPEN_AI_BASE_URL environment variables. Ensure the key is valid and the URL is precisely https://api.toolsindatascience.com/v1/. If you’re encountering quota issues, your key might be temporarily blocked.

Q46: After running LLM embed, I got a large JSON response. What is this, and why is it structured this way?

A46: This large JSON response is the embedding vector for the text you submitted.

JSON (JavaScript Object Notation): It’s a lightweight data-interchange format, commonly used for sending data between a server and a web application (or in this case, between the LLM tool and the AI service). It’s human-readable and easy for machines to parse.
Embedding Vector: This is a numerical representation (a list of numbers) of your input text. It captures the semantic meaning of the text, allowing computers to understand relationships between words and concepts.
Structure: The JSON object contains various key-value pairs (like object, model, data, usage). The actual embedding vector is found within the data field, typically as a nested array of floating-point numbers.

Q47: If I’m using Windows, what are the recommended steps to prepare my system for the course?

A47:

Install WSL: Install Windows Subsystem for Linux.
Install Ubuntu: Choose and install an Ubuntu distribution within WSL (e.g., wsl --install -d Ubuntu).
Update/Upgrade WSL: After installation, open Ubuntu in WSL and run sudo apt update followed by sudo apt upgrade.
Install VS Code: Install VS Code within your WSL instance.
Install Python tools: Install uv and other Python packages like tiktoken using pip install (or uv install) in your WSL.
Install LLM tools: Install the LLM tool and configure your AI Pipe tokens using the export commands for your API key and base URL.
Install Git & GitHub CLI: Install Git and GitHub CLI for version control and interacting with GitHub.
Get Microsoft 365/Excel: Subscribe to the free version of Microsoft 365 to access Excel for relevant GA questions.

Q48: The video links for the sessions aren’t working/available yet.

A48: The video links are typically made available shortly after the live session concludes. There might be a slight delay as they are processed and uploaded. Keep checking the playlist on the main TDS portal page.

Q49: How can I subscribe to the TDS channel to get notifications for new video uploads?

A49: You can subscribe to the TDS YouTube channel (linked on the main TDS portal page). As soon as videos are uploaded, you’ll receive notifications.

Q50: Can I use older versions of Microsoft Excel, like Excel 2021, for GA questions requiring Excel?

A50: No. For GA questions that require Excel, you must use the latest version available through Microsoft 365. The free version of Microsoft 365 provides access to the necessary Excel features.

Q51: Are there other GA questions that require Microsoft 365 or Excel besides the one mentioned?

A51: Yes, some later modules will also require Microsoft 365/Excel. It’s best to have the latest version of Excel ready.

Q52: Is there a bug bounty leaderboard for TDS?

A52: We are currently planning something similar, a “bug bounty leaderboard,” but it’s not official yet. We’re considering piloting it at some point.

Q53: How important is it to understand JSON for this course?

A53: Understanding JSON is very important. It’s a fundamental data exchange format used extensively in this course and in real-world data science applications. There’s a dedicated video for JSON in Week 1 content.

Q54: What is the significance of “x86” vs. “ARM” architecture mentioned earlier?

A54: x86 (or x64) and ARM are different CPU architectures. Most traditional computers (both Windows and Mac) historically used x86/x64. Newer Macs (M-series) and some Windows devices use ARM. The difference matters because software needs to be compiled specifically for the architecture it runs on. Virtualization tools like VirtualBox often don’t support ARM, which can cause installation issues.

Q55: Will the course cover topics like cargo or rust?

A55: No, the course will not cover cargo or rust.

Q56: How do I commit a JSON file to GitHub?

A56: You would use standard Git commands:

git add your_file.json
git commit -m "Add JSON file"
git push

Q57: Can I just install astro-uv instead of uv?

A57: No, you should install uv directly. There might be other packages with similar names, but uv is the specific tool recommended.

Q58: I’m getting a “Git error” in WSL.

A58: A “Git error” can refer to many specific problems. Please paste the exact error message into Discord, and we can help diagnose it.

Q59: Can I download uv using sudo snap install uv?

A59: I do not recommend using snap for installing packages. It’s better to stick with either the curl command (as shown in the uv documentation) or Homebrew for installing uv.

Q60: I’m getting a “permission denied” error when trying to install uv.

A60: A “permission denied” error usually means you’re trying to execute a command without sufficient privileges.

Solution: Make sure you use sudo before the command (e.g., sudo apt install uv).

Q61: What is cURL?

A61: cURL is a command-line tool for making network requests. It’s often used to download files from the internet or interact with web APIs.

Q62: What is Wget?

A62: Wget is another command-line tool, similar to cURL, used for retrieving content from web servers.

Q63: The LLM embed command is saying UV is not found even after installation.

A63: This often happens when uv is installed but its location is not included in your system’s PATH environment variable.

Solution: Ensure that the directory where uv was installed is added to your PATH. The uv installation instructions usually guide you on how to do this.

Q64: My computer is bit old, will it support uv?

A64: uv is a lightweight tool, so it should run on most systems that can handle WSL or a basic Linux distribution. Performance depends more on your overall system resources. If your CPU supports virtualization, a virtual machine might run faster.

Q65: What’s the best way to copy large outputs from the terminal in WSL?

A65: You can redirect the output of your command to a file using the > operator. For example, your_command_here > output.txt. Then, you can open output.txt with a text editor.

Q66: I’m getting an “invalid character” error when trying to paste a command.

A66: This can happen if you’re using Ctrl+C from a Windows application to copy and then Ctrl+V directly into a WSL terminal.

Solution: For copying from a Windows app, use Ctrl+C as usual. But when pasting into the WSL terminal, use Ctrl+Shift+V. For copying within the WSL terminal, use Ctrl+Shift+C.

Q67: I have an old laptop and want to install Linux/WSL. My laptop doesn’t support virtual machines. What options do I have?

A67: If your laptop doesn’t support virtualization (which would prevent running VirtualBox), you still have a few options:

Dual Boot Linux: You can install a full Linux distribution (like Ubuntu or Debian) alongside Windows (dual-boot). This is a more involved process but gives you native Linux performance.
Linux from a USB drive: You can create a bootable USB drive with a lightweight Linux distribution. This allows you to run Linux directly from the USB without installing it on your hard drive. This might be slower but doesn’t require virtualization.
WSL (Windows Subsystem for Linux): As long as your Windows version supports it, WSL doesn’t require full hardware virtualization and can be a good option for older machines.

Q68: I’m getting an “invalid token” error on the LLM embed command, even though I’ve set the token.

A68: An “invalid token” error typically indicates that the API key you’ve set is either incorrect, expired, or not being recognized by the LLM tool.

Solution:
1. Re-verify the key: Go back to the TDS portal page where the key is provided and re-copy it.
2. Reset the environment variable: Use the export command again in your WSL terminal to set the OPEN_AI_API_KEY and OPEN_AI_BASE_URL variables, ensuring no typos and the correct https://api.toolsindatascience.com/v1/ URL.
3. Check for extra characters: Sometimes, copying can introduce hidden characters. Carefully paste the key.
4. Restart WSL: As a last resort, try wsl --shutdown in PowerShell, then restart your system and re-set the variables.

Q69: I’m getting an “invalid value for parameter data” error.

A69: This error occurs if the LLM embed command doesn’t receive the input text in the expected format, which is a JSON array of strings.

Solution: Ensure that the text you’re trying to embed is enclosed in square brackets and quotation marks, even if it’s a single string. For example: LLM embed -c "['my email is this']" ...

Q70: I’m getting a “ModuleNotFoundError: No module named ’tiktoken’” error.

A70: This means the tiktoken Python module is missing from your Python environment.

Solution: Install it using pip (or uv) within your WSL: pip install tiktoken. If you’re using a virtual environment, ensure it’s activated before installation.

Q71: I’m getting errors like “Expected a JSON array, got a string” for the LLM embed command.

A71: This confirms that the LLM embed command is expecting its input data to be a JSON array (even if it contains only one item), but it’s receiving a plain string instead.

Solution: Wrap your input string in square brackets and double quotes to form a JSON array. For example, use LLM embed -c "['your text here']" ...

Q72: LLM embed is giving a “404 Not Found” error. What could be the problem?

A72: A “404 Not Found” error with LLM embed means the tool is unable to reach the specified API endpoint. This is most likely due to an incorrect or malformed OPEN_AI_BASE_URL.

Solution: Double-check that your OPEN_AI_BASE_URL environment variable is set exactly to https://api.toolsindatascience.com/v1/. Any deviation will cause the 404 error.

Q73: LLM embed is giving an “invalid host” error.

A73: Similar to “404 Not Found,” an “invalid host” error indicates a problem with the OPEN_AI_BASE_URL. The system cannot resolve or connect to the host specified in the URL.

Solution: Verify that your OPEN_AI_BASE_URL environment variable is set precisely to https://api.toolsindatascience.com/v1/. A typo or extra character could lead to this error.