2025-03-27 Week 11 - Session 3 - TDS Jan 25#

2025-03-27 Week 11 - Session 3 - TDS Jan 25

Duration: 2h 7m

Here’s an FAQ-style transcription of the provided live tutorial:

Q1: For Project 2, how can I run multiple ngrok tunnels with the free version?

A1: You can’t directly run multiple separate ngrok http 8000 commands. Instead, create a YAML file where you specify all the different ports your applications will use. Then, run a single ngrok command referencing this YAML file. This way, a single ngrok tunnel will listen on multiple ports, which is a good workaround for running different applications.

Q2: For GA2 Q7, regarding GitHub Actions, do I need to include my email address in the action’s name? And will it be parameterized for each student?

A2: Yes, for GA2 Q7, you need to create a GitHub Action with a name that contains your specific email address. This question is parameterized, so it will use your own email ID, as it wouldn’t make sense to evaluate based on someone else’s. You just need to create the action beforehand and provide its URL for evaluation.

Q3: For GA2 Q6, which involves a JSON file, will I download it, and is it parameterized?

A3: We will send you a JSON file for GA2 Q6. It’s possible this file might be parameterized, meaning each student could receive a slightly different version, especially since it’s a small file based on “100 imaginary students.” To confirm, you can compare your JSON file with a friend’s. If they differ, it’s parameterized.

Q4: For the large PDF file question (GA4 Q9), how should I handle its size, especially since I’ve heard it’s not possible to process large files?

A4: The statement “you cannot process a large file” is not entirely true in this context. While the PDF file (likely 60MB) is large, the data within the file is constant for everyone. Only the questions asked about the file are parameterized. My advice is to pre-process the PDF file locally by converting it into a smaller, more efficient format like a CSV file or even load it directly into a pandas DataFrame in memory. The size won’t be an issue then, as you’ll be working with a much smaller, pre-processed dataset. You’ll still perform calculations based on the parameterized questions (e.g., specific student groups or marks), but on your efficient local copy.

Q5: For GA5 Q9, which involves transcribing a YouTube video, do I need to install many transcription libraries in the VM?

A5: No, you won’t need to install many libraries for this. We will provide a very tiny audio file (or a link to one), and you can use a publicly available API (like Gemini, which is free) via a sub-process call to transcribe it. The expected output is just the transcribed text. The audio file will be the same for all students.

Q6: What is the main goal for Project 2, and what should I focus on for the end-term exam?

A6: The primary goal for Project 2 is to make you extremely comfortable working with APIs, as data interchange in the real world heavily relies on them. The project is designed to give you extensive practice in this area, including interacting with different systems and managing data flow.

For the end-term exam, expect coverage of all 7 modules. Key topics include:

  • API Usage: Being comfortable with how APIs work.
  • String Outputs: Handling multi-line, formatting, Markdown, and JSON outputs (including understanding invisible escape characters like \n).
  • Browser Tools: Familiarity with browser developer tools (Elements, Console, Sources, Network, Application tabs).
  • CSS Selectors: Understanding CSS selectors, especially for data extraction purposes (up to combinators).
  • JSON: Knowing common JSON functions (e.g., loads, dumps) and JSON structure.
  • Bash & SQL: Basic commands and use cases, similar to what you’ve seen in GAs.
  • Git: Essential commands (clone, add, push, connect to remote, diff).
  • Image Processing: Understanding image formats (e.g., lossless PNG/WebP, vector SVG) and image size implications.
  • Deployment Tools: Knowledge of static hosting (e.g., GitHub Pages) versus serverless hosting (e.g., Vercel), and what each is used for.
  • GitHub Actions (GHA): Understanding the purpose of GHA for automating tasks, rather than specific syntax.

Q7: Will the next iteration of TDS (next term) include topics like AWS or other hosting applications?

A7: The curriculum for the next term is still in flux. We are actively working on it, and it’s possible that AWS or other advanced hosting topics might be included, along with more content on AI and image processing.

Q8: When the output needs to be a JSON string, and it includes escape characters like \n or \t, how should I handle it?

A8: You will receive a string that includes these escape characters. The correct way to handle this in Python is to use the json.loads() function. This function will convert the string into a proper JSON object, automatically interpreting and handling the escape characters correctly. If you need to convert a JSON object back into a string, you would use json.dumps().

Q9: For YouTube video transcription (GA5 Q9), if an API call is used, do I need to configure a Chrome driver?

A9: No, if you’re using a publicly available API for transcription, you don’t need to configure a Chrome driver. That would typically be required for browser automation tasks, which is not the primary approach suggested for this question.

Q10: For Docker-related questions, like GA2 Q8, what kind of details are expected?

A10: For Docker questions, we expect you to be familiar with basic Docker commands and their use cases. This includes understanding how to containerize an application, deploy it (e.g., to Azure), and pass environment variables to your Docker containers. The focus is on the purpose and practical application of Docker, not necessarily expert-level details.

Q11: Will there be questions about specific frameworks or libraries for image processing, beyond basic concepts?

A11: For image processing (e.g., GA5 Q3), questions will likely focus on fundamental concepts like different image formats (e.g., lossless vs. lossy, vector vs. raster) and how image size impacts data transfer. We generally won’t ask about specific library-related compression techniques or obscure tools.

Q12: Regarding Git-related questions, what aspects should I focus on?

A12: For Git, focus on essential commands and concepts such as git clone, git add, git push, connecting to a remote repository, and understanding git diff. These are fundamental operations and knowing them well will be beneficial.

Q13: How should I approach the PDF to CSV conversion question (GA4 Q9)?

A13: You need to pre-process the PDF file by converting its data to a CSV format within your code. The PDF file itself is provided as a static, non-changing resource. Your task is to extract the relevant data and store it as a CSV, which you can then manipulate as needed based on the parameterized questions.

Q14: For the YouTube video transcription question (GA5 Q9), will I be using Selenium?

A14: For YouTube video transcription, you can use youtube-dlp to extract the audio, and then a suitable publicly available Speech-to-Text API (like Google’s) for transcription. Selenium would be a last resort if no API is found, but the aim is to use simpler, more direct methods.

Q15: What about Collaboration? Is that part of Project 2’s learning goals?

A15: Yes, collaboration is definitely a key learning goal for Project 2. The multiple GAs (General Assessments) are designed to encourage you to work with your peers. We are even exploring larger collaborative projects for future terms, where students contribute, commit code, and have visibility into each other’s work.

Q16: Can you confirm the solution for GA4 Q9 (PDF to CSV conversion) again?

A16: Yes, for GA4 Q9, the PDF file’s content is constant for everyone. You should load this file directly within your function, process its data, and convert it into a CSV format (or a DataFrame in memory). The actual 60MB file is not passed as an input parameter; you handle it internally.

Q17: Is it possible to use a Vercel endpoint and then pass it to the API, instead of 8000 or 9000 ports directly?

A17: Yes, that can be done. You can have a Vercel application already deployed and running, and then pass its URL as an API endpoint. Your application would then send requests to that Vercel endpoint.

Q18: For the End Term Exam, will questions about Git be based on GitHub Actions?

A18: Yes, for Git-related questions, the focus will often be on GitHub Actions (GHA). Questions won’t be about writing GHA syntax but understanding its purpose and when it’s appropriate to use for automating tasks (e.g., versioning updates).

Q19: What is the approach for GA5 Q9, YouTube video transcription, to work with Azure/Vercel?

A19: To handle the YouTube video transcription on platforms like Azure or Vercel, you would use youtube-dlp to extract the audio. Then, a publicly available API (like Google Cloud Speech-to-Text) can be used for the actual transcription. This approach is compatible with cloud deployments as it avoids local browser automation.

Q20: For the upcoming End Term Exam, are all 44 questions from the GAs relevant, and will it mostly focus on Vercel deployment?

A20: All 44 questions from the GAs are relevant for the end term exam, as they cover the entire syllabus. Yes, it will heavily focus on Vercel deployment and containerization.

Q21: For GA2 Q8 (Docker), what is the expected deployment environment and configuration?

A21: For Docker questions, you’re expected to deploy your application within a Docker container. This involves setting up the environment variables within the container. If you’re using something like Selenium with Chrome driver, you’d configure the Chrome driver within the Docker container itself.

Q22: For GA4 Q9 (PDF processing), how are different parameters (e.g., ‘English’, ‘24’, ‘which groups’) handled if the PDF file is constant?

A22: The PDF file itself is constant, but the questions about it are parameterized. Your code should pre-process the PDF once (e.g., convert to CSV or DataFrame). Then, for each parameterized question (e.g., asking for data related to ‘English’ or ‘Group 24’), you’ll apply filtering or calculations to your pre-processed data based on those parameters.

Q23: Are there specific questions where the expected output is not plain text but includes HTML or Markdown elements?

A23: Yes, for questions that require HTML or Markdown output, you will need to generate the appropriate string representation for those formats. This might involve using specific libraries for HTML/Markdown generation or escaping characters correctly.

Q24: What is the solution for GA4 Q9 (PDF to CSV) and will the file change for each student?

A24: The solution for GA4 Q9 involves converting the entire PDF to a CSV format within your code. The PDF file itself is constant and will not change for each student, but the specific questions asked about the data within the PDF may vary. You are expected to load the PDF (which is external to the input) into your function and process it.

Q25: In GA4 Q9 (PDF to CSV), how should I handle the file, as it’s not directly provided as input?

A25: The PDF file will not be passed as input. You’ll need to load the PDF directly into your code function, convert it to a CSV (or a DataFrame), and process it from there.

Q26: For GA5 Q9 (YouTube video transcription), will json.loads be used?

A26: Yes, you will likely receive the transcription as a string with escape characters. You’ll use json.loads to convert this string into a manipulable JSON object within your code.

Q27: How can I handle multiple file uploads in a single request for FastAPI?

A27: In FastAPI, you can modify your function to accept a list of UploadFile objects. This allows your API endpoint to receive and process multiple files sent in a single request.