2025-02-27 Week 7 - Session 3 - TDS Jan 25#

Duration: 2h 24m

Here’s an FAQ summary of the tutorial:

Q1: How can I access session recordings if they haven’t been processed by Google yet?

A1: You can access them earlier by going to your calendar, finding the specific date, and clicking on the live session link directly.

Q2: I tried extracting data from the PDF using Tabula, but it was a mess. What went wrong?

A2: Tabula isn’t very compatible with PDFs that have spaces or breaks in their data structure, which is common in many PDFs. It struggles to recognize these gaps and often misinterprets them, leading to formatting issues. pdfplumber is a better alternative as it’s designed to handle such gaps more gracefully.

Q3: I used a web tool like “Adobe PDF to Excel” and it worked in minutes, converting the entire PDF table, including gaps. Why wasn’t this mentioned earlier?

A3: That’s an excellent point! Often, our first thought for PDF data extraction goes to programming libraries like Tabula or pdfplumber. However, simple, extraordinarily accurate web tools exist that can quickly convert PDFs to Excel/CSV, even with complex formatting. It’s a great example of always checking for readily available tools before diving into coding. I usually recommend keeping an “arsenal” of such websites handy.

Q4: After using the web tool, my Excel data looks mostly good, but some entries that should be on one row are split across two. How do I fix this?

A4: This is a common issue. You’ll need to manually inspect your data. In the Excel file, identify the rows where an entry is split (e.g., a name is on one row, and the rest of its data is on the next). You can then copy the relevant part from the second row and paste it into the correct cell in the first row, then delete the extra, now empty, row.

Q5: Once I’ve cleaned the data, should I save it as CSV? What’s the difference between UTF-8 and a normal CSV?

A5: Yes, saving it as a CSV is a good next step. Before saving, it’s best to copy the data from the cleaned Excel sheet and paste it as values into a new sheet to remove any hidden formatting that might cause issues later. Regarding CSV formats, UTF-8 is an encoding standard that uses 8-bit sequences to represent characters. It’s more robust for handling various characters compared to older ASCII-based CSVs. If a specific encoding is required for your task, it will usually be mentioned in the instructions.

Q6: How do I load this cleaned CSV file into a Pandas DataFrame in Python, especially if it specifies a UTF-8 encoding?

A6: You can use import pandas as pd and then pd.read_csv('your_file.csv', encoding='utf-8'). The encoding='utf-8' parameter ensures it’s read correctly according to the specified standard.

Q7: How do I calculate the normalized WQI (Water Quality Index) for each parameter using the formula (P_value - P_min) / (P_max - P_min) * 100?

A7: First, ensure all NaN (Not a Number) values in your original data are filled, for example, with 0 using df.fillna(0). Then, apply the formula for each parameter. I recommend creating a new DataFrame (e.g., param_df) to store these calculated normalized parameter values. This keeps your original data clean. * Self-correction: If (P_max - P_min) is zero, it will cause a division-by-zero error, resulting in inf (infinity). To handle this, after calculating the parameters, identify and replace inf values with np.nan (from NumPy), and then fill these np.nan with 0.

Q8: After calculating normalized parameters, how do I calculate the final weighted WQI score, given the requirements to sort parameters lexographically and assign weights based on specific indices (e.g., “first 7 multiples of 4”)?

A8: 1. Flatten Parameters: First, transform your param_df into a single 1D array (list) using .values.ravel(). Let’s call this flat_params. 2. Sort Flattened Array: Sort flat_params in descending order. 3. Extract Weights: Use list comprehension to extract values from the sorted flat_params at the specified indices (e.g., 4th, 8th, 12th, etc., for multiples of 4). These are your “weights.” 4. Map Weights to Parameters: Create a dictionary that maps each original parameter name (from your param_df columns, sorted lexographically) to its assigned weight from the extracted weights. 5. Calculate Weighted WQI per Parameter: Create new columns in your param_df (e.g., weighted_temp) where each normalized parameter value is multiplied by its corresponding weight from the dictionary. 6. Sum for Final WQI: Sum all these individual weighted_wqi columns for each row to get the final WQI score for each station. 7. Normalize Final WQI: Finally, normalize this total WQI score to a 0-100 scale using MinMaxScaler (from sklearn.preprocessing).

Q9: How do I identify unsafe stations (where WQI is greater than 50) and then get their state and district names using a separate database file?

A9: 1. Identify Unsafe Stations: Filter your DataFrame to find rows where the normalized WQI is greater than 50. Extract the station_code for these rows. 2. Load External Database: Load the database.db file (which contains mapping of pin codes to states and districts) into a Pandas DataFrame using pd.read_sql_table. 3. Lookup State/District: For each unsafe station_code, you’ll need to query the database.db DataFrame to find its corresponding state and district names. You can use string manipulation (e.g., split the station code to extract a pin code) and then perform a lookup.

Q10: What’s the best way to develop and deploy APIs for this kind of data analysis, and how can I efficiently manage my code?

A10: _ API Development: For creating APIs, FastAPI is a great choice. You can create a GET endpoint that takes parameters like station_code (URL-encoded) and returns the required data. _ Deployment: For local hosting, tools like ngrok can be used. For more robust deployment, Docker is recommended, allowing you to package your application and its dependencies into a container. _ Code Management: _ Code Snippets: Keep an “arsenal” of reusable code snippets ready for common tasks (e.g., Fast API methods like GET/POST, Docker commands, data loading/processing, image resizing/blurring using libraries like PIL). Name them efficiently so you can quickly find them. _ Virtual Environments: Always create and use a virtual environment for your projects. Install all necessary libraries within it. _ Open Book Exams: For open-book exams like mock ROIs, leverage tools like CoDium (VS Code extension) or GitHub Copilot. They are powerful AI helpers that can suggest code as you type. Have your API keys ready for these services. _ CORS Middleware: When developing APIs, remember to add CORS (Cross-Origin Resource Sharing) middleware to allow requests from different domains, especially when hosting locally. _ Practice: Consistently practice by attempting mock ROIs. This helps you get comfortable with coordinating your code snippets and managing your time efficiently. The mock ROI tomorrow is a great opportunity for this!

Q11: Will there be another mock ROI tomorrow, and at what time?

A11: Yes, another mock ROI will be released tomorrow (Wednesday) at 7:15 AM IST. Use the opportunity to apply your prepared code snippets.

Q12: Could you show an example of your own code snippets, or how you prepare them?

A12: (Instructor shares their screen with VS Code). I maintain separate files for different functionalities (e.g., LLM_helpers.py, BeautifulSoup_helpers.py, location_helpers.py, FastAPI_helpers.py, Docker_helpers.py). Each file contains functions or snippets related to that specific task. For instance, in BeautifulSoup_helpers.py, I have code for parsing XML files to extract data. In LLM_helpers.py, I have code for calling LLMs like Gemini, processing responses, and handling embeddings/cosine similarity. This modular approach helps me quickly access and use relevant code during time-bound challenges.

Q13: I’m curious, for the Python environment, is it better to do it in VS Code or Google Colab?

A13: I would always recommend VS Code. While Google Colab is good, it has limitations. If your session closes, all your loaded files and progress are lost, and you have to re-upload and re-run everything. In VS Code, your files are local, and you can easily manage your virtual environment. This is crucial when you’re time-bound in an ROI.

Q14: I noticed that in some of your examples, you use a variable named data, but then you use it for something else later. How do you avoid overriding important data?

A14: That’s a very good observation and a common pitfall! It highlights the importance of using unique variable names for different datasets or intermediate results. In my case, I made a mistake by reassigning data to a transformed version, potentially losing the original. When working on ROIs, always ensure you create new variables (e.g., original_data, cleaned_data, processed_data) to store different stages of your data. This prevents accidental overwrites and saves valuable time debugging.Here’s an FAQ summary of the tutorial: