TA Session 7 July: Project Discussion, and Excel Filters, Slope Regression#

Duration: 1h 51m

Here’s an FAQ summary of the TDS tutorial:

Q1: Quiz 2 just finished, and I haven’t had a chance to look at the project yet. Could you give a brief overview of the project now, and then perhaps a more detailed one later when more students are available?

A1: Absolutely! While there are actually more people here today than our last session, I understand that many might have been focused on Quiz 2. I’m happy to dive into the project details now, as that’s exactly what we’re going to cover next. No need to wait for a later session.

Q2: Can I use Excel, specifically pivot tables, to work on the project data?

A2: Yes, you can use any tool you want – there’s no hard and fast rule! However, be aware that the provided data link for scraping was intentionally designed to be challenging for direct import into Excel. Our tests showed that you likely won’t be able to just “save as” or “import” it directly into a clean Excel table. It’s not about the size of the data, but the way we’ve structured the HTML. But once you’ve successfully scraped the data using other methods, you can definitely import it into Excel to perform pivots, filtering, and any other analysis you need.

Q3: My portal specifies a particular state (e.g., Karnataka) and a specific constituency. Should I focus my work only on that subset of data?

A3: Yes, that’s correct. Your specific state and constituency act as your data filter. You need to work on the data pertaining only to Karnataka and the given constituency. Your individual questions will clearly indicate whether they require a state-wise analysis or a constituency-specific one. If there’s ever any ambiguity, please post your question in Discord for clarification.

Q4: The project description mentions “data cleanup to remove inappropriate or noise from records.” What kind of hurdles should I expect during scraping and data cleaning?

A4: This is a crucial instruction! We’ve deliberately introduced noise and errors into the dataset to give you practice for ROE. Expect challenges during scraping because the HTML isn’t perfectly clean. For data cleaning, you’ll need to apply techniques learned in previous weeks. Common hurdles include:

Missing values: Some data points might be absent.
White spaces: Leading or trailing spaces in text fields can cause issues.
Random text/formatting: You might find unusual characters (e.g., ‘C’ in brackets), or numbers formatted as strings that need conversion.
Data type inconsistencies: Columns might not be in their expected data type (e.g., numbers as strings).
Duplicate entries: You might find repeated rows that need handling.
Outliers/wrong entries: There could be values that are statistically unusual or clearly incorrect (e.g., votes for a non-existent constituency, or deliberately placed wrong entries).
“Gap between elections”: If a candidate skipped elections, their vote growth needs to be calculated between their contested elections, not simply assumed as zero for the non-contested years.

My strong advice is to perform thorough data cleanup before attempting to answer any questions. Practicing these skills now will be invaluable for ROE. If you don’t address these, your answers will likely be incorrect.

Q5: What are the definitions for “Vote share,” “Margin percentage,” “Loses the deposit,” and “Absolute majority”?

A5:

Vote share: The number of votes obtained by a candidate divided by the total votes cast in that specific election year, expressed as a percentage.
Margin percentage: The absolute difference in vote share between two contestants. For example, if one gets 25% and another 20%, the margin is 5%.
Loses the deposit: A candidate “loses their deposit” if they fail to secure at least 1/6th (approximately 16.67%) of the total votes polled in their constituency.
Absolute majority: This refers to a candidate securing a vote share that exceeds 50% (i.e., strictly greater than 50%, so 50.00001% would qualify, but exactly 50% would not).

These terms are clearly explained in the project document, but I wanted to make them explicit as they often cause confusion.

Q6: What should I do if a question asks about growth between elections, but a candidate only participated once in the given data, or if the scope (state-level vs. constituency-level) is unclear for a particular question?

A6: For these “edge cases,” your specific question should include notes or instructions on how to handle them (e.g., answer “NA” if no second entry, or “0” for numeric questions). Always check the exact wording of your question for these directives. If you find any questions where such instructions are missing or the scope (state/constituency) is unclear, please post it in Discord immediately. We will investigate against our internal answer script and issue an official clarification if needed.

Q7: Will the questions be the same for all students, or will they be randomized? How will I know the format for my answers (e.g., string, integer, float)?

A7: The questions themselves are randomized per student, and the set of questions you receive might also differ from other students. Your specific state and constituency are also unique to you. For each question, the format for your answer (string, integer, or float) will be explicitly stated at the end of the question prompt. For example, if a question expects a candidate’s name, it will specify “string.”

Q8: For the qualitative assessment, how many graphs/plots are expected for Q1 (showing outliers for candidate votes per election year) and Q2 (showing name length vs. votes)?

A8: For Q1, you’ll create one graph that displays the box or violin plots for all relevant election years for your specific state/constituency. This will allow you to compare outliers across different election years. For Q2 (name length vs. votes), you’ll create one scatter plot that includes all the data points from your relevant subset. It’s about displaying the relationships within a single graph.

Q9: Will explicit rubrics or guidelines be provided for peer reviewers, especially regarding scoring discrepancies observed in previous courses?

A9: Yes, absolutely. We understand the issues that arose in previous courses regarding peer review and inconsistent scoring. We will make every effort to provide explicit rubrics with very clear and objective parameters for peer reviewers. This will aim to ensure fair and consistent grading. We will also ensure clear instructions are given for how to handle issues like file access, as a reviewer’s inability to open a file shouldn’t unfairly penalize the student.

Q10: Since there’s no Quiz 2 for this subject, will Project 1, ROE, and the end-term exam be the only graded components?

A10: That’s correct. For this subject, your graded components will consist of Project 1 (quantitative and qualitative parts), the ROE, and the end-term examination. There will be no Quiz 2.

Q11: Will the project deadlines, especially for ROE and Project 1, be adjusted given the heavy workload this month (multiple projects/OPP/ROE all due around the same time)?

A11: I understand your concern about overlapping deadlines. Personally, I don’t think Project 1 or ROE will be ready for submission by the originally stated dates (e.g., 20th/21st July), just because of backend deployment schedules. I will discuss this with the instructors and advocate for extending the deadlines to ensure you have sufficient time to complete them without excessive pressure. We will ensure fair treatment and that you’re well-informed about any changes.

Q12: I found the house rent data link was difficult to convert to Excel format for slope calculation. How can I easily convert CSV data to Excel or calculate the slope?

A12: This is a good point, and I’ll quickly demonstrate in Excel.

Open CSV in Excel: Simply open the CSV file directly in Excel.
Save as XLSX: Go to File > Save As and select Excel Workbook (*.xlsx) format. This usually improves compatibility and features.
Apply Filters: Use the Data tab and click Filter. This adds small triangles to column headers.
Filter Data: Click the triangle for your desired column (e.g., ‘City’), uncheck ‘Select All’, then select the specific city (e.g., ‘Hyderabad’). Do the same for other filters (e.g., ‘Furnished’).
Calculate Slope: Copy the filtered ‘Rent’ and ‘Size’ columns into a new sheet.
- In an empty cell, type =SLOPE(.
- Select the ‘Rent’ values (this is your known_ys).
- Type a comma ,.
- Select the ‘Size’ values (this is your known_xs).
- Close the parenthesis ) and press Enter. This will give you the slope.
Calculate Absolute Difference: Use =ABS(value1 - value2) to find the absolute difference between slopes (or other values).

Note: Learning different techniques is beneficial. While Pandas is powerful, knowing quick methods in Excel can be very useful for certain tasks.

Q13: What about the Axis=1 definition in Pandas vs. NumPy? I heard it differs.

A13: That’s a very important distinction, and someone actually brought it up in a previous session! You are correct.

In Pandas, axis=1 generally refers to columns. So, an operation with axis=1 will be applied across rows, column by column.
In NumPy (which Pandas often uses internally or alongside), axis=1 generally refers to rows (the second axis of a 2D array, which corresponds to rows in a typical matrix representation). This difference can be a source of significant confusion and hard-to-diagnose bugs when you’re mixing Pandas and NumPy operations or moving between libraries. Always be explicit in your code and double-check the axis parameter’s meaning for the specific function and library you’re using.

General Announcements & Information:

Week 5 (LLM & API Usage): This material is new for the instructors as well, so we’ll all be learning it together. We will guide you through the process, but don’t expect all LLM questions to be answered immediately today, as it’s still very early.
Project Submission: You need to upload a CSV file in UTF-8 format. Ensure the answers are in the exact order specified in the problem statement.
Data Set for Scraping: The provided data set (4.6 lakh rows) was intentionally designed to challenge you with scraping from imperfect HTML, acting as a valuable, low-pressure practice for ROE. We highly recommend you use this opportunity to hone your scraping skills.
Data Cleaning: This is critical. We’ve deliberately introduced errors and “noise” into the data. Failing to clean your data will result in incorrect answers.
Project Deliverables (Quantitative):
- CSV file (UTF-8)
- Answers in exact order of questions
- Specific state/constituency filter per student
- Data scraped from provided link (intentionally challenging)
- Clean data
- Answers must be in specified format (string, integer, float)
Project Deliverables (Qualitative):
- Peer-reviewed.
- Includes Box/Violin plots to show outliers for candidate votes per election year.
- Scatter plots for candidate name length vs. votes.
- Requires clear labeling of all charts and indications of distinct categories.
- Submission link in the portal is for a Google Drive folder containing your work. Ensure access is set to “Anyone with the link can view/open” and that it’s from your college Google ID.
- Explicit rubrics and clear guidelines for peer reviewers will be provided to ensure fair grading.
Graded Assignment 4: Marks will be released this week.
Final Project/ROE Deadlines: I will discuss with instructors to extend these dates given the current workload this month and backend deployment schedules. We will communicate any changes promptly.
Slope Calculation in Excel: It’s fairly straightforward. You can use the SLOPE function directly. (See Q12 for a demo).
Axis=1 (Pandas vs. NumPy): Be aware that axis=1 means “columns” in Pandas but “rows” in NumPy. This is a common point of confusion when coding.
Instructor Availability: We are here to help. If you have questions about ambiguity, data cleaning, or technical issues, please post in Discord or ask during sessions.