Tools in Data Science

Mon, 01 Jan 0001 00:00:00 +0000

Actor Network Visualization#

Find the shortest path between Govinda & Angelina Jolie using IMDb data using Python: networkx or scikit-network.

You’ll learn how to analyze and visualize the social network of actors using IMDb data, covering:

Data Acquisition and Preparation: Learn how to download and work with IMDb’s non-commercial datasets. This includes using Python to read and process large TSV files.
Data Filtering and Wrangling: Understand how to clean and filter large datasets to focus on relevant information, such as specific actor categories (actors/actresses) and movie titles, to make the data more manageable for analysis.
Actor Pairing and Network Creation: Discover how to identify significant collaborations between actors by setting thresholds for the minimum number of films and co-appearances. You will learn to create a network of actors based on these pairings.
Handling Data Inconsistencies: Learn techniques to manage real-world data issues, such as variations in actor names, to ensure the accuracy of your network.
Network Analysis with Python: Get introduced to powerful Python libraries for network analysis like networkx and scikit-network to build, manipulate, and study the structure of complex networks.
Finding the Shortest Path: Learn how to apply graph algorithms to find the shortest path between two actors in the network, demonstrating the “degrees of separation” concept.
Jupyter Notebook for Data Storytelling: See how to use Jupyter Notebooks to document and present a data analysis project, combining code, text, and visualizations to tell a compelling story.
Exporting Results: Learn how to export your findings, such as the actor pairings, into formats like Excel for further use or presentation.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding in the CLI#

Command-line AI coding tools bring the power of large language models directly to your terminal, enabling scripted automation, pipeline integration, and efficient developer workflows. These tools excel at batch processing, git integration, and system-level automation that GUI tools can’t match.

In this module, you’ll learn:

Claude Code CLI: Terminal-first coding agent for interactive development and review
Codex CLI (OpenAI): Local coding agent with sandbox, approvals, CI-friendly exec
GitHub Copilot CLI: GitHub-native terminal agent with policy-aware approvals and MCP extensions
Gemini CLI: Large-context, multimodal agent with non-interactive mode
Simon Willison’s llm: Shell-native AI with UNIX pipeline integration and extensive tooling

Claude Code CLI#

Claude Code is Anthropic’s terminal-first coding agent (npm: @anthropic-ai/claude-code). It runs locally in your project, understands your codebase, and assists with routine tasks, explanations, and git workflows.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Context Engineering#

Context engineering is the systematic approach to providing AI coding assistants with the right information at the right level of detail. Unlike general prompt engineering, code context engineering focuses specifically on code-related workflows, project structure, and development processes.

Effective context engineering transforms AI from a simple code generator into an intelligent development partner that understands your project’s architecture, conventions, and goals.

In this module, you’ll learn:

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding in IDEs#

AI-powered coding assistants have changed how we write code within traditional IDEs and editors. These tools provide intelligent code completion, chat interfaces, and automated refactoring capabilities that dramatically increase developer productivity.

In this module, you’ll learn:

Copy-paste workflows: Fast iteration cycles using web-based AI assistants for quick fixes and code generation
GitHub Copilot: The industry-standard AI pair programmer for VS Code and other editors
Cursor: An AI-native editor with advanced context understanding and agent capabilities
Windsurf: The first agentic IDE with deep codebase awareness and collaborative AI features

Copy-paste workflows#

The simplest and often most effective AI coding workflow involves copying code and errors to web-based AI assistants, then pasting the improved solutions back into your editor. This approach works with any editor and provides access to the latest AI models.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Online#

Modern AI coding tools have transformed how we build software, enabling rapid prototyping and deployment through natural language interfaces. This module covers the mindset shift toward “vibe-coding” and the essential tools for AI-assisted development.

In this module, you’ll learn:

Inline AI coding sandboxes: Use Claude Artifacts, Gemini Canvas, and ChatGPT Canvas for iterative development
Hosted agent workbenches: Leverage Replit Agent, Bolt.new, and Lovable for end-to-end app creation

Artifacts: Inline AI coding sandboxes#

AI artifacts provide side-by-side code and chat interfaces that revolutionize how we iterate on applications. These tools show visible diffs, run code in real-time, and allow for conversational editing.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Strategies#

Strategic approaches to AI-assisted software development have evolved rapidly in 2025. Unlike simple code completion, these methodologies focus on sustainable, team-oriented workflows that maximize AI productivity while maintaining code quality and developer control.

Modern AI coding strategies combine human expertise with AI capabilities through structured patterns, parallel processing, and iterative feedback loops. The most effective approaches treat AI as a collaborative partner rather than a replacement, establishing clear boundaries and verification processes.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Tests#

Testing AI-generated code is about establishing fast, trustworthy feedback loops before the code lands in main. The goal is not perfect certainty—it is a reliable signal that the patch does what it should, fails when it must, and keeps shipping velocity high.

In this module, you’ll learn:

Scorecard-driven evaluation loops: Track AI agent success with automated suites and selectors
Premortem test-first flows: Lock in failing tests before asking an agent to code
Python-first fuzzing and mutation: Catch almost-right code with Hypothesis and mutmut
Agent-powered smoke tests: Let Playwright bots guard critical user journeys

Build evaluation loops#

Benchmarks stop vibe-coded fixes from quietly regressing. Start with a thin scorecard that runs on every candidate patch, then expand.

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Tools#

AI coding agents perform best when you stage a predictable toolbox around them. The goal is not to spray every binary into the sandbox, but to focus on fast, auditable utilities that agents can compose for data prep, testing, review, and automation. Keep the happy path linear: expose tools, document how to call them, and wire safe defaults before inviting an agent into the repo.

In this module, you’ll learn:

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding#

AI-assisted development pairs human judgment with generative models across every workspace. This module shows how to set up productive collaborations, feed models the right project context, and ship code you trust.

You’ll explore:

Creative prompting that shapes tone and flow through vibe coding sessions and scenario planning.
Environment-specific tooling for working in browser editors, IDE extensions, and CLI companions without losing momentum.
Context engineering practices that stage repositories, snippets, and docs so models stay grounded in your codebase.
Quality safeguards that weave in automated tests, structured reviews, and targeted tool calls to keep AI-written code production ready.

By the end you will know when to lean on AI, how to steer it with precise context, and the guardrails that turn generated snippets into maintainable features.

Mon, 01 Jan 0001 00:00:00 +0000

Archived content#

Videos#

IITM BS Degree - Diploma Level Orientation#

Tools in Data Science Orientation#

Tools in Data Science Live Sessions#

Tools in Data Science Playlist#

Tools in Data Science Course Playlist, 66 videos

Optional: Parse & clean PDF files with Tabula#

Mon, 01 Jan 0001 00:00:00 +0000

Base 64 Encoding#

Base64 is a method to convert binary data into ASCII text. It’s essential when you need to transmit binary data through text-only channels or embed binary content in text formats.

Watch this quick explanation of how Base64 works (3 min):

Here’s how it works:

It takes 3 bytes (24 bits) and converts them into 4 ASCII characters
… using 64 characters: A-Z, a-z, 0-9, + and / (padding with = to make the length a multiple of 4)
There’s a URL-safe variant of Base64 that replaces + and / with - and _ to avoid issues in URLs
Base64 adds ~33% overhead (since every 3 bytes becomes 4 characters)

Common Python operations with Base64:

Mon, 01 Jan 0001 00:00:00 +0000

Terminal: Bash#

UNIX shells are the de facto standard in the data science world and Bash is the most popular. This is available by default on Mac and Linux.

On Windows, install Git Bash or WSL to get a UNIX shell.

Watch this video to install WSL (12 min).

Watch this video to understand the basics of Bash and UNIX shell commands (75 min).

Mon, 01 Jan 0001 00:00:00 +0000

BBC Weather location ID with Python#

You’ll learn how to get the location ID of any city from the BBC Weather API – as a precursor to scraping weather data – covering:

Understanding API Calls: Learn how backend API calls work when searching for a city on the BBC weather website.
Inspecting Web Interactions: Use the browser’s inspect element feature to track API calls and understand the network activity.
Extracting Location IDs: Identify and extract the location ID from the API response using Python.
Using Python Libraries: Import and use requests, json, and urlencode libraries to make API calls and process responses.
Constructing API URLs: Create structured API URLs dynamically with constant prefixes and query parameters using urlencode.
Building Functions: Develop a Python function that accepts a city name, constructs the API call, and returns the location ID.

To open the browser Developer Tools on Chrome, Edge, or Firefox, you can:

Mon, 01 Jan 0001 00:00:00 +0000

TDS Bridge Bootcamp — Combined Checklist/Syllabus#

Day 1 Post-Session Checklist#

Setup Day#

I can open a Linux terminal and recognize the shell prompt (user@machine:~$)
I can run python --version, git --version, and uv --version without errors
I can open VS Code connected to WSL/Linux using code .
I know what ~ means and can navigate to my home directory using cd ~
I can create folders and files using mkdir and touch, and list them with ls
I know the difference between .xyz, ./xyz, and ../xyz
I know that in Linux, everything is a file
I can navigate to the C and D drives within WSL
I understand the difference between > (overwrite) and >> (append)
I have a GitHub account and have created the tds-bootcamp repository

Day 2 Post-Session Checklist#

Linux & Shell Essentials#

I understand what PATH is and why commands like python work without full paths
I can navigate the filesystem without clicking — using cd, ls, and pwd only
I can read, search, and inspect files using cat, head, tail, grep, and wc
I can edit a file using nano (open, edit, save, exit)
I understand pipes (|) and redirection (>, >>, 2>) and can chain commands
I can set an environment variable in .bashrc and apply it with source ~/.bashrc
I know the difference between export VAR=value (available to child processes) and just VAR=value (shell-local)

Day 3 Post-Session Checklist#

VS Code + Python Projects with uv#

I can run a Python script using uv run script.py without setting up a local virtual environment
I know where the temporary virtual environment is created when running the uv add --script script.py pandas command followed by uv run script.py
I can create a new Python project using uv init and understand what pyproject.toml is used for
I can add a dependency (e.g., requests) using uv add and see it reflected in the lockfile
I can create a traditional virtual environment using uv venv and know when to use it
I understand why installing packages globally with pip install is a bad habit
I can open a project in VS Code, select the correct Python interpreter, and run code from the integrated terminal
I know the difference between a .py script and a .ipynb notebook, and when to use each

Day 4 Post-Session Checklist#

HTTP, APIs & Chrome DevTools#

I know the 5 core HTTP methods (GET, POST, PUT, PATCH, DELETE) and what each is used for
I can read a status code and know what went wrong — e.g., 401 vs. 403 vs. 404 vs. 500
I can open Chrome DevTools Network tab, find a request, and inspect its headers, payload, and response
I can copy a browser request as a cURL command and run it in the terminal
I can change the User-Agent in the browser and see the change in the Network tab
I can use curl to make a GET request with query parameters and a POST request with a JSON body
I have a running FastAPI app with at least two endpoints (GET /health and POST /echo)
I can test my API using the Swagger UI at /docs and via curl from the terminal

Day 5 Post-Session Checklist#

Git & GitHub Workflow#

I have set up the basic Git configuration using git config --global user.name "Your Name", git config --global user.email "your.email@example.com", and set the default branch as main using git config --global init.defaultBranch main
I know GitHub allows only one user account per person, so I have merged my accounts (IITM and personal) into a single unified account
I understand the three states of a file in Git: working tree → staging → committed
I can run the daily workflow commands git status, git diff, and git log and know what each shows
I know how .gitignore works and how to use it to ignore files and folders that should not be pushed to the remote repository (e.g., venv, __pycache__, .env)
I can make a commit: git add → git commit -m "message" → git push
I know what origin and main are and can explain them in one sentence each
I can set up SSH key authentication and push to GitHub without entering a password
I can create an annotated tag (git tag -a v0.1.0) and push it to GitHub
I can write a meaningful commit message (not “fixed stuff” or “final.py”)

Mon, 01 Jan 0001 00:00:00 +0000

Day 1 — File Structure: Windows vs Linux#

Goal: Understand how files are organized on Linux vs Windows so you never get confused about “where my file went” or “why can’t Python find my data.”

The Big Picture#

On Windows, your files live under drive letters:

C:\Users\alice\Documents\project\data.csv
D:\backup\photos\

On Linux, everything starts from a single root /:

/home/alice/projects/data.csv
/tmp/scratch.txt

There are no drive letters on Linux. Every file on the entire system hangs off /.

Mon, 01 Jan 0001 00:00:00 +0000

Day 1 — Installation Guide#

Goal: Get a fully working development environment — terminal, editor, Git, Python, and uv — so you can start coding from Day 2 onwards with zero friction.

What You Need Installed by the End#

Tool	Purpose	Verify with
Ubuntu Linux (native or WSL2)	Your operating system / shell environment	`uname -a`
VS Code	Code editor	`code --version`
Git	Version control	`git --version`
Python 3.11+	Programming language	`python3 --version`
uv	Python package/project manager	`uv --version`

Option A — Native Ubuntu Linux#

If you are already running Ubuntu (22.04 or later), you just need to install the tools.

Mon, 01 Jan 0001 00:00:00 +0000

Day 1 — Setup Day: Complete Notes#

TDS Bridge Bootcamp#

Goal of Day 1: Before you write a single line of data science code, you need a working environment you understand. This day is not just “install stuff” — it is about building a mental model of where your files live, how your computer runs programs, and why things fail when they do.

Table of Contents#

1. Linux vs Windows — The Mental Model#

1.1 Why This Matters#

Most Python tutorials assume you just “open a terminal and type.” But if you are on Windows, there are two completely different worlds on your machine: the Windows world and the Linux world (via WSL2). Confusing them is the #1 source of beginner pain.

Mon, 01 Jan 0001 00:00:00 +0000

Day 1 — Path Reading#

Goal: Master absolute paths, relative paths, and special shortcuts (~, ., ..) so you can always tell the system exactly where a file is — or figure out where you are.

What Is a Path?#

A path is the address of a file or folder on your system. Just like a street address tells you where a building is, a path tells the computer where to find a file.

Mon, 01 Jan 0001 00:00:00 +0000

Day 1 Quiz & Exercises#

TDS Bridge Bootcamp — Setup Day#

Instructions: Attempt the MCQs first (no Googling). Then do the exercises in your terminal. The goal is to think before you type.

Part A — 30 Multiple Choice Questions#

Q1. You open a terminal and see this prompt: alice@laptop:~$. What does the ~ mean?

A) The terminal is connected to the internet
B) You are in the root directory /
C) You are in your home directory
D) You are running as administrator

Answer

C — ~ is the shorthand for your home directory.

Mon, 01 Jan 0001 00:00:00 +0000

Day 2 — Basic Script Writing#

Goal: Write your first shell scripts — from simple one-liners to multi-step automation scripts with variables, conditionals, and loops.

What Is a Shell Script?#

A shell script is a text file containing a series of commands that the shell (bash) executes in order. Instead of typing 10 commands one by one, you put them in a file and run it once.

# Instead of typing these every day:
cd ~/projects/tds
git pull
python3 main.py
echo "Done!"

# Put them in a script and run: ./daily.sh

Your First Script#

Step 1: Create the file#

nano ~/hello.sh

Step 2: Write the script#

#!/bin/bash
# My first shell script

echo "Hello, World!"
echo "Today is $(date)"
echo "You are: $(whoami)"
echo "You are in: $(pwd)"

Step 3: Make it executable#

chmod +x ~/hello.sh

Step 4: Run it#

~/hello.sh
# Hello, World!
# Today is Mon Jan 15 10:30:00 IST 2026
# You are: alice
# You are in: /home/alice

The Shebang Line — `#!/bin/bash`#

The first line of every script should be the shebang:

Mon, 01 Jan 0001 00:00:00 +0000

Day 2 — Linux & Shell Essentials: The Survival Toolkit#

TDS Bridge Bootcamp#

Goal of Day 2: Move from “I can type a command if someone tells me what to type” → “I can navigate, manage files, edit text, and write simple shell scripts independently.” These are the skills you will use every single day as a data practitioner.

Table of Contents#

1. Navigating the Filesystem#

1.1 `pwd` — Where Am I?#

pwd
# /home/alice/projects/tds

pwd = Print Working Directory. Always run this if you are lost.

Mon, 01 Jan 0001 00:00:00 +0000

Day 2 Quiz & Exercises#

TDS Bridge Bootcamp — Linux & Shell Essentials#

Instructions: Attempt the MCQs without help first. Then do the terminal exercises. The exercises build on each other — do them in order.

Part A — 20 Multiple Choice Questions#

Q1. You type python in the terminal and get “command not found”. Your friend on the same machine has no problem. What is the most likely cause?

Mon, 01 Jan 0001 00:00:00 +0000

Goal: Move around the Linux filesystem with confidence using pwd, ls, cd, and tree — so you never feel lost in the terminal.

Where Am I? — `pwd`#

The first command to learn. Run it whenever you are lost:

pwd
# /home/alice/projects/tds

pwd = Print Working Directory. It tells you the absolute path of where your terminal is right now.

Every command you run operates relative to this location. If you run ls, it lists what’s in this directory. If you run python3 script.py, it looks for script.py here.

Mon, 01 Jan 0001 00:00:00 +0000

Day 2 — File Operations: touch, mkdir, rm, cp, mv#

Goal: Master the essential file management commands so you can create, copy, move, rename, and delete files and directories entirely from the terminal.

Creating Files — `touch`#

touch creates an empty file (or updates the timestamp of an existing file).

touch notes.txt # create one empty file
touch file1.txt file2.txt file3.txt # create multiple files
touch ~/projects/README.md # create in a specific location

If the file already exists, touch updates its “last modified” time without changing content.

Mon, 01 Jan 0001 00:00:00 +0000

Day 3 Quiz & Exercises#

TDS Bridge Bootcamp — VS Code + Python Projects with uv#

Instructions: Attempt MCQs first, then do the exercises in order in your terminal.

Part A — 20 Multiple Choice Questions#

Q1. You run pip install pandas without a virtual environment. A week later you start a new project that needs a different version of pandas. What is the likely problem?

A) pip won’t allow installing pandas twice
B) Both projects share the same global pandas — changing the version for one breaks the other
C) pip automatically creates separate environments per project
D) You’ll need to uninstall Python and reinstall it

Answer

B — Both projects share the same global pandas — changing the version for one breaks the other

Mon, 01 Jan 0001 00:00:00 +0000

Day 3 — UV Workflow#

Goal: Learn the day-to-day uv commands to create projects, manage dependencies, run scripts, and keep your Python environment reproducible.

What Is UV?#

uv is an extremely fast Python package and project manager built by Astral (the same team behind ruff). It replaces several tools:

Traditional tool	What it does	uv equivalent
`pip`	Install packages	`uv pip install` / `uv add`
`venv`	Create virtual environments	`uv venv`
`pyenv`	Manage Python versions	`uv python install`
`pipx`	Run CLI tools	`uvx`
`pip-tools`	Lock dependencies	`uv lock`

Why uv? It’s 10-100x faster than pip. A full dependency install that takes pip 30 seconds takes uv under 1 second.

Mon, 01 Jan 0001 00:00:00 +0000

Day 3 — Virtual Environments#

Goal: Understand why virtual environments exist, how they work, and how to create and use them — so your projects never break each other’s dependencies.

The Problem: Why Virtual Environments?#

Imagine you have two Python projects:

Project A — needs pandas==1.5.0
Project B — needs pandas==2.1.0

If you install both globally (system-wide), only one version of pandas can exist at a time. Installing one breaks the other.

Mon, 01 Jan 0001 00:00:00 +0000

Day 3 — VS Code Setup#

Goal: Set up VS Code as your primary code editor with the right extensions, terminal integration, and settings so you can write, run, and debug Python efficiently.

What Is VS Code?#

Visual Studio Code (VS Code) is a free, lightweight code editor by Microsoft. It’s the most popular editor among developers because:

It works on Windows, macOS, and Linux
It has a built-in terminal
It has thousands of extensions for every language
It connects to WSL2, remote servers, and containers
It has great Git integration built-in

VS Code is not the same as Visual Studio (which is a full IDE). VS Code is lighter and more flexible.

Mon, 01 Jan 0001 00:00:00 +0000

Day 4 — API Testing with curl#

Goal: Use curl to send HTTP requests from the terminal — GET, POST, headers, JSON bodies — so you can test and debug APIs without leaving the command line.

What Is curl?#

curl (Client URL) is a command-line tool for making HTTP requests. It comes pre-installed on most Linux systems.

curl --version
# curl 8.x.x ...

Think of curl as a browser for your terminal — but instead of rendering a web page, it shows you the raw HTTP response.

Mon, 01 Jan 0001 00:00:00 +0000

Day 4 — Chrome DevTools#

Goal: Use Chrome DevTools to inspect network requests, debug API calls, and understand what happens behind the scenes when you load a web page or call an API.

What Are Chrome DevTools?#

Chrome DevTools is a set of developer tools built into Google Chrome. It lets you:

Inspect HTML/CSS elements
Monitor network requests (API calls, file downloads)
Debug JavaScript
Analyze performance
View cookies and storage

For this course, the Network tab is the most important tool.

Mon, 01 Jan 0001 00:00:00 +0000

Day 4 — HTTP Fundamentals#

Goal: Understand how the web works at the protocol level — HTTP methods, status codes, headers, and request/response structure — so you can debug APIs and build web services.

What Is HTTP?#

HTTP (HyperText Transfer Protocol) is the language that browsers and servers speak. Every time you:

Open a web page
Submit a form
Fetch data from an API
Upload a file

…an HTTP request goes out and an HTTP response comes back.

Mon, 01 Jan 0001 00:00:00 +0000

Day 4 Quiz & Exercises#

TDS Bridge Bootcamp — HTTP, APIs & Chrome DevTools#

Instructions: Attempt MCQs first. Then run the exercises — you’ll need your terminal, a browser with DevTools, and your FastAPI app running.

Part A — 20 Multiple Choice Questions#

Q1. You click “Submit” on a login form. The username and password are sent to the server. Which HTTP method is almost certainly being used?

Mon, 01 Jan 0001 00:00:00 +0000

Day 5 — Git Basic Flow#

Goal: Understand the core Git mental model — working tree, staging area, and commits — so you know exactly what happens at each step and never feel lost.

What Is Git?#

Git is a version control system. It tracks changes to your files over time, letting you:

See what changed, when, and by whom
Undo mistakes by going back to any previous version
Work on features in parallel without breaking things
Collaborate with others without overwriting each other’s work

Every professional developer uses Git daily. It is non-negotiable.

Mon, 01 Jan 0001 00:00:00 +0000

Goal: Set up SSH key authentication for GitHub so you can push and pull without entering passwords, and understand how public-key cryptography works in practice.

Why SSH?#

When you push code to GitHub, GitHub needs to verify you are who you say you are. There are two main ways:

Method	How it works	Pros	Cons
HTTPS + Token	Password/token sent with each request	Easy setup	Must manage tokens
SSH Keys	Cryptographic key pair	No password needed, more secure	One-time setup is longer

We recommend SSH — once set up, it just works. No tokens to manage or expire.

Mon, 01 Jan 0001 00:00:00 +0000

Day 5 — Git: init, add, commit#

Goal: Master the hands-on commands for creating repositories, staging changes, and making commits — the commands you’ll use every single day.

`git init` — Create a New Repository#

# Create a project directory and initialize Git:
mkdir my-project
cd my-project
git init
# Initialized empty Git repository in /home/you/my-project/.git/

What `git init` creates#

ls -la .git/
# HEAD ← points to current branch
# config ← repository-level settings
# hooks/ ← scripts that run on events (commit, push, etc.)
# objects/ ← where Git stores file contents and commits
# refs/ ← branch and tag pointers

You never need to touch .git/ directly. Git commands manage it for you.

Mon, 01 Jan 0001 00:00:00 +0000

Day 5 Quiz & Exercises#

TDS Bridge Bootcamp — Git & GitHub Workflow#

Instructions: Attempt MCQs first. Then do the exercises in order — they build toward your final submission.

Part A — 20 Multiple Choice Questions#

Q1. You edit a file app.py. You run git status and see it listed under “Changes not staged for commit”. What does this mean?

A) The file has been committed to Git
B) The file has been staged but not committed
C) The file was modified but not yet staged
D) The file is new and Git doesn’t know about it

Answer

C — The file was modified but not yet staged

Mon, 01 Jan 0001 00:00:00 +0000

Cleaning Data with OpenRefine#

This session covers the use of OpenRefine for data cleaning, focusing on resolving entity discrepancies:

Data Upload and Project Creation: Import data into OpenRefine and create a new project for analysis.
Faceting Data: Use text facets to group similar entries and identify frequency of address crumbs.
Clustering Methodology: Apply clustering algorithms to merge similar entries with minor differences, such as punctuation.
Manual and Automated Clustering: Learn to merge clusters manually or in one go, trusting the system’s clustering accuracy.
Entity Resolution: Clean and save the data by resolving multiple versions of the same entity using Open Refine.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Notebooks: Google Colab#

Google Colab is a free, cloud-based Jupyter notebook environment that’s become indispensable for data scientists and ML practitioners. It’s particularly valuable because it provides free access to GPUs and TPUs, and for easy sharing of code and execution results.

While Colab is excellent for prototyping and learning, its free tier has limitations - notebooks time out after 12 hours, and GPU access can be inconsistent.

Learn how to mount Google Drive for persistent storage, manage dependencies with !pip install commands, as these are common pain points when getting started.

Mon, 01 Jan 0001 00:00:00 +0000

Converting HTML to Markdown#

When working with web content, converting HTML files to plain text or Markdown is a common requirement for content extraction, analysis, and preservation. For example:

Content analysis: Extract clean text from HTML for natural language processing
Data mining: Strip formatting to focus on the actual content
Offline reading: Convert web pages to readable formats for e-readers or offline consumption
Content migration: Move content between different CMS platforms
SEO analysis: Extract headings, content structure, and text for optimization
Archive creation: Store web content in more compact, preservation-friendly formats
Accessibility: Convert content to formats that work better with screen readers

This tutorial covers both converting existing HTML files and combining web crawling with HTML-to-text conversion in a single workflow – all using the command line.

Mon, 01 Jan 0001 00:00:00 +0000

Converting PDFs to Markdown#

PDF documents are ubiquitous in academic, business, and technical contexts, but extracting and repurposing their content can be challenging. This tutorial explores various command-line tools for converting PDFs to Markdown format, with a focus on preserving structure and formatting suitable for different use cases, including preparation for Large Language Models (LLMs).

Use Cases:

LLM training and fine-tuning: Create clean text data from PDFs for AI model training
Knowledge base creation: Transform PDFs into searchable, editable markdown documents
Content repurposing: Convert academic papers and reports for web publication
Data extraction: Pull structured content from PDF documents for analysis
Accessibility: Convert PDFs to more accessible formats for screen readers
Citation and reference management: Extract bibliographic information from academic papers
Documentation conversion: Transform technical PDFs into maintainable documentation

PyMuPDF4LLM#

PyMuPDF4LLM is a specialized component of the PyMuPDF library that generates Markdown specifically formatted for Large Language Models. It produces high-quality markdown with good preservation of document structure. It’s specifically optimized for producing text that works well with LLMs, removing irrelevant formatting while preserving semantic structure. Requires PyTorch, which adds dependencies but enables more advanced processing capabilities.

Mon, 01 Jan 0001 00:00:00 +0000

Correlation with Excel#

You’ll learn to calculate and interpret correlations using Excel, covering:

Enabling the Data Analysis Tool Pack: Steps to enable the Excel data analysis tool pack.
Correlation Analysis: Understanding statistical association between variables.
Creating a Correlation Matrix: Steps to generate and interpret a correlation matrix.
Scatterplots and Trendlines: Plotting data and adding trend lines to visualize correlations.
Analyzing Results: Comparing correlation coefficients and understanding their implications.
Insights and Further Analysis: Interpreting scatterplots and planning further analysis for deeper insights.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

CORS (Cross-Origin Resource Sharing) is a security mechanism that controls how web browsers handle requests between different origins (domains, protocols, or ports). Data scientists need CORS for APIs serving data or analysis to a browser on a different domain.

Watch this practical explanation of CORS (3 min):

Key CORS concepts:

Same-Origin Policy: Browsers block requests between different origins by default
CORS Headers: Server responses must include specific headers to allow cross-origin requests
Preflight Requests: Browsers send OPTIONS requests to check if the actual request is allowed
Credentials: Special handling required for requests with cookies or authentication

If you’re exposing your API with a GET request publicly, the only thing you need to do is set the HTTP header Access-Control-Allow-Origin: *.

Mon, 01 Jan 0001 00:00:00 +0000

Crawling with the CLI#

Since websites are a common source of data, we often download entire websites (crawling) and then process them offline.

Web crawling is essential in many data-driven scenarios:

Data mining and analysis: Gathering structured data from multiple pages for market research, competitive analysis, or academic research
Content archiving: Creating offline copies of websites for preservation or backup purposes
SEO analysis: Analyzing site structure, metadata, and content to improve search rankings
Legal compliance: Capturing website content for regulatory or compliance documentation
Website migration: Creating a complete copy before moving to a new platform or design
Offline access: Downloading educational resources, documentation, or reference materials for use without internet connection

The most commonly used tool for fetching websites is wget. It is pre-installed in many UNIX distributions and easy to install.

Mon, 01 Jan 0001 00:00:00 +0000

CSS Selectors#

CSS selectors are patterns used to select and style HTML elements on a web page. They are fundamental to web development and data scraping, allowing you to precisely target elements for styling or extraction.

For data scientists, understanding CSS selectors is crucial when:

Web scraping with tools like Beautiful Soup or Scrapy
Selecting elements for browser automation with Selenium
Styling data visualizations and web applications
Debugging website issues using browser DevTools

Watch this comprehensive introduction to CSS selectors (20 min):

Mon, 01 Jan 0001 00:00:00 +0000

Data Aggregation in Excel#

You’ll learn data aggregation and visualization techniques in Excel, covering:

Data Cleanup: Remove empty columns and rows with missing values.
Creating Excel Tables: Convert raw data into tables for easier manipulation and formula application.
Date Manipulation: Extract week, month, and year from date columns using Excel functions (WEEKNUM, TEXT).
Color Scales: Apply color scales to visualize clusters and trends in data over time.
Pivot Tables: Create pivot tables to aggregate data by location and date, summarizing values weekly and monthly.
Sparklines: Use sparklines to visualize trends within pivot tables, making data patterns more apparent.
Data Bars: Implement data bars for graphical illustrations of numerical columns, showing trends and waves.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Analysis with Datasette#

Datasette is an open-source tool for exploring and publishing data. Created by Simon Willison, it turns any SQLite database into an instant web interface with powerful exploration features, JSON APIs, and visualization capabilities—all without writing code.

Unlike traditional database tools that require SQL knowledge upfront, Datasette provides an interactive interface for exploring data through faceting, filtering, and full-text search. It’s particularly powerful for data journalism, analysis workflows, and sharing datasets with non-technical audiences.

Mon, 01 Jan 0001 00:00:00 +0000

Data Analysis with DuckDB#

You’ll learn how to perform data analysis using DuckDB and Pandas, covering:

Parquet for Data Storage: Understand why Parquet is a faster, more compact, and better-typed storage format compared to CSV, JSON, and SQLite.
DuckDB Setup: Learn how to install and set up DuckDB, along with integrating it into a Jupyter notebook environment.
File Format Comparisons: Compare file formats by speed and size, observing the performance difference between saving and loading data in CSV, JSON, SQLite, and Parquet.
Faster Queries with DuckDB: Learn how DuckDB uses parallel processing, columnar storage, and on-disk operations to outperform Pandas in speed and memory efficiency.
SQL Query Execution in DuckDB: Run SQL queries directly on Parquet files and Pandas DataFrames to compute metrics such as the number of unique flight routes delayed by certain time intervals.
Memory Efficiency: Understand how DuckDB performs analytics without loading entire datasets into memory, making it highly efficient for large-scale data analysis.
Mixing DuckDB and Pandas: Learn to interleave DuckDB and Pandas operations, leveraging the strengths of both tools to perform complex queries like correlations and aggregations.
Ranking and Filtering Data: Use SQL and Pandas to rank arrival delays by distance and extract key insights, such as the earliest flight arrival for each route.
Joining Data: Create a cost analysis by joining datasets and calculating total costs of flight delays, demonstrating DuckDB’s speed in joining and aggregating large datasets.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Analysis with Python#

You’ll learn practical data analysis techniques in Python using Pandas, covering:

Reading Parquet Files: Utilize Pandas to read Parquet file formats for efficient data handling.
Dataframe Inspection: Methods to preview and understand the structure of a dataset.
Pivot Tables: Creating and interpreting pivot tables to summarize data.
Percentage Calculations: Normalize pivot table values to percentages for better insights.
Correlation Analysis: Calculate and interpret correlation between variables, including significance testing.
Statistical Significance: Use statistical tests to determine the significance of observed correlations.
Datetime Handling: Extract and manipulate date and time information from datetime columns.
Data Visualization: Generate and customize heat maps to visualize data patterns effectively.
Leveraging AI: Use ChatGPT to generate and refine analytical code, enhancing productivity and accuracy.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Analysis with SQL#

You’ll learn how to perform data analysis using SQL (via Python), covering:

Database Connection: How to connect to a MySQL database using SQLAlchemy and Pandas.
SQL Queries: Execute SQL queries directly from a Python environment to retrieve and analyze data.
Counting Rows: Use SQL to count the number of rows in a table.
User Activity Analysis: Query and identify top users by post count.
Post Concentration: Determine if a small percentage of users contribute the majority of posts using SQL aggregation.
Correlation Calculation: Calculate the Pearson correlation coefficient between user attributes such as age and reputation.
Regression Analysis: Compute the regression slope to understand the relationship between views and reputation.
Handling Large Data: Perform calculations on large datasets by fetching aggregated values from the database rather than entire datasets.
Statistical Analysis in SQL: Use SQL as a tool for statistical analysis, demonstrating its power beyond simple data retrieval.
Leveraging AI: Use ChatGPT to generate SQL queries and Python code, enhancing productivity and accuracy.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data analysis#

Data Analysis: Introduction Podcast by NotebookLM

Mon, 01 Jan 0001 00:00:00 +0000

Data Cleansing in Excel#

You’ll learn basic but essential data cleaning techniques in Excel, covering:

Find and Replace: Use Ctrl+H to replace or remove specific terms (e.g., removing “[more]” from country names).
Changing Data Formats: Convert columns from general to numerical format.
Removing Extra Spaces: Use the TRIM function to clean up unnecessary spaces in text.
Identifying and Removing Blank Cells: Highlight and delete entire rows with blank cells using the “Go To Special” function.
Removing Duplicates: Use the “Remove Duplicates” feature to eliminate duplicate entries, demonstrated with country names.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation in DuckDB#

DuckDB’s SQL engine can handle large files quickly. Below are common cleaning tasks using the DuckDB CLI.

Create a Sample Dataset#

Let’s create a sample dataset that mimics real business data patterns - incomplete customer records, time-series orders, and regional variations. Before working with messy production data, you need a controlled environment to test data cleaning techniques. This sample represents common e-commerce scenarios: missing customer info (20% of orders), seasonal patterns (15-day cycles), and geographic segmentation that drive business decisions like inventory placement and marketing campaigns.

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation in the Editor#

You’ll learn how to use a text editor Visual Studio Code to process and clean data, covering:

Format JSON files
Find all and multiple cursors to extract specific fields
Sort lines
Delete duplicate lines
Replace text with multiple cursors

Here are the links used in the video:

City-wise product sales JSON

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation in the Shell#

You’ll learn how to use UNIX tools to process and clean data, covering:

curl (or wget) to fetch data from websites.
gzip (or xz) to compress and decompress files.
wc to count lines, words, and characters in text.
head and tail to get the start and end of files.
cut to extract specific columns from text.
uniq to de-duplicate lines.
sort to sort lines.
grep to filter lines containing specific text.
sed to search and replace text.
awk for more complex text processing.

Data preparation in the shell - Notebook

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation#

Data preparation is crucial because raw data is rarely perfect.

It often contains errors, inconsistencies, or missing values. For example, marks data may have ‘NA’ or ‘absent’ for non-attendees, which you need to handle.

This section teaches you how to clean up data, convert it to different formats, aggregate it if required, and get a feel for the data before you analyze.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Sourcing#

Before you do any kind of data science, you obviously have to get the data to be able to analyze it, visualize it, narrate it, and deploy it. And what we are going to cover in this module is how you get the data.

There are three ways you can get the data.

The first is you can download the data. Either somebody gives you the data and says download it from here, or you are asked to download it from the internet because it’s a public data source. But that’s the first way—you download the data.
The second way is you can query it from somewhere. It may be on a database. It may be available through an API. It may be available through a library. But these are ways in which you can selectively query parts of the data and stitch it together.
The third way is you have to scrape it. It’s not directly available in a convenient form that you can query or download. But it is, in fact, on a web page. It’s available on a PDF file. It’s available in a Word document. It’s available on an Excel file. It’s kind of structured, but you will have to figure out that structure and extract it from there.

In this module, we will be looking at the tools that will help you either download from a data source or query from an API or from a database or from a library. And finally, how you can scrape from different sources.

Mon, 01 Jan 0001 00:00:00 +0000

Data Storytelling with LLMs#

Large Language Models (LLMs) can help create compelling data stories by assisting at every step of the data-to-story value chain:

Data Engineering (scraping, cleaning)
Data Analysis (modeling, insights)
Data Visualization (charts, narratives)

Watch this talk (30m) on data storytelling with LLMs

Prerequisites#

To follow this tutorial, you’ll need:

ChatGPT Plus ($20/month) - recommended for better models
Basic understanding of data analysis concepts

Optional but useful:

Mon, 01 Jan 0001 00:00:00 +0000

Data Storytelling#

Transcript#

Once you’ve analyzed your data and put it together into charts, you still want to communicate it in a way that people will remember, that people will share. And that’s where storytelling comes in. What we’re going to look at in this module are ways of telling stories.

There are many ways in which you can narrate a story, and one way is just to show the numbers. This in itself can be powerful. What you see here, for example, is a correlation matrix which has conditional formatting on Excel that will help you understand how two securities are correlated. And it’s pretty easy to see that, for example, the Pakistani rupee is negatively correlated with many of the securities. Just the numbers can give people an understanding of what’s happening behind the scenes and convey a story.

Mon, 01 Jan 0001 00:00:00 +0000

Data Transformation in Excel#

You’ll learn data transformation techniques in Excel, covering:

Calculating Ratios: Compute metro area to city area and metro population to city population ratios.
Using Pivot Tables: Create pivot tables to aggregate data and identify outliers.
Filtering Data: Apply filters in pivot tables to analyze specific subsets of data.
Counting Data Occurrences: Use pivot tables to count the frequency of specific entries.
Creating Charts: Generate charts from pivot table data to visualize distributions and outliers.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Data Visualization with ChatGPT#

ChatGPT and other Large Language Models (LLMs) can help create compelling data visualizations by:

Finding and analyzing datasets
Generating visualization code
Improving visual design
Creating data stories

Watch this workshop (2h) on creating data visualizations with ChatGPT

Prerequisites#

To follow this tutorial, you’ll need:

Gemini (free) - good for processing images and video
A ChatGPT Plus subscription ($20/month) - recommended for access to advanced models and coding capabilities
GitHub account - for publishing visualizations
Basic familiarity with HTML/CSS/JavaScript

Other useful but optional tools include:

Mon, 01 Jan 0001 00:00:00 +0000

Data Visualization with Seaborn#

Seaborn is a data visualization library for Python. It’s based on Matplotlib but a bit easier to use, and a bit prettier.

This video tutorial provides a comprehensive guide to Seaborn, a powerful data visualization library built on Matplotlib. You’ll learn how to:

Understand Seaborn’s core purpose: It’s a data visualization library built on Matplotlib that simplifies plotting, often creating complex plots with just one line of code.
Install and set up Seaborn: Learn how to install it using pip or conda, and how to import necessary libraries like NumPy, Pandas, and Matplotlib for seamless integration.
Load and manage data: Primarily use Seaborn’s built-in datasets (e.g., car_crashes, tips, flights, iris, attention) for practice, and understand how to load other file types via Pandas.
Create various distribution plots:
- Distribution plots (displot): Visualize univariate distributions (distributions of a single variable), including histograms and Kernel Density Estimation (KDE) plots, and learn to define bins.
- Joint plots (jointplot): Compare two distributions by default as a scatter plot, and generate regression lines or show KDE or hexagon distributions.
- KDE plots (kdeplot): Create standalone plots for kernel density estimations.
- Pair plots (pairplot): Plot relationships across all numerical values in a DataFrame, showing histograms on the diagonal and scatter plots elsewhere.
- Rug plots (rugplot): Visualize single column data points as sticks, indicating where data is denser.
Master various categorical plots:
- Bar plots (barplot): Analyze distributions of categorical data against numerical data, understanding variance bars and how to change the aggregation estimator (e.g., median, std, cov) using NumPy functions.
- Count plots (countplot): Simply count the number of occurrences for categorical variables.
- Box plots (boxplot): Compare variables by showing quartiles, median, standard deviation, whiskers, and outliers.
- Violin plots (violinplot): A combination of box plots and KDE plots, visualizing the density estimation of data points and how to split them for comparison.
- Strip plots (stripplot): Draw scatter plots with one categorical variable, often used with box plots, and learn to use jitter to spread points and dodge to separate categories.
- Swarm plots (swarmplot): Similar to strip plots but adjusts points to prevent overlap, often layered on top of violin plots.
Generate matrix plots for correlation and data patterns:
- Heat maps (heatmap): Visualize data in a matrix format, requiring data to be prepared using correlation matrices (.corr()) or pivot tables (.pivot_table()), and how to add annotations.
- Cluster maps (clustermap): Create hierarchically clustered heat maps that calculate distances and reposition data to find specific patterns and clusters.
Utilize powerful grid systems for complex visualizations:
- Pair grids (PairGrid): Gain specific control over plot placement within a grid, allowing you to map different plot types (e.g., scatter, histogram, KDE) to the upper, lower, or diagonal sections.
- Facet grids (FacetGrid): Print multiple plots in a grid, defining columns and rows based on categorical data, and apply various styling options to individual subplots.
Create and customize regression plots:
- Regression plots (lmplot): Study relationships between numerical variables, customize markers, sizes, and colors, and separate data into columns or rows based on other variables for multi-faceted analysis.
Apply extensive styling and customization options:
- Set overall plot styles (sns.set_style) like white, darkgrid, whitegrid, dark, and ticks.
- Adjust plot size (plt.figure(figsize=...)).
- Change plot context (sns.set_context) for different uses: paper (Jupyter), talk (presentation), poster.
- Control font scale (font_scale).
- Manage axis visibility by turning spines on or off.
- Change plot color schemes using palettes and exploring Matplotlib’s color maps.
- Reposition plot legends for better readability.
- Customize marker symbols, sizes, line widths, and edge colors in various plot types.

Mon, 01 Jan 0001 00:00:00 +0000

Data visualization#

Effective visuals connect analysis to decisions. This module spotlights the tools that help you design narratives, build interactive experiences, and adapt outputs for every kind of audience.

You’ll practice:

Story-first planning with data storytelling frameworks and LLM partners such as ChatGPT to refine messaging.
Portable presentations spanning HTML decks in RevealJS, Markdown slides in Marp, and notebook-native sessions in Marimo.
Specialized builders including RAWgraphs, Seaborn, Excel forecasting charts, Flourish animations, PowerPoint motion, and Kumu network diagrams.
Collaborative perspectives through actor network visualizations and AI-assisted storytelling that keep stakeholders aligned.

Use these resources to turn complex analyses into clear, compelling stories wherever your viewers meet them.

Mon, 01 Jan 0001 00:00:00 +0000

Data Transformation with dbt#

You’ll learn how to transform data using dbt (data build tool), covering:

dbt Fundamentals: Understand what dbt is and how it brings software engineering practices to data transformation
Project Setup: Learn how to initialize a dbt project, configure your warehouse connection, and structure your models
Models and Materialization: Create your first dbt models and understand different materialization strategies (view, table, incremental)
Testing and Documentation: Implement data quality tests and auto-generate documentation for your data models
Jinja Templating: Use Jinja for dynamic SQL generation, making your transformations more maintainable and reusable
References and Dependencies: Learn how to reference other models and manage model dependencies
Sources and Seeds: Configure source data connections and manage static reference data
Macros and Packages: Create reusable macros and leverage community packages to extend functionality
Incremental Models: Optimize performance by only processing new or changed data
Deployment and Orchestration: Set up dbt Cloud or integrate with Airflow for production deployment

Here’s a minimal dbt model example, models/staging/stg_customers.sql:

Mon, 01 Jan 0001 00:00:00 +0000

Deployment Tools#

Publishing a project means moving assets off your laptop and into reliable environments. This module maps the tooling that packages content, automates delivery, and keeps services reachable for collaborators and users.

You’ll cover:

Content preparation by compressing images and authoring Markdown so sites load fast and stay readable.
Hosting options from static GitHub Pages sites and Google Colab notebooks to Vercel serverless functions, Docker and Podman containers, and GitHub Codespaces devcontainers.
Automation and connectivity using GitHub Actions CI/CD, secure ngrok tunnels, and REST APIs with correct CORS handling.
Production fundamentals including FastAPI backends, Google Auth integration, and local LLM endpoints with Ollama.

Together these tools let you launch prototypes quickly, iterate safely, and graduate data products into production-grade deployments.

Mon, 01 Jan 0001 00:00:00 +0000

Development Tools#

NOTE: The tools in this module are PRE-REQUISITES for the course. You would have used most of these before. If most of this is new to you, please take this course later.

Some tools are fundamental to data science because they are industry standards and widely used by data science professionals. Mastering these tools will align you with current best practices and making you more adaptable in a fast-evolving industry.

Mon, 01 Jan 0001 00:00:00 +0000

Browser: DevTools#

Chrome DevTools is the de facto standard for web development and data analysis in the browser. You’ll use this a lot when debugging and inspecting web pages.

Here are the key features you’ll use most:

Elements Panel
- Inspect and modify HTML/CSS in real-time
- Copy CSS selectors for web scraping
- Debug layout issues with the Box Model
```
// Copy selector in Console
copy($0); // Copies selector of selected element
```
Console Panel

Mon, 01 Jan 0001 00:00:00 +0000

Containers: Docker, Podman#

Docker and Podman are containerization tools that package your application and its dependencies into a standardized unit for software development and deployment.

Docker is the industry standard. Podman is compatible with Docker and has better security (and a slightly more open license). In this course, we recommend Podman but Docker works in the same way.

Initialize the container engine:

podman machine init
podman machine start

Common Operations. (You can use docker instead of podman in the same way.)

Mon, 01 Jan 0001 00:00:00 +0000

Embeddings: OpenAI and Local Models#

Embedding models convert text into a list of numbers. These are like a map of text in numerical form. Each number represents a feature, and similar texts will have numbers close to each other. So, if the numbers are similar, the text they represent mean something similar.

This is useful because text similarity is important in many common problems:

Search. Find similar documents to a query.
Classification. Classify text into categories.
Clustering. Group similar items into clusters.
Anomaly Detection. Find an unusual piece of text.

You can run embedding models locally or using an API. Local models are better for privacy and cost. APIs are better for scale and quality.

Mon, 01 Jan 0001 00:00:00 +0000

Extracting Audio and Transcripts#

Media Processing: FFmpeg#

FFmpeg is the standard command-line tool for processing video and audio files. It’s essential for data scientists working with media files for:

Extracting audio/video for machine learning
Converting formats for web deployment
Creating visualizations and presentations
Processing large media datasets

Basic Operations:

# Basic conversion
ffmpeg -i input.mp4 output.avi

# Extract audio
ffmpeg -i input.mp4 -vn output.mp3

# Convert format without re-encoding
ffmpeg -i input.mkv -c copy output.mp4

# High quality encoding (crf: 0-51, lower is better)
ffmpeg -i input.mp4 -preset slower -crf 18 output.mp4

Common Data Science Tasks:

Mon, 01 Jan 0001 00:00:00 +0000

Web Framework: FastAPI#

FastAPI is a modern Python web framework for building APIs with automatic interactive documentation. It’s fast, easy to use, and designed for building production-ready REST APIs.

Here’s a minimal FastAPI app, app.py:

# /// script
# requires-python = ">=3.11"
# dependencies = [
# "fastapi",
# "uvicorn",
# ]
# ///

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
async def root():
 return {"message": "Hello!"}


if __name__ == "__main__":
 import uvicorn

 uvicorn.run(app, host="0.0.0.0", port=8000)

Run this with uv run app.py.

Mon, 01 Jan 0001 00:00:00 +0000

May 2025 Feedback#

Based on feedback from 33 students, via ChatGPT.

Insights#

⭐ “Time-pressure” complainers actually rated ROE slightly higher. Students who mentioned ROE time limits gave ROE 2.61 vs 2.33 for others (+0.28). This suggests the most engaged cohort both feels the time squeeze and still perceives value—hinting the exam is closer to a “desirable difficulty” than a pure frustration. (Concept supported by learning-science literature on “desirable difficulties”.) (bjorklab.psych.ucla.edu)
Project-2 is love-it or hate-it. Median is 5/5 with many top scores, but variance is highest (σ≈1.54). Those flagging “difficulty/prereqs” rated P2 1.32 points lower than others—so P2 is excellent for prepared learners and punishing for under-prepared ones (bimodal fit).
Calling out TA/Discourse correlates with higher satisfaction. Mentions of TA/Discourse associate with slightly higher ratings across all four metrics (Δ overall +0.08, P1 +0.15, P2 +0.32, ROE +0.05). That’s counter to the intuition that “more TA talk = more complaints”; it reads as “the engaged students leaned on Discourse and benefited.”
Students know it’s hard—and still rate learning high. Despite many “hard/fast” comments, overall learning is strong (~4.0/5). That aligns with the course’s stated design (“ROE is hard”; “programming skills are a pre-requisite”), and suggests difficulty didn’t suppress perceived learning.
“Alignment” is a minority complaint—but a leverage point. Only a few explicitly said “align projects/ROE with taught content,” yet those who did had lower ROE ratings (−0.57) and overall learning (−0.47). A small group may be disproportionately dissatisfied due to scope drift.

Interesting observations#

ROE dominates the pain map. Time limit (45 min per the schedule), ambiguity, and “what exactly are we proving?” recur. A few ask for mock portals and clearer rubric.
P2 teaches beyond comfort. Many “pushes us beyond” / “real-world exposure” notes in “Best things,” while the “Worst” ask for more scaffolding. Classic split between stretch vs support.
Ops gaps show up: evaluation delays; release/schedule slips; a few network/portal reliability mentions; form UX complaints (Google Form structure).
Tooling friction: requests for dependable GPT/AI-Pipe access and deployment help; some struggled with “free versions” / rate limits. (AI-Pipe is in your ecosystem.)
Students explicitly ask for “instant feedback” on submissions (linting/auto-checks), not just grades—hinting appetite for continuous formative signals.
A minority asks for tougher projects (and longer ROE). The tails want even more challenge/time—a useful signal for differentiated tracks.
Desire for predictable cadence (earlier release, stick to schedule) appears in multiple corners even if counts are lower; these often co-occur with lower satisfaction.

Actions#

Gate early with a hard readiness check + branching track. Make GA-1 (already linked) the gate:
- If GA-1 < threshold: auto-route to a “Foundation track” of P2 (same outcomes, more scaffolds, smaller surface area).
- If GA-1 ≥ threshold: “Pro track” (current P2). Rationale: removes P2 bimodality; keeps challenge for prepared students; respects “desirable difficulty” while preventing wipeouts. Implementation is trivial (routing by score in your existing exam infra).
ROE redesign: keep pressure, fix friction.
- Keep 45-min “pressure-cooker” (it likely drives durable learning), but reduce non-constructive friction:
  - Two 10-min micro-mocks in the exact ROE environment the week prior (auto-graded, unlimited retries).
  - Stable prompt rubric with 2–3 canonical “what good looks like” exemplars.
  - Network-independent tasks (bundle data; pre-cache assets) and a simple “connectivity hiccup” auto-extend of 3–5 min if the portal detects time-outs. (bjorklab.psych.ucla.edu)
TA/Discourse “SLA + Triage Bot.”
- Publish an SLA (e.g., TA initial touch <6 h, resolution <24 h).
- Add a Discourse clarifier bot that: (i) de-dupes to canonical answers; (ii) proposes clearer wording for confusing prompts; (iii) tags to the right TA. (Start with a rules-only bot; graduate to LLM summaries later.) This leans into the positive correlation you’re already seeing.
Instant feedback everywhere.
- Add pre-submission checks: schema validators, smoke tests, minimal unit tests for projects; “red/green” badges on submit.
- For P2, expose a thin public checker (5–10 hidden tests retained for grading).
Cadence discipline via CI. Turn “release on time” from a promise into automation: a release calendar in the repo, GitHub Actions that publish content at fixed timestamps, and an automated grading queue dashboard (counts, average turnaround, SLA breaches).
Tooling reliability: AI-Pipe + deployment helpers.
- Provide a one-click “fallback” API key via AI-Pipe with rate-limit hints and a local stub; publish two minimal deployment recipes (Vercel/Cloudflare) with a pre-flight checklist. (Your repo already points to AI-Pipe.)
Micro-scaffolds for P2 only (don’t blunt the challenge). Ship optional scaffolds: sample input/output pairs, data dictionaries, 2 starter snippets. These trim unproductive thrash while preserving the tough core.
Rubric transparency + sample graded artifacts. Post 2 anonymized “A/B/C” exemplars with rubric mapping; students asked for “what exactly are we proving?” — this answers that.

Contrarian take:

Mon, 01 Jan 0001 00:00:00 +0000

Fine tuning LLMs: Gemini#

This material will likely be added in the May 2025 term. It is not part of the Jan 2025 term.

Mon, 01 Jan 0001 00:00:00 +0000

Forecasting with Excel#

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Function Calling with OpenAI#

Function Calling allows Large Language Models to convert natural language into structured function calls. This is perfect for building chatbots and AI assistants that need to interact with your backend systems.

OpenAI supports Function Calling – a way for LLMs to suggest what functions to call and how.

Here’s a minimal example using Python and OpenAI’s function calling that identifies the weather in a given location.

Mon, 01 Jan 0001 00:00:00 +0000

Geospatial Analysis with Excel#

You’ll learn how to create a data-driven story about coffee shop coverage in Manhattan, covering:

Data Collection: Collect and scrape data for coffee shop locations and census population from various sources.
Data Processing: Use Python libraries like geopandas for merging population data with geographic maps.
Map Creation: Generate coverage maps using tools like QGIS and Excel to visualize coffee shop distribution and population impact.
Visualization: Create physical, Power BI, and video visualizations to present the data effectively.
Storytelling: Craft a narrative around coffee shop competition, including strategic insights and potential market changes.

Here are links that explain how the video was made:

Mon, 01 Jan 0001 00:00:00 +0000

Geospatial Analysis with Python#

You’ll learn how to perform geospatial analysis for location-based decision making, covering:

Distance Calculation: Compute distances between various store locations and a reference point, such as the Empire State Building.
Data Visualization: Visualize store locations on a map using Python libraries like Folium.
Store Density Analysis: Determine the number of stores within a specified radius.
Proximity Analysis: Identify the closest and farthest stores from a specific location.
Decision Making: Use geospatial data to assess whether opening a new store is feasible based on existing store distribution.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Geospatial Analysis with QGIS#

You’ll learn how to use QGIS for geographic data processing, covering:

Shapefiles and KML Files: Create and manage shapefiles and KML files for storing and analyzing geographic information.
Downloading QGIS: Install QGIS on different operating systems and familiarize yourself with its interface.
Geospatial Data: Access and utilize shapefiles from sources like Diva-GIS and integrate them into QGIS projects.
Creating Custom Shapefiles: Learn how to create custom shapefiles when existing ones are unavailable, including creating a shapefile for South Sudan.
Editing and Visualization: Use QGIS tools to edit shapefiles, add attributes, and visualize geographic data with various styling and labeling options.
Exporting Data: Export shapefiles or KML files for use in other applications, such as Google Earth.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Version Control: Git, GitHub#

Git is the de facto standard for version control of software (and sometimes, data as well). It’s a system that keeps track of changes you make to files and folders. It allows you to revert to a previous state, compare changes, etc. It’s a central tool in any developer’s workflow.

GitHub is the most popular hosting service for Git repositories. It’s a website that shows your code, allows you to collaborate with others, and provides many useful tools for developers.

Mon, 01 Jan 0001 00:00:00 +0000

CI/CD: GitHub Actions#

GitHub Actions is a powerful automation platform built into GitHub. It helps automate your development workflow - running tests, deploying applications, updating datasets, retraining models, etc.

Understand the basics of YAML configuration files
Explore the pre-built actions from the marketplace
How to handle secrets securely
Triggering a workflow
Staying within the free tier limits
Caching dependencies to speed up workflows

Here is a sample .github/workflows/iss-location.yml that runs daily, appends the International Space Station location data into iss-location.jsonl, and commits it to the repository.

Mon, 01 Jan 0001 00:00:00 +0000

IDE: GitHub Codespaces#

GitHub Codespaces is a cloud-hosted development environment built right into GitHub that gets you coding faster with pre-configured containers, adjustable compute power, and seamless integration with workflows like Actions and Copilot.

Why Codespaces helps

Reproducible onboarding: Say goodbye to “works on my machine” woes—everyone uses the same setup for assignments or demos.
Anywhere access: Jump back into your project from a laptop, tablet, or phone without having to reinstall anything.
Rapid experimentation & debugging: Spin up short-lived environments on any branch, commit, or PR to isolate bugs or test features, or keep longer-lived codespaces for big projects.

Mon, 01 Jan 0001 00:00:00 +0000

AI Editor: GitHub Copilot#

AI Code Editors like GitHub Copilot, Cursor, Windsurf, Roo Code, Cline, Continue.dev, etc. use LLMs to help you write code faster.

Most are built on top of VS Code. These are now a standard tool in every developer’s toolkit.

GitHub Copilot is free (as of May 2025) for 2,000 completions and 50 chats.

You should learn about:

Code Suggestions, which is a basic feature.
Using Chat, which lets you code in natural language.
Changing the chat model. The free version includes Claude 3.5 Sonnet, a good coding model.
Prompts to understand how people use AI code editors.

Mon, 01 Jan 0001 00:00:00 +0000

Static hosting: GitHub Pages#

GitHub Pages is a free hosting service that turns your GitHub repository directly into a static website whenever you push it. This is useful for sharing analysis results, data science portfolios, project documentation, and more.

Common Operations:

# Create a new GitHub repo
mkdir my-site
cd my-site
git init

# Add your static content
echo "<h1>My Site</h1>" > index.html

# Push to GitHub
git add .
git commit -m "feat(pages): initial commit"
git push origin main

# Enable GitHub Pages from the main branch on the repo settings page

Best Practices:

Mon, 01 Jan 0001 00:00:00 +0000

Google Authentication with FastAPI#

Secure your API endpoints using Google ID tokens to restrict access to specific email addresses.

Google Auth is the most commonly implemented single sign-on mechanism because:

It’s popular and user-friendly. Users can log in with their existing Google accounts.
It’s secure: Google supports OAuth2 and OpenID Connect to handle authentication.

Here’s how you build a FastAPI app that identifies the user.

Mon, 01 Jan 0001 00:00:00 +0000

Google Charts#

Google Chart 1: How To Create Chart Or Graph On HTML CSS Website | Google Charts Tutorial
Google Chart 2: Using Google Charts with googleVis

Mon, 01 Jan 0001 00:00:00 +0000

Google Data Studio#

Google Data Studio: Google Data Studio Tutorial for Beginners🔥

Mon, 01 Jan 0001 00:00:00 +0000

Hosting LLMs on Runpod#

This material will likely be added in the May 2025 term. It is not part of the Jan 2025 term.

Mon, 01 Jan 0001 00:00:00 +0000

HTTP Requests: curl, wget, HTTPie, Postman#

Making HTTP requests is essential for interacting with web APIs, downloading data, and testing web services. Whether you’re fetching data from a public API, testing your own REST endpoints, or automating data collection, these tools make HTTP interactions simple and efficient.

This guide covers four popular approaches: curl (the universal standard), wget (for downloads), HTTPie (user-friendly alternative), and Postman (GUI-based testing).

Watch these tutorials to understand HTTP requests and API testing (60 min):

Mon, 01 Jan 0001 00:00:00 +0000

Container hosting: Hugging Face Spaces with Docker#

Container platforms let you deploy applications in isolated, portable environments that include all dependencies. They’re perfect for ML applications that need custom environments, specific packages, or persistent state. Here are some common real-life uses:

A text summarization API that processes documents using custom NLP models (runs for 10-30 seconds per request)
An image classification tool that requires specific computer vision libraries (runs for 5-15 seconds per upload)
A chatbot with custom fine-tuned models that need GPU acceleration (runs for 2-5 seconds per message)

Unlike serverless functions, containers can maintain state, install any packages, write to the filesystem, and run background processes. They provide more control over the environment but require more configuration.

Mon, 01 Jan 0001 00:00:00 +0000

Hybrid Retrieval Augmented Generation (Hybrid RAG) with TypeSense#

Hybrid RAG combines semantic (vector) search with traditional keyword search to improve retrieval accuracy and relevance. By mixing exact text matches with embedding-based similarity, you get the best of both worlds: precision when keywords are present, and semantic recall when phrasing varies. TypeSense makes this easy with built-in hybrid search and automatic embedding generation.

Below is a fully self-contained Hybrid RAG tutorial using TypeSense, Python, and the command line.

Mon, 01 Jan 0001 00:00:00 +0000

Images: Compression#

Image compression is essential when deploying apps. Often, pages have dozens of images. Image analysis runs over thousands of images. The cost of storage and bandwidth can grow over time.

Here are things you should know when you’re compressing images:

Image dimensions are the width and height of the image in pixels. This impacts image size a lot
Lossless compression (PNG, WebP) preserves exact data
Lossy compression (JPEG, WebP) removes some data for smaller files
Vector formats (SVG) scale without quality loss
WebP is the modern standard, supporting both lossy and lossless

Here’s a rule of thumb you can use as of 2025.

Mon, 01 Jan 0001 00:00:00 +0000

JSON#

JSON (JavaScript Object Notation) is the de facto standard format for data exchange on the web and APIs. Its human-readable format and widespread support make it essential for data scientists working with web services, APIs, and configuration files.

For data scientists, JSON is essential when:

Working with REST APIs and web services
Storing configuration files and metadata
Parsing semi-structured data from databases like MongoDB
Creating data visualization specifications (e.g., Vega-Lite)

Watch this comprehensive introduction to JSON (15 min):

Mon, 01 Jan 0001 00:00:00 +0000

All Lab Assignments#

Week 01 — Development Environment & Tooling#

Lab: Publish a Python library to PyPI using UV
Lab: Web Traffic Debugging with Burp Suite

Week 02 — Deployment & API Engineering#

Lab: Deploying Private LLMs with vLLM & API Gateway
Lab: WebSocket Chat with Redis & PostgreSQL

Week 03 — LLM Engineering#

Week 04 — RAG & Hybrid RAG#

CAPSTONE: BS Degree Chatbot
Lab: RAGAS evaluation dashboard (Naive vs Hybrid vs Contextual)

Week 05 — Agentic AI#

CAPSTONE: Autonomous Research Agent

Week 06 — Web Data Acquisition & OSINT#

CAPSTONE: Job Posting Scraper & Tracker
CAPSTONE: AI Signature Detection & Cropper
CAPSTONE: Live Multilingual Travel Translator
Lab: Scheduled scraper with GitHub Actions cron (DuckDB + Parquet)
Lab: Open-Source Organisation Dossier (cited OSINT)

Week 07 — CI/CD, Security & Cloud Infrastructure#

Mon, 01 Jan 0001 00:00:00 +0000

Capstone Projects#

#	Project	Week	Description
C1	BS Degree Chatbot	Week 04	Hybrid RAG chatbot + RAGAS evaluation
C2	Autonomous Research Agent	Week 05	LangChain daily research blog with memory, validation, and GitHub Actions
C3	Job Posting Scraper & Tracker	Week 06	Hidden API → validated schema → SQLite state → Parquet → DuckDB dashboard
C4	AI Signature Detection & Cropper	Week 06	Detect → crop → independent verifier, scored on precision/recall
C5	Live Multilingual Travel Translator	Week 06	STT → translate → TTS pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Lab 1.1 — Publish a Python Library to PyPI using UV#

What you’ll build

A real Python package, installable by the world via pip install your-package-name, published to PyPI from GitHub Actions using Trusted Publishing (no API tokens, no secrets in your repo). We’ll use UV for every step.

Time: 60–90 minutes. Difficulty: ⭐⭐☆☆☆. Ship: your name on pypi.org.

What the Finished Thing Looks Like#

By the end:

pip install tds-hello-<yourname>
python -c "from tds_hello import greet; print(greet('World'))"
# Hello, World! — from tds-hello v0.1.0

And every git tag v* push auto-publishes a new version.

Mon, 01 Jan 0001 00:00:00 +0000

Lab 1.2 — Web Traffic Debugging with Burp Suite#

What you’ll build A local HTTP proxy laboratory. You will write a single-file Python script to host an interactive security & API sandbox, and then use Burp Suite’s Proxy and Repeater tools to intercept, analyze, modify, and replay HTTP traffic to solve three hands-on debugging challenges.

Time: 45–60 minutes. Difficulty: ⭐⭐☆☆☆. Learning Outcomes: Intercept and modify outgoing client requests, bypass client-side parameters, perform ID enumeration in the Repeater, and inject custom headers.

Mon, 01 Jan 0001 00:00:00 +0000

Lab: Deploying Private LLMs with vLLM & API Gateway#

This practical lab guides you through serving open-source Large Language Models (LLMs) like Gemma 2B or Qwen 2.5 locally and in production using vLLM (or Ollama/llama.cpp as a CPU fallback). To prevent data leakage and secure your infrastructure, you will build a FastAPI API Gateway that wraps your model server behind secure API keys and supports real-time OpenAI-compatible token streaming.

flowchart LR
 Client[Web Browser / SDK] -->|1. Request with API Key| Gateway[FastAPI Gateway]
 Gateway -->|2. Validate Key| Vault[(Authorized Keys)]
 Gateway -->|3. Forward Request| LLMServer[vLLM Inference Engine]
 LLMServer -->|4. Generate Tokens| GPU[NVIDIA / AMD GPU]
 GPU -->|5. Compute| LLMServer
 LLMServer -->|6. Stream Response| Gateway
 Gateway -->|7. SSE Stream| Client

Theoretical Context: vLLM & Quantization#

Why vLLM?#

Standard PyTorch or Hugging Face serving frameworks handle requests sequentially or in static batch queues. This causes massive latency and wastes GPU capacity. vLLM solves this with:

Mon, 01 Jan 0001 00:00:00 +0000

Lab: WebSocket Chat with Redis & PostgreSQL#

This practical lab guides you through building a real-time, authenticated chat application. You will implement Google OAuth for authentication, use a PostgreSQL database to persist messages, integrate Redis to cache sessions and chat histories, and establish WebSockets for instantaneous messaging.

flowchart TD
 Browser[Web Browser]
 FastAPI[FastAPI Backend]
 Google[Google OAuth Server]
 Postgres[(PostgreSQL Database)]
 Redis[(Redis Cache)]

 Browser -->|1. OAuth Redirect & Callback| Google
 Browser -->|2. HTTP Requests with Session Cookie| FastAPI
 Browser -->|3. Establish WebSocket Connection| FastAPI
 FastAPI -->|Check/Set Session & Cache History| Redis
 FastAPI -->|Query/Persist Messages & Users| Postgres

Real Developer Use Case#

In production systems, raw WebSockets are stateful and memory-intensive. Keeping chat histories or sessions in-memory limits scalability. By storing persistent records in PostgreSQL and using Redis for high-speed session checks and chat history caches, you create a stateless, horizontally-scalable backend capable of handling thousands of concurrent users.

Mon, 01 Jan 0001 00:00:00 +0000

Lab 3.2 — Cost-Tracking Dashboard via LangSmith#

What you’ll build: A benchmark system that runs the same task using four different prompt strategies — zero-shot, few-shot, Chain-of-Thought, and Self-Consistency — traces every call to LangSmith, then exposes a FastAPI endpoint that returns a live cost comparison dashboard.

┌────────────────────────────────────────────────────────────┐
│ Input: "What is 15% of 840?" │
│ │
│ Strategy 1: Zero-shot → "126" $0.000012 │
│ Strategy 2: Few-shot → "126" $0.000018 │
│ Strategy 3: CoT → "...= 126" $0.000042 │
│ Strategy 4: Self-Consist. → "126 (7/7)" $0.000294 │
│ │
│ → LangSmith shows every call with token counts & cost │
│ → /dashboard endpoint returns live comparison JSON │
└────────────────────────────────────────────────────────────┘

Architecture#

graph TD
 A[benchmark.py] -->|traceable| B(run_zero_shot)
 A -->|traceable| C(run_few_shot)
 A -->|traceable| D(run_cot)
 A -->|traceable| E(run_self_consistency)
 B -->|LangSmith API| F[LangSmith Cloud Dashboard]
 C -->|LangSmith API| F
 D -->|LangSmith API| F
 E -->|LangSmith API| F
 G[FastAPI Server] -->|Query API| F
 G -->|Endpoint: /dashboard| H[Live Cost JSON Dashboard]

**Step 1 — Project Setup**

mkdir tds-lab-3-2
cd tds-lab-3-2
uv init --no-workspace

Install dependencies:

Mon, 01 Jan 0001 00:00:00 +0000

Lab 3.1 — YouTube → Subtitles → Topics → JSON Pipeline#

What you’ll build: A CLI tool that takes any YouTube URL and produces a structured JSON summary containing topics, key Q&A pairs, chapter breakdowns, and answer timestamps — all extracted by Claude from the auto-generated subtitles.

# Input
python pipeline.py "https://www.youtube.com/watch?v=VIDEO_ID"

# Output: output/VIDEO_ID_summary.json
{
 "title": "Introduction to Docker",
 "duration_minutes": 45,
 "topics": ["containerization", "Dockerfile", "docker-compose", ...],
 "chapters": [
 {
 "title": "What is Docker?",
 "start_time": "00:00",
 "end_time": "05:30",
 "summary": "Docker packages apps into containers..."
 }
 ],
 "qa_pairs": [
 {
 "question": "What is the difference between image and container?",
 "answer": "An image is a blueprint; a container is a running instance.",
 "timestamp": "03:42"
 }
 ]
}

Architecture#

graph TD
 A[YouTube URL] -->|yt-dlp| B(VTT Subtitle File)
 B -->|vtt_parser.py| C(Timestamped Segments)
 C -->|Claude Haiku| D[Topic Extraction]
 C -->|Claude Haiku| E[Chapter Segmentation]
 C -->|Claude Haiku| F[Q&A Extraction]
 D --> G[Assemble VideoSummary JSON]
 E --> G
 F --> G

**Step 1 — Project Setup**

mkdir tds-lab-3-1
cd tds-lab-3-1
uv init --no-workspace

Install dependencies:

Mon, 01 Jan 0001 00:00:00 +0000

Capstone — BS Degree Chatbot#

Objective#

Build a production-grade RAG chatbot that answers user questions about the IIT Madras BS Data Science programme using official documentation.

Requirements#

Scrape academics, courses, and qualifier pages from the official IIT Madras BS program site.
Build an ingestion pipeline using a vector database (e.g., Qdrant) with contextual chunking.
Implement a hybrid retrieval strategy combining dense vector embeddings and BM25 sparse index, merged via RRF and an optional reranker.
Build a FastAPI backend with a /chat endpoint and a Streamlit-based UI.
Setup a RAGAS evaluation suite to measure retrieval and generation accuracy across different retrieval strategies.

Deliverables#

Source code.
Link to the live Streamlit chatbot application.
RAGAS evaluation report comparing Naive, Hybrid, and Contextual RAG.

Mon, 01 Jan 0001 00:00:00 +0000

Lab — RAGAS Evaluation Dashboard#

Objective#

Build an interactive Streamlit dashboard that runs RAGAS evaluation across multiple RAG retrieval strategies side-by-side.

Requirements#

Implement three RAG strategies: Naive RAG (dense only), Hybrid RAG (dense + BM25 + RRF), and Contextual RAG (using Anthropic’s chunk context injection).
Generate a set of test questions and ground-truth answers using an LLM.
Evaluate the retrieval strategies using RAGAS metrics (Faithfulness, Context Recall, Factual Correctness, Response Relevancy).
Build a Streamlit dashboard showing metrics comparison tables, strategy charts, and per-question breakdowns.

Deliverables#

Source code.
Screenshot of the Streamlit dashboard comparing all three strategies.
Downloadable evaluation results JSON file.

Mon, 01 Jan 0001 00:00:00 +0000

Capstone — Autonomous Daily AI Research Blog#

Build a static blog where a LangChain agent researches recent AI news every morning, writes a short cited digest, remembers previous coverage, commits the new Markdown post, and deploys the updated site to GitHub Pages.

This is a single-agent loop, not a multi-agent demo. The agent can only search; normal Python code validates and writes the post. This keeps the project small, understandable, and safer.

Mon, 01 Jan 0001 00:00:00 +0000

Capstone — AI Signature Detection & Cropper#

Find handwritten signatures on scanned documents, crop them out, verify them with a second model, and be honest about your error rate.

⏱ ~6–8 hours 🔗 needs: Vision Models for Scraping · Image Processing Pipeline · Document Parsing

A realistic document-AI pipeline: detect a region, crop it, and use a second, independent model to check the first one. The interesting engineering is in the verification and the measurement, not the detection call.

Mon, 01 Jan 0001 00:00:00 +0000

Capstone — Job Posting Scraper & Tracker#

Build a scraper that runs itself, notices what changed, and answers questions about the market.

⏱ ~6–8 hours 🔗 needs: Hidden JSON APIs · Change Detection & Dedup · DuckDB + Parquet · Scheduled Scraping

Anyone can scrape a page once. This capstone is about the harder, more valuable thing: a pipeline that runs unattended for weeks, doesn’t duplicate data, doesn’t get banned, and produces a dataset worth querying.

Mon, 01 Jan 0001 00:00:00 +0000

Capstone — Live Multilingual Travel Translator#

Speech in, speech out, in another language — a three-model pipeline where every stage can fail, and your job is to make the whole thing not fall over.

⏱ ~6–8 hours 🔗 needs: Speech AI · FastAPI · Structured Output

Chaining STT → translation → TTS is easy to demo and hard to make reliable: errors compound, latency stacks, and each stage has its own failure mode. This capstone is graded on the engineering around the models.

Mon, 01 Jan 0001 00:00:00 +0000

Lab — Open-Source Organisation Dossier#

Assemble a fully-cited profile of an organisation’s public footprint, where every single claim carries its source, timestamp, and confidence level.

⏱ ~3–4 hours 🔗 needs: OSINT — Infrastructure & Records · Google Dorking · Legal & Ethical Scraping

The skill being graded is not collection — anyone can paste tool output. It’s verification: knowing what your evidence actually supports, and saying so honestly.

⚖️ Scope rules — a submission that breaks any of these scores zero.

Mon, 01 Jan 0001 00:00:00 +0000

Lab — Scheduled Scraper with GitHub Actions#

Get a scraper running on a free daily cron that commits its own data — the foundation every other Week 6 lab builds on.

⏱ ~2 hours 🔗 needs: Scheduled Scraping · DuckDB + Parquet

A small, complete pipeline: fetch → store → schedule → query. Deliberately uses a public, documented API so nothing here is ethically ambiguous.

Objective#

Collect Hacker News top stories daily, accumulate them as Parquet, and query the result with DuckDB.

Mon, 01 Jan 0001 00:00:00 +0000

Lab — Red-Team Your Own LLM API#

Attack a system you built, prove which defence stopped each attack, and leave behind a regression suite that keeps it fixed.

⏱ ~4–5 hours 🔗 needs: LLM Security — Offensive · LLM Safety — Defensive · OWASP LLM Top 10

You’ll build a deliberately weak LLM service, break it, harden it, and then prove the hardening works — with tests that run in CI.

⚖️ Target only your own deployment. Every attack in this lab runs against the service you build and host. Probing a third party’s LLM product is unauthorised testing. If you want a harder target, make your own service harder.

Mon, 01 Jan 0001 00:00:00 +0000

Lab — Full CI/CD Pipeline to Cloud Run#

One git push → tests, image build, deploy, health check. Plus the parts people skip: a rollback, a cost cap, and an approval gate.

⏱ ~4–5 hours 🔗 needs: GitHub Actions Advanced · Advanced Docker · Serverless Functions

Everything in Week 7 assembled into one working pipeline.

What you’re building#

flowchart LR
 P["git push"] --> T["Test matrix + lint"]
 T --> B["Build image<br/>(cached, multi-stage)"]
 B --> SC["Scan for CVEs + secrets"]
 SC --> ST["Deploy to STAGING"]
 ST --> H["Smoke test /health"]
 H -->|"main only"| A{"Manual approval"}
 A --> PR["Deploy to PRODUCTION"]
 PR --> V["Verify + auto-rollback on failure"]

Requirements#

1. The app. Any small FastAPI service with /health and one real endpoint. Reuse your Week 6 scraper API or the Week 7 hardened LLM service.

Mon, 01 Jan 0001 00:00:00 +0000

Milestone: Full GCP Walkthrough#

GCP is not one big computer in the cloud. It is a set of managed services, identities, regions, bills, and logs. Your job is to connect only the few pieces your product needs—and be able to explain every connection.

⏱ ~18 min read · ~45 min guided walkthrough 🔗 needs: Cloud Storage for ML · BigQuery ML · Cost Alerting & Budgets

Mon, 01 Jan 0001 00:00:00 +0000

Large Language Models#

This module covers the practical usage of large language models (LLMs).

LLMs incur a cost. For the May 2025 batch, use aipipe.org as a proxy. Emails with @ds.study.iitm.ac.in get a $1 per calendar month allowance. (Don’t exceed that.)

Read the AI Pipe documentation to learn how to use it. But in short:

Replace OPENAI_BASE_URL, i.e. https://api.openai.com/v1 with https://aipipe.org/openrouter/v1... or https://aipipe.org/openai/v1...
Replace OPENAI_API_KEY with the AIPIPE_TOKEN
Replace model names, e.g. gpt-4.1-nano, with openai/gpt-4.1-nano

For example, let’s use Gemini 2.0 Flash Lite via OpenRouter for chat completions and Text Embedding 3 Small via OpenAI for embeddings:

Mon, 01 Jan 0001 00:00:00 +0000

Live Sessions#

Live sessions by the instructors and TAs are recorded and uploaded to the YouTube Tools in Data Science channel.

These are transcribed by faq.sh into FAQ-style markdown files, which are stored here.

Sessions#

Video | 2025 10 04 Project 1 - Session 1 TDS Sep 2025 (2h 24m)
Video | FAQ | 2025 10 03 week 2 - Session 4 TDS Sep 2025 (2h 16m)
Video | FAQ | 2025 09 30 week 2 - Session 3 TDS Sep 2025 (1h 45m)
Video | FAQ | 2025 09 28 week 2 - Session 2 TDS Sep 2025 (2h 36m)
Video | FAQ | 2025 09 27 week 2 - Session 1 TDS Sep 2025 (2h 53m)
Video | FAQ | 2025 09 24 week 1 - Session 5 TDS Sep 2025 (1h 41m)
Video | FAQ | 2025 09 29 Week 1 - Session 4 TDS Sep 2025 (1h 38m)
Video | 2025 09 21 Week 1 - Session 2 TDS Sep 2025 (2h 37m)
Video | FAQ | 2025 09 22 Week 1 - Session 3 TDS Sep 2025 (1h 58m)
Video | FAQ | 2025 09 20 Week 1 - Session 1 TDS Sep 2025 (2h 52m)
Video | FAQ | 2025 08 04 project 2 - Q&A 2 TDS May 2025 (1h 59m)
Video | FAQ | 2025 07 26 project 2 - Session 2 TDS May 2025 (1h 42m)
Video | FAQ | 2025 07 24 project 2 - Session 1 TDS May 2025 (1h 50m)
Video | FAQ | 2025 07 21 project 2 - Q&A 1 TDS May 2025 (1h 32m)
Video | FAQ | 2025 07 18 MOCK ROE session 4 - TDS May 2025 (2h 7m)
Video | FAQ | 2025 07 17 MOCK ROE session 3 - TDS May 2025 (1h 33m)
Video | FAQ | 2025 07 16 MOCK ROE session 2 - TDS May 2025 (0h 53m)
Video | FAQ | 2025 07 10 MOCK ROE session1 - TDS May 2025 (1h 56m)
Video | FAQ | 2025 07 06 Week 6 Session 2 - TDS May 2025 (1h 49m)
Video | FAQ | 2025 07 05 Week 6 Session 1 - TDS May 2025 (1h 17m)
Video | FAQ | 2025 06 28 Week 5 Session 1 - TDS May 2025 (2h 8m)
Video | FAQ | 2025 06 15 Week 4 1 Session 1 - TDS May 2025 (1h 26m)
Video | FAQ | 2025 06 12 Project 1 - DCS TDS May 2025 (1h 2m)
Video | FAQ | 2025 06 12 Project 1 Session 3 - TDS May 2025 (2h 26m)
Video | FAQ | 2025 06 11 Project 1 Session 2 - TDS May 2025 (2h 59m)
Video | FAQ | 2025 06 10 Project 1 Session 1 - TDS May 2025 (2h 28m)
Video | FAQ | 2025 05 29 Week 3 Session 4 - TDS May 2025 (2h 44m)
Video | FAQ | 2025 05 28 Week 3 Session 3 - TDS May 2025 (1h 30m)
Video | FAQ | 2025 05 25 Week 3 Session 2 - TDS May 2025 (1h 38m)
Video | FAQ | 2025 05 24 Week 3 Session 1 - TDS May 2025 (1h 47m)
Video | FAQ | 2025 05 23 Week 2 Session 2 - Github Actions, Docker - TDS May 2025 (3h 17m)
Video | FAQ | 2025 05 22 Week 2 Session 2 - Local Lamma, Ngrok, Google Authentication TDS May 2025 (2h 40m)
Video | FAQ | 2025 05 21 Week 2 Session 1 TDS May 2025 (2h 31m)
Video | FAQ | 2025 05 17 Week 1 Session 5 TDS May 2025 (2h 37m)
Video | FAQ | 2025 05 16 Week 1 Session 4 Env Settings, Env Variables, git and github TDS May 2025 (2h 12m)
Video | FAQ | 2025 05 15 Week 1 Session 3 Linux, BASH, git & github TDS May 2025 (3h 3m)
Video | FAQ | 2025 05 11 Week 1 Session 2 Setup & Debugging TDS May 2025 (3h 3m)
Video | FAQ | 2025-05-10 Week 1 - Session 1: Introduction to TDS - TDS May 25 (2h 29m)
Video | FAQ | 2025-04-09 Live Session - Qs & As - TDS Jan 25 (1h 19m)
Video | FAQ | 2025-04-08 End Term Session 1 - TDS Jan 25 (1h 59m)
Video | FAQ | 2025-03-27 Week 11 - Session 3 - TDS Jan 25 (2h 7m)
Video | FAQ | 2025-03-26 Week 11 - Session 2 - TDS Jan 25 (1h 27m)
Video | FAQ | 2025-03-25 Week 11 - Session 1 - TDS Jan 25 (1h 17m)
Video | FAQ | 2025-03-20 Week 10 - Session 3 - TDS Jan 25 (1h 44m)
Video | FAQ | 2025-03-18 Week 10 - Session 1 - TDS Jan 25 (2h 24m)
Video | FAQ | 2025-03-13 Week 9 - Session 3 - TDS Jan 25 (1h 58m)
Video | FAQ | 2025-03-11 Week 9 - Session 1 - TDS Jan 25 (2h 11m)
Video | FAQ | 2025-03-12 Week 9 - Session 2 - TDS Jan 25 (1h 14m)
Video | FAQ | 2025-03-04 Week 8 - Session 1 - TDS Jan 25 (1h 52m)
Video | FAQ | 2025-03-04 Week 8 - Session 3 - TDS Jan 25 (1h 52m)
Video | FAQ | 2025-03-04 Week 8 - Session 2 - TDS Jan 25 (0h 47m)
Video | FAQ | 2025-02-27 Week 7 - Session 3 - TDS Jan 25 (2h 24m)
Video | FAQ | 2025-02-26 Week 7 - Session 2 - TDS Jan 25 (5h 8m)
Video | FAQ | TA’s Session TDS – 2024 11 10 19 49 (1h 39m)
Video | FAQ | 2025-02-18 Week 6 - Session 3 - TDS Jan 25 (1h 34m)
Video | FAQ | 2025-02-18 Week 6 - Session 1 - TDS Jan 25 (2h 8m)
Video | FAQ | 2025-02-13 Week 5 - Session 3 - TDS Jan 25 (3h 12m)
Video | FAQ | 2025-02-11 Week 5 - Session 1 - TDS Jan 25 (3h 57m)
Video | FAQ | 2025-02-06 Week 4 - Session 4 - TDS Jan 25 (2h 16m)
Video | FAQ | 2025-02-06 Week 4 - Session 3 - TDS Jan 25 (2h 24m)
Video | FAQ | 2025-02-04 Week 4 - Session 1 - TDS Jan 25 (1h 27m)
Video | FAQ | 2025-02-01 Week 3 - Session 5 - TDS Jan 25 (2h 34m)
Video | FAQ | 2025-01-31 Week 3 - Session 4 - TDS Jan 25 (2h 45m)
Video | FAQ | 2025-01-30 Week 3 - Session 3 - TDS Jan 25 (2h 35m)
Video | FAQ | 2025-01-29 Week 3 - Session 2 - TDS Jan 25 (1h 28m)
Video | FAQ | 2025-01-28 Week 3 - Session 1 - TDS Jan 25 (2h 0m)
Video | FAQ | 2025-01-23 Week 2 - Session 4 - TDS Jan 25 (2h 5m)
Video | FAQ | 2025-01-22 Week 2 - Session 3 - TDS Jan 25 (1h 19m)
Video | FAQ | 2025-01-21 Week 2 - Session 2 - TDS Jan 25 (2h 15m)
Video | FAQ | 2025-01-20 Week 2 - Session 1 - TDS Jan 25 (2h 3m)
Video | FAQ | 2025-01-17 Week 1 - Session 2 - TDS Jan 25 (2h 17m)
Video | FAQ | 2025-01-16 Week 1 - Session 1 - TDS Jan 25 (1h 59m)
Video | FAQ | 2025-01-15 Introduction to Tools for Data Science Jan 25 (1h 18m)
Video | FAQ | Mock QP - TA Session - TDS 2024 12 19 18 51 (2h 25m)
Video | FAQ | Project 2 TA’s Session TDS 2024 12 05 19 52 (3h 0m)
Video | FAQ | TA’s Session TDS 2024 12 05 19 52 GMT+05 30 Recording (3h 0m)
Video | FAQ | ROE Session TDS 2024 11 16 19 52 (1h 13m)
Video | FAQ | TA’s Session TDS – 2024 11 07 19 58 GMT+05 30 – How to hack Project 1 and possibly ROE, pdf scrapi (2h 14m)
Video | FAQ | ROE Session TDS 2024 11 15 19 53 (2h 0m)
Video | FAQ | TA’s Session TDS – 2024 11 10 19 49 GMT+05 30 – Recording (1h 39m)
Video | FAQ | TA’s Session TDS 2024 11 14 19 54 (1h 52m)
Video | FAQ | TA’s Session TDS – 2024 11 04 16 56 IST – Recording (1h 55m)
Video | FAQ | TA’s Session TDS – 2024 10 27 19 55 IST – Recording (1h 45m)
Video | FAQ | TA’s Session TDS – 2024 10 24 19 50 IST – Recording 2 (2h 41m)
Video | FAQ | TA’s Session TDS – 2024-10-20 19:50 IST – Recording (1h 59m)
Video | FAQ | TAs Session - 2024 10 17 20:09:44 (1h 50m)
Video | FAQ | TA’s Session TDS – 2024 10 13 19 49 IST – Recording (2h 11m)
Video | FAQ | TA’s Session TDS – 2024 10 06 19 54 IST – Recording (1h 35m)
Video | FAQ | TA’s Session TDS – 2024 09 29 19 49 IST – Recording (1h 40m)
Video | FAQ | TA Session 29th August: Revision: Modules 7 to 9, CSS selectors, Javascript (1h 25m)
Video | FAQ | TA Session 25th August: Revision: Modules 4 to 6 (1h 51m)
Video | FAQ | TA Session 22 August: Mock and End Term, Revision: Modules 1 to 3 (2h 6m)
Video | FAQ | TA Session 19 August: Streamlit, Heroku (1h 47m)
Video | FAQ | TA Session 19 July: ROE Survival : Part 2 (2h 19m)
Video | FAQ | TA Session 18 July: The best ROE Survival Strategies, Tricks and Tips: Part 1 (2h 12m)
Video | FAQ | “TA session 12 July : week 5 , LLM API and embedding” (1h 54m)
Video | FAQ | TA session 14 July : ROE intro and strategy (1h 15m)
Video | FAQ | TA Session 11 July: LLM APi Setup and Use (2h 48m)
Video | FAQ | TA Session 7 July: Project Discussion, and Excel Filters, Slope Regression (1h 51m)
Video | FAQ | TA Session 4 July: More Pandas, Parquet processing, Forecast.ETS, Database Processing and Week 4 GA (1h 48m)
Video | FAQ | TA Session 30 June : More complex webscraping (2h 5m)
Video | FAQ | TA Session 27 June : Processing files with Pandas, Open Refine and Week 3 Graded Assignment (1h 51m)
Video | FAQ | TA session 20 June : Week 2 concepts ,API, JSON, PDF scraping and GA discussion (2h 32m)
Video | FAQ | A session 23 June : Web scraping with cookies and understanding JSON (1h 57m)
Video | FAQ | TA Session 16 June : Introduction to HTML and web scraping (2h 7m)
Video | FAQ | TA session 13 June : Introduction and week 1,2 recap (2h 2m)
Video | FAQ | ET revision : Attribute related to Anonymization and autocorrelation plot (1h 2m)
Video | FAQ | TA session : PYQ April 23 discussion part - 2 (2h 9m)
Video | FAQ | TA session : PYQ April 23 discussion part - 1 (1h 44m)
Video | FAQ | TA session : PYQ Dec 23 discussion (1h 12m)
Video | FAQ | 05 ROE revision session jan 24 (2h 56m)
Video | FAQ | 04 - XML intro and scraping (2h 3m)
Video | FAQ | 02 - Fundamentals of web scraping with urllib and BeautifulSoup (1h 37m)
Video | FAQ | 03 - Intermediate web scraping use of cookies (2h 19m)
Video | FAQ | 01 - Intro to Web scraping and HTML (1h 33m)
Video | FAQ | Live Session - 2 (0h 29m)

Mon, 01 Jan 0001 00:00:00 +0000

Local LLMs: Llamafile#

You would have heard of Large Language Models (LLMs) like GPT-4, Claude, and Llama. Some of these models are available for free, but most of them are not.

An easy way to run LLMs locally is Mozilla’s Llamafile. It’s a single executable file that works on Windows, Mac, and Linux. No installation or configuration needed - just download and run.

Watch this Llamafile Tutorial (6 min):

Mon, 01 Jan 0001 00:00:00 +0000

LLM Agents: Building AI Systems That Can Think and Act#

LLM Agents are AI systems that can define and execute their own workflows to accomplish tasks. Unlike simple prompt-response patterns, agents make multiple LLM calls, use tools, and adapt their approach based on intermediate results. They represent a significant step toward more autonomous AI systems.

What Makes an Agent?#

An LLM agent consists of three core components:

Mon, 01 Jan 0001 00:00:00 +0000

LLM Evaluations with PromptFoo#

Test-drive your prompts and models with automated, reliable evaluations.

PromptFoo is a test-driven development framework for LLMs:

Developer-first: Fast CLI with live reload & caching (promptfoo.dev)
Multi-provider: Works with OpenAI, Anthropic, HuggingFace, Ollama & more (GitHub)
Assertions: Built‑in (contains, equals) & model‑graded (llm-rubric) (docs)
CI/CD: Integrate evals into pipelines for regression safety (CI/CD guide)

To run PromptFoo:

Mon, 01 Jan 0001 00:00:00 +0000

Gemini Flash Experimental Image Generation and Editing APIs#

In March 2025, Google introduced native image generation and editing capabilities in the Gemini 2.0 Flash Experimental model. You can now generate and iteratively edit images via a single REST endpoint (Experiment with Gemini 2.0 Flash native image generation, Generate images | Gemini API | Google AI for Developers).

(How to use Latest Gemini 2.0 Native Image Generation with API?)

Mon, 01 Jan 0001 00:00:00 +0000

LLM Sentiment Analysis#

OpenAI’s API provides access to language models like GPT 4o, GPT 4o mini, etc.

For more details, read OpenAI’s guide for:

Start with this quick tutorial:

Here’s a minimal example using curl to generate text:

curl https://api.openai.com/v1/chat/completions \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 -d '{
 "model": "gpt-4o-mini",
 "messages": [{ "role": "user", "content": "Write a haiku about programming." }]
 }'

Let’s break down the request:

Mon, 01 Jan 0001 00:00:00 +0000

OpenAI TTS-1 for Text-to-Speech Generation#

OpenAI’s Text-to-Speech API (TTS-1) converts text into natural-sounding speech using state-of-the-art neural models. Released in March 2025, it offers multiple voices and control over speaking style and speed.

Simple speech generation#

To generate speech from text, send a POST request to the speech endpoint:

curl https://api.openai.com/v1/audio/speech \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 -H "Content-Type: application/json" \
 -d '{
 "model": "tts-1",
 "input": "Hello! This is a test of the OpenAI text to speech API.",
 "voice": "alloy"
 }' --output speech.mp3

Generation options#

Control the output with these parameters:

Mon, 01 Jan 0001 00:00:00 +0000

LLM Text Extraction#

JSON is one of the most widely used formats in the world for applications to exchange data.

This video explains how to use LLMs to extract structure from unstructured data, covering:

LLM for Data Extraction: Use OpenAI’s API to extract structured information from unstructured data like addresses.
JSON Schema: Define a JSON schema to ensure consistent and structured output from the LLM.
Prompt Engineering: Craft effective prompts to guide the LLM’s response and improve accuracy.
Data Cleaning: Use string functions and OpenAI’s API to clean and standardize data.
Data Analysis: Analyze extracted data using Pandas to gain insights.
LLM Limitations: Understand the limitations of LLMs, including potential errors and inconsistencies in output.
Production Use Cases: Explore real-world applications of LLMs for data extraction, such as customer service email analysis.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

LLM Video Screen-Scraping#

Video screen-scraping with LLMs is a powerful technique for extracting structured data from screen recordings. This approach works with any visible screen content and bypasses traditional web scraping limitations like authentication or anti-scraping measures.

Key benefits:

No setup cost or authentication handling
Works with any visible screen content
Full control over data exposure
Extremely cost-effective (< $0.001 per short video)
Bypasses anti-scraping measures
Handles varying formats and layouts

Quick Start Example#

Here’s a basic workflow using Google’s AI Studio and Gemini:

Mon, 01 Jan 0001 00:00:00 +0000

LLM Website Scraping#

LLM-powered website scraping represents a paradigm shift from traditional web scraping. Instead of writing explicit code to navigate DOM structures and CSS selectors, you describe what data you want and let the LLM figure out how to extract it.

This approach works for both simple public pages and complex scenarios requiring authentication, JavaScript rendering, and dynamic interactions. LLMs can write scraping code, control browsers, adapt to layout changes, and even handle captchas through browser automation.

Mon, 01 Jan 0001 00:00:00 +0000

LLM CLI: llm#

llm is a command-line utility for interacting with large language models—simplifying prompts, managing models and plugins, logging every conversation, and extracting structured data for pipelines.

Basic Usage#

Install llm. Then set up your OPENAI_API_KEY environment variable. See Getting started.

TDS Students: See Large Language Models for instructions on how to get and use OPENAI_API_KEY.

# Run a simple prompt
llm 'five great names for a pet pelican'

# Continue a conversation
llm -c 'now do walruses'

# Start a memory-aware chat session
llm chat

# Specify a model
llm -m gpt-4.1-nano 'Summarize tomorrow’s meeting agenda'

# Extract JSON output
llm 'List the top 5 Python viz libraries with descriptions' \
 --schema-multi 'name,description'

Or use llm without installation using uvx:

Mon, 01 Jan 0001 00:00:00 +0000

Making Open Data Useful: Lessons from Diagram Chasing#

Many datasets are technically available but practically unusable. Your job is to make them useful for people who are not data scientists. Design for the lowest friction, and let people share one precise slice of truth with a link.

See how Diagram Chasing does this across projects like CBFC Watch, Time Use Explorer, BLR Water Log, Votes in a name?, and Who is my neta? (CBFC Watch)

Mon, 01 Jan 0001 00:00:00 +0000

Interactive Notebooks: Marimo#

Marimo is a new take on notebooks that solves some headaches of Jupyter. It runs cells reactively - when you change one cell, all dependent cells update automatically, just like a spreadsheet.

Marimo’s cells can’t be run out of order. This makes Marimo more reproducible and easier to debug, but requires a mental shift from the Jupyter/Colab way of working.

It also runs Python directly in the browser and is quite interactive. Browse the gallery of examples. With a wide variety of interactive widgets, It’s growing popular as an alternative to Streamlit for building data science web apps.

Mon, 01 Jan 0001 00:00:00 +0000

Documentation: Markdown#

Markdown is a lightweight markup language for creating formatted text using a plain-text editor. It’s the standard for documentation in software projects and data science notebooks.

Watch this introduction to Markdown (19 min):

Common Markdown syntax:

# Heading 1
## Heading 2

**bold** and *italic*

- Bullet point
- Another point
 - Nested point

1. Numbered list
2. Second item

[Link text](https://url.com)
![Image alt](image.jpg)

```python
# Code block
def hello():
 print("Hello")
```

> Blockquote

There is also a GitHub Flavored Markdown standard which is popular. This includes extensions like:

Mon, 01 Jan 0001 00:00:00 +0000

Marks Dashboard#

T - Score Calculation#

The final score for the course is calculated as follows:

Components (All out of 100)	Weight
Grade Assessments (GA) Best 7/9	20%
Project 1 (P1)	20%
Project 2 (P2)	20%
Remote Online Exam (ROE)	20%
End Term (ET)	20%

End Term Eligibility: Best 4/5 first GAs Average >= 40#

T = 0.2 GA + 0.2 P1 + 0.2 P2 + 0.2 ROE + 0.2 ET

Mon, 01 Jan 0001 00:00:00 +0000

Marp: Markdown Presentation Ecosystem#

Marp (Markdown Presentation) is a powerful tool for creating presentations using Markdown. It converts Markdown files into slideshows, making it ideal for:

Technical Presentations: Code snippets, diagrams, and technical content
Documentation: Creating slide decks from existing documentation
Academic Slides: Research presentations with math equations and citations
Conference Talks: Professional presentations with custom themes
Teaching Materials: Educational content with rich formatting
GitHub Pages: Hosting presentations on the web

This tutorial covers using Marp in VS Code to create professional presentations.

Mon, 01 Jan 0001 00:00:00 +0000

Multimodal Embeddings#

Multimodal embeddings map text and images into the same vector space, enabling direct comparison between, say, a caption — “A cute cat” — and an image of that cat. This unified representation powers real-world applications like:

Cross-modal search (e.g. “find images of a sunset” via text queries)
Content recommendation (suggesting visually similar products to text descriptions)
Clustering & retrieval (grouping documents and their associated graphics)
Anomaly detection (spotting unusual image–text pairings)

By reducing different data types to a common numeric form, you unlock richer search, enhanced recommendations, and tighter integration of visual and textual data.

Mon, 01 Jan 0001 00:00:00 +0000

Narratives with Comics#

Sample sheet

Mon, 01 Jan 0001 00:00:00 +0000

Narratives with Excel#

Sample Excel

Mon, 01 Jan 0001 00:00:00 +0000

Network Analysis in Python#

You’ll learn how to use network analysis to identify clusters and connections between nodes in a dataset, covering:

Network Construction: Build a network from the IMDb database, where nodes represent actors and edges represent shared movie appearances.
Clustering: Apply clustering techniques to detect communities within the network, using the scikit-network library.
Matrix Operations: Utilize matrix operations to efficiently analyze actor relationships and interactions.
Community Detection: Implement algorithms to identify and interpret clusters, examining how different actor clusters are connected.
Application of Findings: Explore practical applications of network analysis, such as social network analysis and its potential uses in various domains.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Tunneling: ngrok#

Ngrok is a tool that creates secure tunnels to your localhost, making your local development server accessible to the internet. It’s essential for testing webhooks, sharing work in progress, or debugging applications in production-like environments.

Run the command uvx ngrok http 8000 to create a tunnel to your local server on port 8000. This generates a public URL that you can share with others.

Mon, 01 Jan 0001 00:00:00 +0000

Nominatim API with Python#

You’ll learn how to get the latitude and longitude of any city from the Nominatim API.

Introduction to Nominatim: Understand how Nominatim, from OpenStreetMap, works similarly to Google Maps for geocoding.
Installation and Import: Learn to install and import geopy and nominatim.
Using the Locator: Create a locator object using Nominatim and set up a user agent.
Geocoding an Address: Use locator.geocode to input an address (e.g., Eiffel Tower) and fetch geocoded data.
Extracting Data: Access detailed information like latitude, longitude, bounding box, and accurate address from the JSON response.
Classifying Locations: Identify the type of place (e.g., tourism, university) using the response data.
Practical Example: Geocode “IIT Madras” and retrieve its full address, type (university), and other relevant information.

Here are links and references:

Mon, 01 Jan 0001 00:00:00 +0000

JavaScript tools: npx#

npx is a command-line tool that comes with npm (Node Package Manager) and allows you to execute npm package binaries and run one-off commands without installing them globally. It’s essential for modern JavaScript development and data science workflows.

For data scientists, npx is useful when:

Running JavaScript-based data visualization tools
Converting notebooks and documents
Testing and formatting code
Running development servers

Here are common npx commands:

# Run a package without installing
npx http-server . # Start a local web server
npx prettier --write . # Format code or docs
npx eslint . # Lint JavaScript
npx ts-node script.ts # Run TypeScript directly
npx esbuild app.js # Bundle JavaScript
npx jsdoc . # Generate JavaScript docs

# Run specific versions
npx prettier@3.6 --write . # Use prettier 3.6

# Execute remote scripts (use with caution!)
npx github:user/repo # Run from GitHub

Watch this introduction to npx (6 min):

Mon, 01 Jan 0001 00:00:00 +0000

Local LLM Runner: Ollama#

ollama is a command-line tool for running open-source large language models entirely on your own machine—no API keys, no vendor lock-in, full control over models and performance.

Basic Usage#

Download Ollama for macOS, Linux, or Windows and add the binary to your PATH. See the full Docs ↗ for installation details and troubleshooting.

# List installed and available models
ollama list

# Download/pin a specific model version
ollama pull gemma3:1b-it-qat

# Run a one-off prompt
ollama run gemma3:1b-it-qat 'Write a haiku about data visualization'

# Launch a persistent HTTP API on port 11434
ollama serve

# Interact programmatically over HTTP
curl -X POST http://localhost:11434/api/chat \
 -H 'Content-Type: application/json' \
 -d '{"model":"gemma3:1b-it-qat","prompt":"Hello, world!"}'

Key Features#

Model management: list/pull — Install and switch among Llama 3.3, DeepSeek-R1, Gemma 3, Mistral, Phi-4, and more.
Local inference: run — Execute prompts entirely on-device for privacy and zero latency beyond hardware limits.
Persistent server: serve — Expose a local REST API for multi-session chats and integration into scripts or apps.
Version pinning: pull model:tag — Pin exact model versions for reproducible demos and experiments.
Resource control: --threads / --context — Tune CPU/GPU usage and maximum context window for performance and memory management.

Real-World Use Cases#

Quick prototyping. Brainstorm slide decks or blog outlines offline, without worrying about API quotas: ollama run gemma-3 'Outline a slide deck on Agile best practices'
Data privacy. Summarize sensitive documents on-device, retaining full control of your data: cat financial_report.pdf | ollama run phi-4 'Summarize the key findings'
CI/CD integration. Validate PR descriptions or test YAML configurations in your pipeline without incurring API costs: git diff origin/main | ollama run llama2 'Check for style and clarity issues'
Local app embedding. Power a desktop or web app via the local REST API for instant LLM features: curl -X POST http://localhost:11434/api/chat -d '{"model":"mistral","prompt":"Translate to German"}'

Read the full Ollama docs ↗ for advanced topics like custom model hosting, GPU tuning, and integrating with your development workflows.

Mon, 01 Jan 0001 00:00:00 +0000

Outlier Detection with Excel#

You’ll learn how to identify and handle outliers in data using Excel, covering:

Understanding Outliers: Definition of outliers and their impact on statistical analysis.
Calculating Quartiles: Using Excel formulas to calculate Q1 (first quartile) and Q3 (third quartile).
Interquartile Range (IQR): Finding the IQR by subtracting Q1 from Q3.
Determining Bounds: Calculating lower and upper bounds using 1.5 times the IQR.
Identifying Outliers: Using Excel functions to determine if data points fall outside the calculated bounds.
Visualizing Data: Creating box plots to visualize outliers and data distribution.
Handling Outliers: Deciding whether to exclude or keep outliers based on their impact on analysis.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Parsing JSON#

JSON is everywhere—APIs, logs, configuration files—and its nested or large structure can challenge memory and processing. In this tutorial, we’ll explore tools to flatten, stream, and query JSON data efficiently.

For example, we’ll often need to process a multi-gigabyte log file from a web service where each record is a JSON object.

This requires us to handle complex nested structures, large files that don’t fit in memory, or extract specific fields. Here are the key tools and techniques for efficient JSON parsing:

Mon, 01 Jan 0001 00:00:00 +0000

Profile Data with Python#

This session covers the use of the pandas_profiling library for generating comprehensive data reports in Python:

Library Installation and Import: Learn how to install and import the pandas_profiling library.
Profile Report Generation: Generate an HTML report with a single line of code using ProfileReport.
Descriptive Statistics: View detailed descriptive statistics such as variance, standard deviation, and kurtosis.
Outlier Detection: Identify and analyze outliers within the dataset.
Correlation Analysis: Understand how variables are correlated with each other using visual representations.
Handling Missing Values: Get insights on missing data and decide on imputation or removal strategies.
Initial Data Insights: Use the report to gather early warnings and insights before starting the data cleaning and modeling process.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Project 1 - LLM-based Automation Agent#

This project is due on 16 Feb 2025 EoD IST. Results will be announced by 26 Feb 2025.

For questions, use this Discourse thread.

Background#

You have joined the operations team at DataWorks Solutions, a company that processes large volumes of log files, reports, and code artifacts to generate actionable insights for internal stakeholders. In order to improve operational efficiency and consistency, the company has mandated that routine tasks be automated and integrated into their Continuous Integration (CI) pipeline.

Mon, 01 Jan 0001 00:00:00 +0000

Project 2 - TDS Solver#

This project is due on 31 Mar 2025 EoD IST. Results will be announced by 15 Apr 2025.

For questions, use this Discourse thread.

Background#

You are a clever student who has joined IIT Madras’ Online Degree in Data Science. You have just enrolled in the Tools in Data Science course.

To make your life easier, you have decided to build an LLM-based application that can automatically answer any of the graded assignment questions.

Mon, 01 Jan 0001 00:00:00 +0000

Project: Data Analyst Agent#

Deploy a data analyst agent. This is an API that uses LLMs to source, prepare, analyze, and visualize any data.

Your application exposes an API endpoint. You may host it anywhere. Let’s assume it’s at https://app.example.com/api/.

The endpoint must accept a POST request, e.g. POST https://app.example.com/api/ with a data analysis task description and optional attachments in the body. For example:

curl "https://app.example.com/api/" -F "questions.txt=@question.txt" -F "image.png=@image.png" -F "data.csv=@data.csv"

questions.txt will ALWAYS be sent and contain the questions. There may be zero or more additional files passed.

Mon, 01 Jan 0001 00:00:00 +0000

Project: LLM Analysis Quiz#

![WARNING] The API Project Endpoint will be at https://tds-llm-analysis.s-anand.net/project2 This will be active from 3:00 pm - 4:00 pm IST on Sat 29 Nov 2025.

In this project, students will build an application that can solve a quiz that involves data sourcing, preparation, analysis, and visualization using LLMs. You will also answer a viva about your design choices.

Fill out this Google Form. It asks for:

Your email address
A secret string (used to verify your requests).
A system prompt that resists revealing the given code word (which will be appended to the system prompt). Max 100 chars.
A user prompt that will override any such system prompt to reveal the code word. Max 100 chars.
Your API endpoint URL (where you will accept POST requests with quiz tasks). Prefer HTTPS.
Your GitHub repo URL (where your code is hosted). Make sure it’s public and has an MIT LICENSE when we evaluate. You may keep it private during development.

Prompt Testing#

Here’s how we will test your system and user prompts:

Mon, 01 Jan 0001 00:00:00 +0000

Project: LLM Code Deployment#

In this project, students will build an application that can build, deploy, update an application!

Build. The student:
- receives & verifies a request containing an app brief
- uses an LLM-assisted generator to build the app,
- deployes to GitHub Pages,
- then pings an evaluation API with repo details
Evaluate. The instructors:
- run automated static, dynamic (Playwright), and and LLM checks
- store and publish the results after the deadline
- send a second request tailored to the student’s codebase
Revise. The student
- verifies secret
- updates the app based on the request
- re‑deploys Pages
- then pings a second evaluation API with repo metadata.

Request#

The request is a JSON file like this:

Mon, 01 Jan 0001 00:00:00 +0000

Project: TDS Virtual TA#

Create a virtual Teaching Assistant Discourse responder.

Background#

You are a clever student who has joined IIT Madras’ Online Degree in Data Science. You have just enrolled in the Tools in Data Science course.

Out of kindness for your teaching assistants, you have decided to build an API that can automatically answer student questions on their behalf based on:

Course content with content for TDS Jan 2025 as on 15 Apr 2025.
TDS Discourse posts with content from 1 Jan 2025 - 14 Apr 2025.

Scrape the data#

To make sure you can answer these questions, you will need to extract the data from the above source.

Mon, 01 Jan 0001 00:00:00 +0000

Project 1 (After Week 3)#

P1 is available at https://exam.sanand.workers.dev/tds-2026-05-p1

Mon, 01 Jan 0001 00:00:00 +0000

Project 2 (After Week 6)#

P2 is available at https://exam.sanand.workers.dev/tds-2026-05-p2

Mon, 01 Jan 0001 00:00:00 +0000

Prompt Engineering#

Prompt engineering is the process of crafting effective prompts for large language models (LLMs).

One of the best ways to approach prompt engineering is to think of LLMs as a smart colleague (with amnesia) who needs explicit instructions.

The most authoritative guides are from the LLM providers themselves:

Here are some best practices:

Use prompt optimizers#

They rewrite your prompt to improve it. Explore:

Be clear, direct, and detailed#

Be explicit and thorough. Include all necessary context, goals, and details so the model understands the full picture.

Mon, 01 Jan 0001 00:00:00 +0000

Pydantic AI#

Pydantic is Python’s most widely used data validation library. Pydantic AI extends this to build production-grade AI agents with structured outputs and type safety.

What’s Pydantic?#

Python doesn’t enforce type hints at runtime. Pydantic validates data automatically using type annotations:

# /// script
# requires-python = ">=3.11"
# dependencies = ["pydantic"]
# ///

from pydantic import BaseModel, Field


class User(BaseModel):
 name: str
 age: int = Field(..., gt=0, le=120)
 email: str


# Automatic validation and type conversion
user = User(name="Alice", age="25", email="alice@example.com")
print(user.age, type(user.age)) # 25 <class 'int'>

# Invalid data raises ValidationError
try:
 User(name="Bob", age=-5, email="invalid")
except Exception as e:
 print(f"Validation failed: {e}")

Pydantic is used by FastAPI, LangChain, the OpenAI SDK, and 8,000+ packages on PyPI. It’s fast (core written in Rust), integrates with IDEs, and generates JSON schemas automatically.

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval Augmented Generation (RAG) with the CLI#

Retrieval Augmented Generation (RAG) combines retrieval (searching a knowledge base) with generation (using an LLM) to produce answers grounded in your own documents. Instead of relying solely on a general-purpose LLM, RAG lets you feed it the most relevant chunks from your corpus at query time, improving accuracy, reducing hallucinations, and allowing you to answer domain‑specific questions without fine‑tuning.

In particular, you can answer questions that are hard to answer with a keyword search. For example:

Mon, 01 Jan 0001 00:00:00 +0000

RAWgraphs#

RAWgraphs

You’ll learn about RAWGraphs and how to create data visualizations, covering:

Understanding RAWGraphs: Learn that RAWGraphs is an open-source data visualization tool designed to simplify the visualization of complex data for everyone.
Data Import: Discover how to import your dataset into RAWGraphs by copying and pasting it from any spreadsheet application.
Chart Selection: Understand how to choose from a variety of charts that are built on top of d3js, such as a scatter plot, suitable for your data.
Chart Creation: Create different kinds of charts, notably non-traditional charts that go beyond bars, lines, pie, etc.
Mapping Data Dimensions: Learn the process of mapping your dataset’s dimensions to the visual variables of your chosen chart. This includes examples like mapping rating to the horizontal axis, production budget to the vertical axis, box office to bubble areas, genre to color, and movie names to labels.
Customization and Export: Find out how to customize your visualization and export it as a vector or raster image, which can then be fine-tuned using any graphic editor.
Further Resources: Be informed about where to find more information on RAWGraphs’ website and how to contribute to the project through GitHub.

Watch these videos to learn how to create non-traditional data visualizations with RawGraphs.

Mon, 01 Jan 0001 00:00:00 +0000

Tools Glossary#

Every tool and term used across the course, with one line on what it’s for and where it’s taught.

Use Ctrl+F. Terms are grouped by what you’re trying to do, not alphabetically — you usually know the job, not the name.

Getting data off the web#

Term	What it is
Hidden / internal API	The JSON endpoint a page’s own JavaScript calls. Almost always better than parsing HTML → Hidden JSON APIs
httpx	Modern Python HTTP client; sync + async, HTTP/2, connection reuse
selectolax	Very fast HTML parser; CSS selectors (a lighter alternative to BeautifulSoup)
Playwright	Browser automation that renders JavaScript; auto-waits for elements
Selenium	The older browser-automation standard; manual waits
Patchright	Drop-in Playwright fork patched to hide automation signals
Camoufox	Anti-detect Firefox build with fingerprint spoofing
curl_cffi	HTTP client that impersonates a browser’s TLS/JA3 and HTTP/2 fingerprint
CDX API	Internet Archive’s index API — lists every snapshot of a URL
WARC	Web ARChive format; raw request/response records used by Common Crawl
JSON-LD	Structured data (schema.org) embedded in a page for search engines
Sitemap index	A sitemap listing other sitemaps; needs one extra level of parsing
robots.txt	A site’s machine-readable crawling preferences. Not a law; strong evidence → Legal & Ethical
Crawl-delay	A `robots.txt` directive asking for a minimum gap between requests

Being blocked (and not being blocked)#

Term	What it is
JA3 / JA4	Fingerprints of a TLS handshake; identifies your client before any header is read
HTTP/2 fingerprint	Identification from SETTINGS frames and header ordering
Turnstile	Cloudflare’s CAPTCHA replacement; usually invisible
Bot score	Cloudflare’s 1–99 automation likelihood; 1 = certainly bot
`cf_clearance`	Cookie proving a challenge was passed; bound to IP + User-Agent
WAF	Web Application Firewall — rule-based request filtering
Verified bot	A known-good crawler (Googlebot, uptime monitors) that rules should exempt
Exponential backoff	Doubling the wait after each failure
Jitter	Randomness added to backoff so clients don’t retry in lockstep
`Retry-After`	Header telling you exactly how long to wait after a 429
Idempotent	Safe to run twice — the second run changes nothing

Storing and querying#

Term	What it is
SQLite	In-process OLTP database; ideal for scraper state
DuckDB	In-process OLAP database; SQL over Parquet/CSV/JSON files
Parquet	Columnar, compressed file format; 5–10× smaller than CSV
OLTP / OLAP	Many small transactions vs few large analytical scans
Content hash	Hash of meaningful fields, used to detect changes
Stable ID	An identifier for a record that survives re-runs
polars	Fast DataFrame library; a Pandas alternative
Dead-letter queue	Where messages go after repeated processing failures

Documents, images, audio, video#

Term	What it is
trafilatura	Extracts the main article content from a page (drops boilerplate)
markdownify	Converts HTML to Markdown verbatim
MarkItDown	Microsoft converter: PDF/DOCX/PPTX/XLSX → Markdown
Jina Reader	Hosted `r.jina.ai/<url>` → clean Markdown
PyMuPDF	Fast PDF text and table extraction
Docling	Layout-aware document → structured Markdown
Surya	Open multilingual OCR and layout analysis
OCR	Optical Character Recognition — text from images
EXIF	Image metadata; can include GPS coordinates
VLM	Vision-Language Model — reads images and answers in text
Qwen3-VL	Leading open-weight vision-language family (2026)
Whisper / faster-whisper	Speech-to-text; `faster-whisper` is the quicker runtime
Parakeet	NVIDIA STT; fastest self-hosted English throughput
WhisperX	Whisper plus word-level alignment and speaker diarization
Diarization	Working out who spoke when
WER	Word Error Rate — the standard STT accuracy metric
ffmpeg / ffprobe	Convert/extract media; inspect media metadata

Recon and OSINT#

Term	What it is
OSINT	Open-Source Intelligence — findings assembled from public sources
Dorking	Using advanced search operators to find specific content
GHDB	Google Hacking Database — catalogue of exposure patterns
Certificate Transparency	Public append-only logs of issued TLS certificates
crt.sh	Search interface over CT logs
RDAP	The modern, structured replacement for WHOIS
Subdomain enumeration	Discovering hostnames for a domain (amass, subfinder)
Shodan / Censys	Search engines for internet-facing services
Corroboration	Confirming a claim across independent sources before asserting it
Provenance	The record of where a claim came from and when
Aggregation harm	Individually-public facts becoming harmful once combined
Gitleaks / TruffleHog	Secret scanners for git history

Security, cloud, CI/CD#

Term	What it is
Prompt injection	Untrusted text being treated as instructions by a model
Indirect injection	Injection delivered through content the model reads
OWASP LLM Top 10	The reference list of LLM application risks
Least privilege	Granting only the permissions strictly required
Multi-stage build	A Dockerfile pattern keeping build tools out of the final image
Layer cache	Docker’s reuse of unchanged build steps
Cold start	First-request latency after scaling to zero
Scale to zero	Running no instances (and paying nothing) when idle
IaC	Infrastructure as Code — declared in version-controlled files
Terraform state	Terraform’s record of what actually exists; never commit it
Drift	Real infrastructure diverging from the declared config
WIF	Workload Identity Federation — cloud auth without long-lived keys
At-least-once delivery	A message may arrive more than once; consumers must be idempotent
Pub/Sub	Publish–subscribe messaging; decouples producers from consumers

Course-wide#

Term	What it is
uv	Fast Python package/project manager used throughout
PEP 723	Inline script metadata — dependencies declared in the file itself
RAG	Retrieval-Augmented Generation
MCP	Model Context Protocol
OIDC	OpenID Connect
Structured output	Forcing a model to return schema-valid JSON

Mon, 01 Jan 0001 00:00:00 +0000

Command Cheatsheet#

The commands you’ll actually retype. Copy, adapt, move on.

uv — Python, without the ceremony#

uv run script.py # run a PEP 723 script; deps install automatically
uv run --with httpx script.py # add a one-off dependency
uvx ruff check . # run a tool without installing it
uvx duckdb -c "SELECT 42" # same, for DuckDB
uv init myproject && cd myproject # new project
uv add httpx polars duckdb # add dependencies
uv sync --frozen # install exactly the lockfile (use in CI)
uv python install 3.13 # install a Python version

The PEP 723 header that makes a single file runnable anywhere:

Mon, 01 Jan 0001 00:00:00 +0000

Regression with Excel#

You’ll learn to perform regression analysis using Excel, covering:

Data Preparation: Understanding the cleaned dataset and necessary columns for analysis.
Enabling the Tool: How to enable the Data Analysis Tool Pack in Excel.
Types of Regression: Differences between simple and multiple linear regression.
Setting Up Regression: Steps to input dependent (new deaths) and independent variables (new cases, new tests, new vaccinations, stringency index) for the analysis.
Interpreting Output: Reading the regression output, focusing on adjusted R-squared, significance value (F-test), and P-values.
Coefficient Interpretation: Understanding the impact of each independent variable on the dependent variable, including scaling factors (per 1000 units).
Model Evaluation: Evaluating the model based on significance values and understanding the implications of unexpected results (e.g., stringency index).
Further Analysis: Recognizing the need for additional analysis when encountering unexpected or inconclusive results.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

REST APIs#

REST (Representational State Transfer) APIs are the standard way to build web services that allow different systems to communicate over HTTP. They use standard HTTP methods and JSON for data exchange.

Watch this comprehensive introduction to REST APIs (52 min):

Key Concepts:

HTTP Methods
- GET: Retrieve data
- POST: Create new data
- PUT/PATCH: Update existing data
- DELETE: Remove data
Status Codes
- 2xx: Success (200 OK, 201 Created)
- 4xx: Client errors (400 Bad Request, 404 Not Found)
- 5xx: Server errors (500 Internal Server Error)

Here’s a minimal REST API using FastAPI. Run this server.py script via uv run server.py:

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval Augmented Generation#

Watch the walkthrough above, then dive into the notebook for step-by-step practice.

You will learn to implement Retrieval Augmented Generation (RAG) to enhance language models’ responses by incorporating relevant context, covering:

LLM Context Limitations: Understanding the constraints of context windows in large language models.
Retrieval Augmented Generation: The technique of retrieving and using relevant documents to enhance model responses.
Embeddings: How to convert text into numerical representations that are used for similarity calculations.
Similarity Search: Finding the most relevant documents by calculating cosine similarity between embeddings.
OpenAI API Integration: Using the OpenAI API to generate responses based on the most relevant documents.
Tourist Recommendation Bot: Building a bot that recommends tourist attractions based on user interests using embeddings.
Next Steps for Implementation: Insights into scaling the solution with a vector database, re-rankers, and improved prompts for better accuracy and efficiency.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

RevealJS: Web-Based Presentations#

RevealJS is a powerful HTML presentation framework that turns web pages into interactive slideshows. It’s particularly useful for:

Data Presentations: Interactive charts, live data demos
Technical Talks: Code with syntax highlighting, math equations
Web Portfolios: Showcase projects with live demos
Educational Content: Interactive learning materials
Conference Talks: Remote-friendly with speaker notes
Documentation: API and library showcases

This tutorial covers creating interactive presentations with RevealJS.

Mon, 01 Jan 0001 00:00:00 +0000

Scheduled Scraping with GitHub Actions#

GitHub Actions provides an excellent platform for running web scrapers on a schedule. This tutorial shows how to automate data collection from websites using GitHub Actions workflows.

Key Concepts#

Scheduling: Use cron syntax to run scrapers at specific times
Dependencies: Install required packages like httpx, lxml
Data Storage: Save scraped data to files and commit back to the repository
Error Handling: Implement robust error handling for network issues and HTML parsing
Rate Limiting: Respect website terms of service and implement delays between requests

Here’s a sample scrape.py that scrapes the IMDb Top 250 movies using httpx and lxml:

Mon, 01 Jan 0001 00:00:00 +0000

Scraping emarketer#

In this live scraping session, we explore a real-life scenario where Straive had to scrape data from emarketer.com for a demo. This is a fairly realistic and representative way of how one might go about scraping a website.

You’ll learn:

Scraping: How to extract data from web pages, including constructing URLs, fetching page content, and parsing HTML using packages like lxml and httpx.
Caching: Implementing a caching strategy to avoid redundant data fetching for efficiency and reliability.
Error Handling and Debugging: Practical tips for troubleshooting, such as using liberal print statements, breakpoints for in-depth debugging, and the concept of “rubber duck debugging” to clarify problems.
LLMs: Benefits of Gemini / ChatGPT for code suggestions and troubleshooting.
Real-World Application: How quick proofs of concept to showcase capabilities to clients, emphasizing practice over theory.

Mon, 01 Jan 0001 00:00:00 +0000

Scraping IMDb with JavaScript#

You’ll learn how to scrape the IMDb Top 250 movies directly in the browser using JavaScript on the Chrome DevTools, covering:

Access Developer Tools: Use F12 or right-click > Inspect to open developer tools in Chrome or Edge.
Inspect Elements: Identify and inspect HTML elements using the Elements tab.
Query Selectors: Use document.querySelectorAll and document.querySelector to find elements by CSS class.
Extract Text Content: Retrieve text content from elements using JavaScript.
Functional Programming: Apply map and arrow functions for concise data processing.
Data Structuring: Collect and format data into an array of arrays.
Copying Data: Use the copy function to transfer data to the clipboard.
Convert to Spreadsheet: Use online tools to convert JSON data to CSV or Excel format.
Text Manipulation: Perform text splitting and cleaning in Excel for final data formatting.

Here are links and references:

Mon, 01 Jan 0001 00:00:00 +0000

Scraping: Live Sessions#

Fundamentals of web scraping with urllib and BeautifulSoup

Intermediate web scraping use of cookies

XML intro and scraping

Mon, 01 Jan 0001 00:00:00 +0000

Scraping PDFs with Tabula#

You’ll learn how to scrape tables from PDFs using the tabula Python library, covering:

Import Libraries: Use Beautiful Soup for URL parsing and Tabula for extracting tables from PDFs.
Specify Save Location: Mount Google Drive to save scraped PDFs.
Identify PDF URLs: Parse the given URL to identify and select all PDF links.
Download PDFs: Loop through identified links, saving each PDF to the specified location.
Extract Tables: Use Tabula to read tabular content from the downloaded PDFs.
Control Extraction Area: Specify page and area coordinates to accurately extract tables, avoiding extraneous text.
Save Extracted Data: Convert the extracted table data into structured formats like CSV for further analysis.

Here are links and references:

Mon, 01 Jan 0001 00:00:00 +0000

Scraping with Excel#

You’ll learn how to import tables on the web using Excel, covering:

Data Import from Web: Use the query feature in Excel to scrape data from websites.
Establishing Web Connections: Connect Excel to a web page using a URL.
Using Query Editor: Navigate the query editor to view and manage web data tables.
Loading Data: Load data from the web into Excel for further manipulation.
Data Transformation: Remove unnecessary columns and transform data as needed.
Applying Transformations: Track applied steps in the sequence for reproducibility.
Refreshing Data: Refresh the imported data to get the latest updates from the web.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Scraping with Google Sheets#

You’ll learn how to import tables on the web using Google Sheets’s =IMPORTHTML() formula, covering:

Import HTML Formula: Use =IMPORTHTML(URL, “query”, index) to fetch tables or lists from a web page.
Granting Access: Allow access for formulas to fetch data from external sources.
Checking Imported Data: Verify if the imported table matches the data on the web page.
Handling Errors: Understand common issues and how to resolve them.
Sorting Data: Copy imported data as values and sort it within Google Sheets.
Freezing Rows: Use frozen rows to maintain headers while sorting.
Live Formulas: Learn how web data updates automatically when the source changes.
Other Import Functions: IMPORTXML, IMPORTFEED, IMPORTRANGE, and IMPORTDATA for advanced data fetching options.

Here are links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Splitting Text in Excel#

You’ll learn how to transform a single-column data set into multiple, organized columns based on specific delimiters using the “Text to Columns” feature.

Here are links used in the video:

US Senate Legislation - Votes

Mon, 01 Jan 0001 00:00:00 +0000

Spreadsheet: Excel, Google Sheets#

You’ll use spreadsheets for data cleaning and exploration. The most popular spreadsheet program is Microsoft Excel followed by Google Sheets.

You may be already familiar with these. If not, make sure to learn the basics of both.

Go through the Microsoft Excel video training and make sure you cover:

Watch this video for an introduction to Google Sheets (49 min):

Mon, 01 Jan 0001 00:00:00 +0000

Database: SQLite#

Relational databases are used to store data in a structured way. You’ll often access databases created by others for analysis.

PostgreSQL, MySQL, MS SQL, Oracle, etc. are popular databases. But the most installed database is SQLite. It’s embedded into many devices and apps (e.g. your phone, browser, etc.). It’s lightweight but very scalable and powerful.

Watch these introductory videos to understand SQLite and how it’s used in Python (34 min):

Mon, 01 Jan 0001 00:00:00 +0000

System Requirements#

This course requires the following:

Software to be installed on your computer.
System permissions to be granted.
Access to the Websites listed below.

Software#

Core tools:

Visual Studio Code
uv for Python
NodeJS for JavaScript
Docker or Podman
GitHub Copilot

Module-specific tools:

System permissions#

Open Developer Tools on Chrome / Edge - to use F12 or Inspect in browser
Open port 8000-9000 for local development server
pip install and npm install to install packages

Websites#

accounts.google.com
agents.md
ai.google.dev
aipipe.org
aiproxy.sanand.workers.dev
aircloak.com
airflow.apache.org
aistudio.google.com
anthropic.com
api-atlas.nomic.ai
api.example.com
apify.com
api.jina.ai
api.openai.com
api.open-notify.org
app.example.com
app.flourish.studio
apple.com
appsource.microsoft.com
arxiv.org
aspiegel.com
atlas.nomic.ai
azure.microsoft.com
base64decode.org
bbc.com
beautiful-soup-4.readthedocs.io
bing.com
blog.gramener.com
bolt.new
calendar.google.com
cb-prod.seek.study.iitm.ac.in
cdn.jsdelivr.net
cdn.openai.com
chat.deepseek.com
chatgpt.com
chat.openai.com
chat.qwen.ai
claude.ai
cli.github.com
cline.bot
cloud.google.com
cmdlinetips.com
codeium.com
code.visualstudio.com
coffee-reviews.prayashm.com
colab.research.google.com
console.cloud.google.com
context7.com
continue.dev
cors-test.codehappy.devs
cran.r-project.org
cursor.com
cyberscoop.com
dashboard.ngrok.com
dataforseo.com
data.gov
data.gov.in
data.gov.ru
datameet.org
datasetsearch.research.google.com
datasets.imdbws.com
datasette.io
dbeaver.io
dbfopener.com
deepwiki.com
desktop.github.com
devanshikat.github.io
developer.chrome.com
developer.imdb.com
developer.mozilla.org
developers.googleblog.com
developers.google.com
dev.mysql.com
discourse.onlinedegree.iitm.ac.in
diva-gis.org
dl.acm.org
docker.com
docs.anthropic.com
docs.astral.sh
docs.cursor.com
docs.docker.com
docs.getdbt.com
docs.github.com
docs.google.com
docs.klarna.com
docs.kumu.io
docs.lovable.dev
docs.nomic.ai
docs.npmjs.com
docs.python.org
docs.python-requests.org
docs.replit.com
docs.sqlalchemy.org
docs.together.ai
docs.windsurf.com
dopiaza.org
drive.google.com
drive.usercontent.google.com
duckdb.org
elastic.co
en.wikipedia.org
evanhahn.github.io
example.com
exam.sanand.workers.dev
explainxkcd.com
fastapi.tiangolo.com
fastapi-users.github.io
ffmpeg.lav.io
ffmpeg.org
flourish.studio
flukeout.github.io
fmwconcepts.com
forms.gle
freecodecamp.org
gemini.google
gemini.google.com
generativelanguage.googleapis.com
geopy.readthedocs.io
getdbt.com
github.com
github.github.com
gitingest.com
gitlab.com
gitlens.amod.io
git-lfs.github.com
git-scm.com
gnu.org
googlecloudcommunity.com
google.com
gramener.com
groups.google.com
heroku.com
howstat.com
httpbin.org
httpie.io
httrack.com
hub.docker.com
hub.getdbt.com
huggingface.co
imagemagick.org
imageoptim.com
imdb.com
imgs.xkcd.com
img.youtube.com
i.ytimg.com
jamendo.com
jeroenjanssens.com
jina.ai
jmespath.org
jqlang.org
jqplay.org
jsoneditoronline.org
json-generator.com
jsonlines.org
jsonlint.com
jsonpath.com
jsonpathfinder.com
json-schema.org
jsonschemavalidator.net
judgments.ecourts.gov.in
kaggle.com
keras.io
khanacademy.org
kumu.io
learn.microsoft.com
linkedin.com
llm.datasette.io
llmstxt.org
localhost
locator-service.api.bbci.co.uk
lovable.dev
lxml.de
lynx.invisible-island.net
macstories.net
magickstudio.imagemagick.org
mail.google.com
makersuite.google.com
mapshaper.org
marimo.app
marimo.io
marketplace.visualstudio.com
marp.app
matplotlib.org
medium.com
microsoft.com
mistral.ai
modelcontextprotocol.io
mongodb.com
mupdf.com
nchandrasekharr.github.io
news.ycombinator.com
ngrok.com
nodejs.org
nominatim.org
notebooklm.google.com
npmjs.com
numpy.org
observablehq.com
obsproject.com
ollama.com
openai.com
openpolicyagent.org
openrefine.org
openrouter.ai
ourworldindata.org
packaging.python.org
pages.github.com
pandas.pydata.org
pandoc.org
parquet.apache.org
pillow.readthedocs.io
platform.openai.com
playwright.dev
pngquant.org
podman.io
pokeapi.co
posit.co
postgresql.org
postman.com
pre-commit.com
projector.tensorflow.org
promptfoo.dev
pydantic-docs.helpmanual.io
pymotw.com
pymupdf.readthedocs.io
pypi.org
python-httpx.org
python-visualization.github.io
qgis.org
quotes.toscrape.com
rasagy.in
rattle.togaware.com
raw.githubusercontent.com
rawgraphs.io
realpython.com
redis.io
relational-data.org
revealjs.com
rishabhmakes.github.io
roocode.com
sanand0.github.io
s-anand.net
scikit-learn.org
scikit-network.readthedocs.io
screentogif.com
seaborn.pydata.org
seek.onlinedegree.iitm.ac.in
senate.gov
shapechef.com
sharp.pixelplumbing.com
simonwillison.net
soundcloud.com
sourceforge.net
sqlite.org
sqlitestudio.pl
sqlite-utils.datasette.io
sqlmodel.tiangolo.com
sqlzoo.net
squoosh.app
stackoverflow.blog
stackoverflow.com
statsmodels.org
stats.stackexchange.com
stedolan.github.io
story-b0f1c.web.app
storytellingwithdata.com
study.iitm.ac.in
superlinked.com
support.anthropic.com
support.apple.com
support.google.com
support.microsoft.com
support.office.com
survey.stackoverflow.co
swagger.io
tableau.com
tabula-py.readthedocs.io
tds.s-anand.net
textblob.readthedocs.io
texttospeech.googleapis.com
thoughtworks.com
timeanddate.com
tools.s-anand.net
tracebit.com
twitter.com
typesense.org
unstructured.io
upload.wikimedia.org
url.com
us-central1-aiplatform.googleapis.com
vercel.com
w3m.sourceforge.net
w3.org
weather-broker-cdn.api.bbci.co.uk
web.archive.org
webmaster.petalsearch.com
whitehouse.gov
whois.com
wikipedia.readthedocs.io
windsurf.com
wordlift.io
xpdfreader.com
yoast.com
your-app.vercel.app
youtu.be
youtube.com
youtubetranscript.com

Mon, 01 Jan 0001 00:00:00 +0000

TDS GPT Reviewer#

After the later parts of this course’s contents were written, we ran it through a Technical Content Reviewer GPT.

Take a look at the GPT’s instructions. These were generated by the OpenAI Prompt Generation tool.

As a **Content Reviewer** for a high school–level course on Tools in Data Science, your job is to evaluate provided content (such as text, code snippets, or references) with a focus on correctness, clarity, and conciseness, and offer actionable feedback for improvement.

1. **Check for Correctness and Consistency**
 - Verify technical and factual accuracy.
 - Ensure internal consistency without contradictions.
2. **Check for Clarity and Approachability**
 - Ensure content is understandable for a high school student with limited prior knowledge.
 - Identify and simplify jargon or advanced concepts.
3. **Check for Conciseness**
 - Assess if content is direct and free of unnecessary verbosity.
 - Identify areas for streamlining to enhance readability.
4. **Provide Feedback for Improvement**
 - Offer actionable suggestions for fixing, clarifying, or reorganizing content.
 - Propose alternative phrasing if text is vague, complex, or verbose.

# Steps

1. Carefully read the entire content before forming conclusions.
2. List factual inconsistencies or missing details causing confusion.
3. Suggest simpler terms or analogies for complex language.
4. Point out unnecessary repetition or filler text.
5. Provide direct examples of how to improve the highlighted issues.

# Output Format

Respond using **Markdown** with the following structure:

1. **Summary of Findings**
 - A concise paragraph outlining overall strengths and weaknesses.
2. **Detailed Review**
 - **Correctness and Consistency**: Note factual errors or inconsistencies, suggesting corrections.
 - **Clarity and Approachability**: Identify overly advanced or unclear sections, offering simpler alternatives.
 - **Conciseness**: Highlight long or repetitive sections with suggestions for tightening the text.
3. **Actionable Improvement Suggestions**
 - Provide specific sentences, bullet points, or rewritten examples to illustrate improvements.

# Notes

- Maintain a constructive review tone, not content generation.
- Even if content is perfect, confirm with suggestions for minor improvements (e.g., adding an example or clarifying a subtle point).

Content creation prompts#

In addition, here are a few prompts used to create the content:

Mon, 01 Jan 0001 00:00:00 +0000

TDS TA Instructions#

The TDS TA is a virtual assistant that helps you with your doubts.

It has been trained on course content created as follows:

# Clone the course repository
git clone https://github.com/sanand0/tools-in-data-science-public.git
cd tools-in-data-science-public

# Create a prompt file for the TA
PYTHONUTF8=1 uvx files-to-prompt --cxml *.md -o tds-content.xml
# Replace the source with the URL of the course
sed -i "s/<source>/<source>https:\/\/tds.s-anand.net\/#\//g" tds-content.xml

Additionally, we visit each of the evaluation links on https://exam.sanand.workers.dev/, copy it as Markdown, and add it to the content, called ga1.md, ga2.md, etc.

Mon, 01 Jan 0001 00:00:00 +0000

Topic Modeling#

You’ll learn to use text embeddings to find text similarity and use that to create topics automatically from text, covering:

Embeddings: How large language models convert text into numerical representations.
Similarity Measurement: Understanding how similar embeddings indicate similar meanings.
Embedding Visualization: Using tools like Tensorflow Projector to visualize embedding spaces.
Embedding Applications: Using embeddings for tasks like classification and clustering.
OpenAI Embeddings: Using OpenAI’s API to generate embeddings for text.
Model Comparison: Exploring different embedding models and their strengths and weaknesses.
Cosine Similarity: Calculating cosine similarity between embeddings for more reliable similarity measures.
Embedding Cost: Understanding the cost of generating embeddings using OpenAI’s API.
Embedding Range: Understanding the range of values in embeddings and their significance.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Transforming Images#

Image Processing with PIL (Pillow)#

Pillow is Python’s leading library for image processing, offering powerful tools for editing, analyzing, and generating images. It handles various formats (PNG, JPEG, GIF, etc.) and provides operations from basic resizing to complex filters.

Here’s a minimal example showing common operations:

# /// script
# requires-python = ">=3.11"
# dependencies = ["Pillow"]
# ///

from PIL import Image, ImageEnhance, ImageFilter


async def process_image(path: str) -> Image.Image:
 """Process an image with basic enhancements."""
 with Image.open(path) as img:
 # Convert to RGB to ensure compatibility
 img = img.convert("RGB")

 # Resize maintaining aspect ratio
 img.thumbnail((800, 800))

 # Apply enhancements
 img = ImageEnhance.Contrast(img).enhance(1.2)

 return img.filter(ImageFilter.SHARPEN)


if __name__ == "__main__":
 import asyncio

 img = asyncio.run(process_image("input.jpg"))
 img.save("output.jpg", quality=85)

Key features and techniques you’ll learn:

Mon, 01 Jan 0001 00:00:00 +0000

Unicode#

Ever noticed when you copy-paste some text and get garbage symbols? Or see garbage when you load a CSV file? This video explains why. It covers how computers store text (called character encoding) and why it sometimes goes wonky.

Learn about ASCII (the original 7-bit encoding system that could only handle 128 characters), why that wasn’t enough for global languages, and how modern solutions like Unicode save the day by letting us use any character from any language.

Mon, 01 Jan 0001 00:00:00 +0000

Python tools: uv#

Install uv.

uv is a fast Python package and project manager that’s becoming the standard for running Python scripts. It replaces tools like pip, conda, pipx, poetry, pyenv, twine, and virtualenv into one, enabling:

Python Version Management: uv installs and manages multiple Python versions, allowing developers to specify and switch between versions seamlessly.
Virtual Environment Handling: It automates the creation and management of virtual environments, ensuring isolated and consistent development spaces for different projects.
Dependency Management: With support for the pyproject.toml format, uv enables precise specification of project dependencies. It maintains a universal lockfile, uv.lock, to ensure reproducible installations across different systems.
Project Execution: The uv run command allows for the execution of scripts and applications within the managed environment, streamlining development workflows.

Here are some commonly used commands:

Mon, 01 Jan 0001 00:00:00 +0000

Vector Databases#

Vector databases are specialized databases that store and search vector embeddings efficiently.

Use vector databases when your embeddings exceed available memory or when you want it run fast at scale. (This is important. If your code runs fast and fits in memory, you DON’T need a vector database. You can can use numpy for these tasks.)

Vector databases are an evolving space.

The first generation of vector databases were written in C and typically used an algorithm called HNSW (a way to approximately find the nearest neighbor). Some popular ones are:

Mon, 01 Jan 0001 00:00:00 +0000

Serverless hosting: Vercel#

Serverless platforms let you rent a single function instead of an entire machine. They’re perfect for small web tools that don’t need to run all the time. Here are some common real-life uses:

Mon, 01 Jan 0001 00:00:00 +0000

Vibe Analysis#

Vibe analysis is analyzing data as if the analysis itself doesn’t exist—you focus only on business outcomes, skipping intermediate steps.

This approach emerged from vibe coding, where you code as if code doesn’t exist. With vibe analysis, you give an LLM full context, state your goal, and review only the answers. The LLM handles exploration, cleaning, modeling, visualization, and deployment.

Watch these comprehensive talks on vibe analysis and the changing role of data scientists:

Mon, 01 Jan 0001 00:00:00 +0000

Vibe-coding#

The rise of AI coding tools has introduced a new development philosophy: vibe-coding.

Vibe coding is different from AI coding. It’s “fully giving in to the vibes…” and “forgetting that the code even exists”, i.e., accept-all edits, don’t hand-edit diffs, etc. The goal is to get to a working prototype as fast as possible.

Ship scrappy first drafts fast: Focus on getting a working version quickly rather than polished code. AI can generate functional prototypes in minutes, allowing you to validate ideas before investing in refinement.

Mon, 01 Jan 0001 00:00:00 +0000

Vision Models#

You’ll learn how to use LLMs to interpret images and extract useful information, covering:

Setting Up Vision Models: Integrate vision capabilities with LLMs using APIs like OpenAI’s Chat Completion.
Sending Image URLs for Analysis: Pass URLs or base64-encoded images to LLMs for processing.
Reading Image Responses: Get detailed textual descriptions of images, from scenic landscapes to specific objects like cricketers or bank statements.
Extracting Data from Images: Convert extracted image data to various formats like Markdown tables or JSON arrays.
Handling Model Hallucinations: Address inaccuracies in extraction results, understanding how different prompts can affect output quality.
Cost Management for Vision Models: Adjust detail settings (e.g., “detail: low”) to balance cost and output precision.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Animated Data with Flourish#

You’ll learn how to create and customize animations in Flourish, covering:

Core Animation Principles: Understand how to use animation to both engage and inform, and learn the concept of “object constancy” to turn data visualizations into compelling data stories.
Configuring Race Charts: Master the animation controls for Line Chart Races and Bar Chart Races, including setting durations for timelines, transitions, and individual stages.
Morphing Between Chart Types: Learn to use the Line Bar Pie template to create seamless morphing transitions between different chart types (e.g., line to streamgraph) within a Flourish story.
Animating Scatter Plots: Discover how to set up and control animations in scatter plots, including using the “Animation stagger” setting to create dynamic, cascading dot movements.
Controlling Animation Logic: Understand how to manage animations for both related and unrelated datasets by toggling the “Only animate series with same name” option.
Animation in Other Templates: Get an overview of the animation settings available in various other templates, including Survey, Heatmap, Spider, and Hierarchy charts.
Advanced Data Explorer Animations: Learn about the powerful Data Explorer template (an Enterprise feature) and its ability to create advanced animations, such as morphing dots from a survey layout into their corresponding locations on a projection map.

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Animated Data with PowerPoint#

You’ll learn how to create an animated bar chart race in PowerPoint, covering:

Using the Morph Transition: How to leverage PowerPoint’s Morph feature to animate the resizing and reordering of bars between slides, which is the core technique for the race effect.
Manual Bar Creation: The process of using individual shapes (rectangles) instead of a standard chart and how to size them accurately to represent your data values.
Adding and Syncing Audio: How to record a voiceover for each slide, add background music that plays across the entire presentation, and sync them with the animations.
Automating Playback: Setting up automatic slide transitions to create a self-playing, video-like presentation that advances after the narration for each slide is complete.
Exporting as a Video: The steps to save your final animation as an MPEG-4 video file directly from PowerPoint and the suggestion of alternative methods like OBS for higher quality.
Prototyping Data Stories: Understanding the manual effort required for this method and how it allows for careful control when prototyping and crafting a specific visual narrative.

How to make bar chart race in PowerPoint#

Source: How to make a bar chart race in PowerPoint

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Forecasts with Excel#

Excel File used in the video

You’ll learn to perform exploratory data visualization on time-series financial data using Excel, covering:

Visualizing Trends with Sparklines: Use PivotTables and Sparklines to create compact, in-cell charts that reveal the performance trend of various financial securities (currencies, indices, commodities) over time.
Forecasting and Measuring Volatility: Learn to calculate key metrics like the average value, use the GROWTH function to forecast the next day’s performance, and measure volatility using standard deviation.
Analyzing Relationships with Scatter Plots: Create scatter plots to visually investigate how two different securities move in relation to each other.
Quantifying Correlation with R-Squared: Add trendlines and display the R-squared value on a scatter plot to quantitatively measure how much of the movement in one variable is explained by another.
Using the Data Analysis ToolPak: Learn how to enable and use Excel’s powerful Data Analysis ToolPak to perform more advanced statistical analysis.
Creating a Correlation Matrix: Use the ToolPak to efficiently generate a complete correlation matrix, showing the correlation coefficient for every pair of securities in the dataset.
Interpreting a Correlation Matrix with Conditional Formatting: Apply color scales to the matrix to instantly identify strong positive correlations, negative correlations, and outliers.
Discovering Insights and Forming Hypotheses: Learn how to use these visualizations to spot patterns and form data-driven hypotheses, such as the high correlation between the currencies of neighboring countries.

Here’s another version of this analysis, delivered as part of a talk:

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Machine Learning#

You’ll learn about improving customer retention, understanding black box models, and using clustering for market segmentation:

Churn Reduction: Use decision trees to identify customers likely to leave.
Cost Efficiency: Compare customer acquisition vs. retention costs.
Model Improvement: Apply SVMs and neural networks for better accuracy.
Project Challenges: Understand issues with black box models in implementation.
K-Means Clustering: Segment markets using demographic data.
Data Visualization: Interpret clustering results using maps and charts.
Correlation Analysis: Identify relationships between currency exchange rates.
Tool Proficiency: Utilize Excel, Python, and JavaScript for analysis and communication.
Practical Application: Tailor marketing strategies based on cluster characteristics.

Here are the links used in the video:

Mon, 01 Jan 0001 00:00:00 +0000

Visualizing Network Data with Kumu#

You’ll learn about visualizing and analyzing relationships and networks using Kumu, covering:

Understanding Kumu: What Kumu is and its primary function as a tool to visualize complex relationships within data. You’ll learn that it’s applicable beyond social analysis to any scenario involving relationships between entities.
Social Network Analysis: How Kumu facilitates social network analysis, helping to understand how different people and communities are connected, and identifying common interests.
Data Preparation for Kumu: The process of preparing raw data, specifically IMDb actor data for Indian movies, to be uploaded to Kumu. This includes filtering for only movies and Indian movies.
Creating Actor Networks: How to construct an actor collaboration matrix where each element denotes the number of movies two actors have acted in together. This involves a method using matrix multiplication of a movie-actor matrix with its transpose.
Optimizing Sparse Matrices: Understanding that actor collaboration matrices are often sparse (contain many zero entries) and how to make computations fast and memory-efficient using the compressed sparse row (CSR) format from the scipy library in Python.
Preparing Data for Kumu Upload: How to convert the processed matrix data into the “from node to node” format, along with the strength of the connection (number of shared movies), which is required for Kumu.
Filtering Data Effectively:
- Filtering by Year: How to filter movie data by release year (e.g., movies released after 1950) by converting the ‘start year’ column to an integer data type, and troubleshooting common issues like newline characters (/n) within string data.
- Filtering by Language/Region: How to filter for specific regions or languages, such as Indian movies, by applying language options within data processing functions or by filtering a dedicated region data frame.
- Filtering Actor Pairs: How to reduce the data size by filtering for actors who have acted in a minimum number of movies and for actor pairs who have acted together in a minimum number of movies.
Visualizing and Analyzing Networks in Kumu: How the prepared data creates a network of actors in Kumu, allowing you to observe clusters and understand direct and indirect connections.
Exploring Network Connections: How to search for specific actors (e.g., Mohanlal) and examine their direct connections (e.g., Mohanlal and Saikumar acted in 8 movies) and indirect connections within the network.
Introduction to Community Detection: A brief mention of community detection as a method to identify groups within the network and understand their interconnections, to be explored in other tutorials.
General Data Science Practices: The importance of using resources like Google and documentation for problem-solving, even for seemingly simple tasks, and the necessity of ensuring correct data types for operations.

Mon, 01 Jan 0001 00:00:00 +0000

Editor: VS Code#

Your editor is the most important tool in your arsenal. That’s where you’ll spend most of your time. Make sure you’re comfortable with it.

Visual Studio Code is, by far, the most popular code editor today. According to the 2025 StackOverflow Survey ~75% of developers use it. We recommend you learn it well. Even if you use another editor, you’ll be working with others who use it, and it’s a good idea to have some exposure.

Mon, 01 Jan 0001 00:00:00 +0000

Web Scraping with Playwright in Python#

Scrape JavaScript‑heavy sites effortlessly with Playwright.

(youtube.com)

Playwright offers:

JavaScript rendering: Executes page scripts so you scrape only after content appears. (playwright.dev)
Headless & headed modes: Run without UI or in a real browser for debugging. (playwright.dev)
Auto‑waiting & retry: Built‑in locators reduce flakiness. (playwright.dev)
Multi‑browser support: Chromium, Firefox, WebKit—all from one API. (playwright.dev)

Example: Scraping a JS‑Rendered Site#

We’ll scrape Quotes to Scrape (JS)—a site that loads quotes via JavaScript, so a simple requests call gets only an empty shell (quotes.toscrape.com). Playwright runs the scripts and gives us the real content:

Mon, 01 Jan 0001 00:00:00 +0000

VS Code — Part 1: Basics#

VS Code is not just a text editor. It is a full development workspace: files, terminal, Git, debugger, extensions, and settings all integrated in one window. Most beginners treat it like Notepad. That is fine early on, but it limits you.

Here’s what you will use VS Code for in real work:

Open project → write code → run in terminal → read errors
→ debug step by step → format code → commit to Git → repeat

Each section below maps a VS Code feature to a real need in that cycle.

Mon, 01 Jan 0001 00:00:00 +0000

VS Code — Part 2: Python, Debugging, Git, and Workflow#

This continues from Part 1. Make sure you have the vscode-lab/ folder open in VS Code with app.py, students.csv, config.json, and README.md created.

Python Setup: interpreter first, packages second#

Write this in app.py:

import csv
import json

with open("config.json", "r") as f:
 config = json.load(f)

print("App:", config["appName"])
print("Currency:", config["currency"])

with open("students.csv", "r") as f:
 reader = csv.DictReader(f)
 for row in reader:
 print(row["name"], row["marks"], row["city"])

Run it:

Mon, 01 Jan 0001 00:00:00 +0000

uv — Part 1: Python Environment Management#

uv is a fast, modern replacement for pip, venv, pyenv, and pipx — all rolled into one tool built by Astral. Instead of juggling multiple tools, you use uv for almost everything Python-environment-related.

Here’s the comparison at a glance:

What you need	Old way	With `uv`
Install Python	system installer / pyenv	`uv python install 3.12`
Create virtual env	`python -m venv .venv`	`uv venv`
Install packages	`pip install requests`	`uv pip install requests`
Project dependencies	`requirements.txt` manually	`pyproject.toml` + `uv.lock`
Run one-off tools	`pipx run ruff`	`uvx ruff`
Run a script	activate venv → python file.py	`uv run file.py`

Install uv#

# Linux/macOS
curl -LsSf https://astral.sh/uv/install.sh | sh

# Verify
uv --version

uv is a standalone binary — it doesn’t depend on Python being installed. You can install it on a fresh machine and let it manage Python for you.

Mon, 01 Jan 0001 00:00:00 +0000

uv — Part 2: Tools, Advanced Workflows, and Best Practices#

This continues from Part 1. You should already know uv venv, uv pip install, uv init, and uv add.

`requirements.txt` vs `pyproject.toml` — when to use which#

Situation	Use
Working on an old project or cloning from GitHub	`requirements.txt`
Your deployment platform expects `requirements.txt`	`requirements.txt`
Starting a new project	`pyproject.toml` + `uv.lock`
Want reproducible, locked environments	`pyproject.toml` + `uv.lock`
May eventually publish as a package	`pyproject.toml`

For learning: understand requirements.txt to work with existing projects. Use pyproject.toml for everything you build yourself.

Mon, 01 Jan 0001 00:00:00 +0000

Bash Scripting — Practical Notes#

Bash is useful because every Linux server, Docker container, CI runner, and cloud VM gives you a shell. Python is better for large programs, but Bash excels at gluing commands together: download → filter → transform → run script → save logs → schedule daily job. The core idea: small commands + pipes + files + automation.

1. How the terminal reads your command#

When you type:

Mon, 01 Jan 0001 00:00:00 +0000

Practical Git + GitHub Notes#

Git is used by developers to track code changes, work safely in branches, collaborate through GitHub, review changes through PRs, and recover from mistakes. Think of Git as your local project history machine, and GitHub as the hosted collaboration platform.

flowchart LR
 A[Working directory<br/>files you edit] -->|git add| B[Staging area<br/>selected changes]
 B -->|git commit| C[Local Git repo<br/>history on laptop]
 C -->|git push| D[GitHub remote<br/>shared repo]
 D -->|git fetch / pull| C

Git stores snapshots of your project, not just file versions. Branches are lightweight pointers to commits, which is why branching is fast and cheap.

Mon, 01 Jan 0001 00:00:00 +0000

SQLite — Practical Notes#

SQLite = one database inside one file. No server to install, no process to manage — just a .db file you can copy, query, and share.

CSV / JSON SQLite .db file PostgreSQL
simple files → real queryable DB → big server DB
weak search local + portable multi-user production

Use SQLite when you want:

scraping data
course projects
local APIs
small dashboards
RAG metadata
logs / cache
searchable notes
quick demos

Avoid SQLite when many machines/users must write to the same DB at the same time. Then use PostgreSQL.

Mon, 01 Jan 0001 00:00:00 +0000

HTTP Clients — Practical Notes#

1. Big Picture#

flowchart LR
 A[Client: curl / Python / Browser] --> B[HTTP Request]
 B --> C[Server / API]
 C --> D[HTTP Response]
 D --> A

 B --> B1[Method: GET/POST/PATCH]
 B --> B2[URL]
 B --> B3[Headers]
 B --> B4[Body JSON/Form/File]

 D --> D1[Status Code]
 D --> D2[Headers]
 D --> D3[Body JSON/Text/File]

HTTP client means: a tool or library used to talk to APIs.

Common clients:

Client	Best use
`curl`	Test/debug APIs from terminal
Python `requests`	Simple Python API work
Python `httpx`	Modern sync + async API work
httpbin	Safe API testing playground
Postman / REST Client	GUI or project-based API testing

2. HTTP Request Structure#

METHOD URL
Headers
Body

Example:

Mon, 01 Jan 0001 00:00:00 +0000

Requestly + Burp Suite — practical network debugging notes#

Use these tools only on your own apps, local projects, staging systems, or targets where you have permission. They are powerful because they can inspect and modify HTTP/HTTPS traffic.

flowchart LR
 A[Browser] --> B{Debug need?}
 B --> C[Requestly]
 B --> D[Burp Suite]

 C --> C1[Quick browser rules]
 C --> C2[Redirect API]
 C --> C3[Modify headers]
 C --> C4[Mock response]

 D --> D1[Full proxy]
 D --> D2[Intercept request]
 D --> D3[Inspect HTTPS]
 D --> D4[Replay in Repeater]

What these tools are used for#

Requestly is best for quick frontend/QA debugging directly in the browser: redirect URLs, modify headers/query params, mock API responses, and inject scripts/styles. Its browser extension works without setting up a local proxy, and Requestly describes it as useful for frontend debugging and simulating API responses.

Mon, 01 Jan 0001 00:00:00 +0000

Data Formats — Practical Notes#

Data formats are how programs store, send, read, and explain data.

In real developer work:

API response -> JSON
Python project -> TOML / pyproject.toml
GitHub Actions -> YAML
README / notes -> Markdown
Image/PDF in JSON -> Base64
Fast internal data -> MessagePack / Protobuf
LLM prompt data -> JSON / Markdown / TOON
All text -> UTF-8

flowchart LR
 A[Data in program] --> B{Where is it going?}
 B --> C[API / web app: JSON]
 B --> D[Config: YAML or TOML]
 B --> E[Docs: Markdown]
 B --> F[Binary inside text: Base64]
 B --> G[High speed internal: MessagePack]
 B --> H[LLM context: JSON / TOON / Markdown]

Start with a small lab#

mkdir data-formats-lab
cd data-formats-lab

# Use uv if you have it
uv init --bare
uv add pyyaml msgpack tomli-w

# Optional but very useful for JSON
sudo apt-get update
sudo apt-get install -y jq

# Check tools
python --version
jq --version

UTF-8: the rule below every format#

Always assume text is Unicode, and store/send it as UTF-8. Unicode gives each character a number; UTF-8 turns that character into bytes. Normal ASCII characters use 1 byte, many non-Latin characters use 2–3 bytes, and emoji can use 4 bytes.

Mon, 01 Jan 0001 00:00:00 +0000

GitHub Pages — Practical Notes#

GitHub Pages hosts static websites directly from a GitHub repository. Static means the final output is only:

HTML + CSS + JavaScript + images + fonts + JSON files

No backend server, no database server, no secret server-side environment variables. It is good for portfolios, documentation, project demos, blogs, landing pages, static dashboards, and built frontend apps. GitHub Pages supports free HTTPS and custom domains; project sites usually live at https://<username>.github.io/<repo>/.

Mon, 01 Jan 0001 00:00:00 +0000

LaTeX — Practical Notes#

LaTeX is used when Markdown is not enough: math equations, project reports, research-style PDFs, citations, figures, tables, and academic documents. Pandoc can convert Markdown to LaTeX/PDF without writing raw .tex files.

flowchart LR
 A[Markdown .md] -->|pandoc| B[PDF via LaTeX]
 C[LaTeX .tex file] -->|pdflatex / latexmk| D[PDF]
 D --> E[polished report]

Use LaTeX in two ways#

For beginners, start with Overleaf (overleaf.com) — no install, browser-based editor, live PDF preview. For local or automated work, install LaTeX and Pandoc so reports can be built from the terminal or GitHub Actions.

Mon, 01 Jan 0001 00:00:00 +0000

FastAPI — Practical Notes#

FastAPI is used when you want to turn Python code into an HTTP API. In real developer/data-science work, this means:

Python function → API endpoint → browser/app/script can call it

Examples:

train model locally → expose /predict
clean CSV → expose /clean
summarize text → expose /summarize
long job → start it now, check result later
cache expensive result → Redis

flowchart LR
 A[Client: browser / curl / Python] --> B[FastAPI app]
 B --> C[Python function]
 C --> D[JSON response]
 B --> E[Redis cache / job status]
 B --> F[Background task]

Start a project first.

Mon, 01 Jan 0001 00:00:00 +0000

Browser security → CORS → Middleware in FastAPI#

When a frontend in the browser calls an API, the browser checks origin.

Origin = scheme + domain + port

http://localhost:3000
https://localhost:3000
http://localhost:8000
https://api.example.com

These are different origins:

http://localhost:3000 frontend
http://localhost:8000 backend API

So this frontend call is cross-origin:

fetch("http://localhost:8000/predict")

The browser has a security rule called Same-Origin Policy. It stops JavaScript on one origin from freely reading sensitive data from another origin. CORS is the controlled way for a server to say: “This frontend origin is allowed to read my API response.” MDN defines CORS as an HTTP-header mechanism where the server indicates which other origins may access resources.

Mon, 01 Jan 0001 00:00:00 +0000

Google OAuth with FastAPI: sessions, bearer tokens, browser storage#

Before Google OAuth, understand two login styles.

flowchart LR
 A[User logs in] --> B{How does server remember user?}
 B --> C[Session cookie]
 B --> D[Bearer token]

A session means the browser stores a small cookie. On every request, the browser automatically sends it back.

sequenceDiagram
 participant B as Browser
 participant API as FastAPI

 B->>API: POST /login
 API-->>B: Set-Cookie: session=abc...
 B->>API: GET /me with Cookie
 API-->>B: user data

In FastAPI/Starlette, SessionMiddleware gives signed cookie-based sessions, and the session cookie is HttpOnly, so JavaScript cannot read it directly.

Mon, 01 Jan 0001 00:00:00 +0000

Config Management — Practical Notes#

Config management means same code, different values. Your app code should not change between laptop, GitHub Actions, Hugging Face Spaces, staging, or production. Only environment variables change. This is the 12-factor idea: store config in the environment, not hardcoded in code.

flowchart LR
 Code["main.py / app.py<br/>same everywhere"]
 Local["Local .env<br/>dev values"]
 Bash["~/.bashrc<br/>machine-wide values"]
 GitHub["GitHub Actions Secrets<br/>CI/CD values"]
 HF["Hugging Face Space Secrets<br/>deployment values"]

 Local --> Code
 Bash --> Code
 GitHub --> Code
 HF --> Code

Real developers use config management for database URLs, API keys, model names, debug mode, allowed CORS origins, logging level, ports, feature flags, and deployment secrets. The golden rule: commit code and .env.example; never commit .env.

Mon, 01 Jan 0001 00:00:00 +0000

Podman Compose — Practical Notes#

Use containers when your app is not one process anymore. Real projects usually have API + database + frontend + admin UI + cache. Podman runs each part in an isolated container; Compose starts/stops them together with one file.

flowchart LR
 Dev[Developer laptop] --> Compose[compose.yml]
 Compose --> API[backend container]
 Compose --> FE[frontend container]
 Compose --> DB[postgres container]
 Compose --> UI[pgadmin container]
 DB --> Vol[(named volume)]
 API --> DB
 UI --> DB

Podman is free and open source (Apache License). podman-compose is a Compose implementation for Podman focused on rootless and daemonless use.

Mon, 01 Jan 0001 00:00:00 +0000

Deployment Platforms — Practical Notes#

Deployment means: your app stops living only on localhost and starts running on a public URL.

Real developer flow:

flowchart LR
 A[Code on laptop] --> B[Test locally]
 B --> C[Push to GitHub]
 C --> D[Platform builds app]
 D --> E[Platform runs app]
 E --> F[Public URL]
 F --> G[Logs, health checks, env vars]

Use deployment platforms when you want to share APIs, demos, dashboards, ML apps, assignments, prototypes, or production services.

Mon, 01 Jan 0001 00:00:00 +0000

Python + FastAPI Logging and Testing — practical notes#

In real developer work, logging answers “what happened in the running app?” and testing answers “does the code still work after changes?”

flowchart LR
 User[User / Browser / Client] --> API[FastAPI App]
 API --> Code[Business Logic]
 Code --> DB[(DB / File / Cache)]

 API --> Logs[Logs: what happened?]
 Code --> Tests[Tests: does it still work?]

 Logs --> Debug[Debug production issue]
 Tests --> SafeChange[Change code safely]

Python’s standard logging module is the normal built-in way to create loggers with logging.getLogger(__name__), and pytest is commonly used for small readable tests that scale to bigger apps. FastAPI’s TestClient lets you test API endpoints using normal Python test functions and normal assert statements.

Mon, 01 Jan 0001 00:00:00 +0000

Observability + Prometheus with Python/FastAPI#

Observability means: when your app runs, you can understand what is happening from outside. OpenTelemetry defines observability around telemetry data such as logs, metrics, and traces; the application must be instrumented to emit this data. Prometheus is mainly for metrics: it collects/scrapes metric values from HTTP endpoints, stores them as time series, and lets you query them with PromQL.

flowchart LR
 User[User / Client] --> API[FastAPI App]

 API --> Logs[Logs<br/>What happened?]
 API --> Metrics[Metrics<br/>How much / how fast?]
 API --> Traces[Traces<br/>Where did time go?]

 Logs --> Debug[Debug errors]
 Metrics --> Alert[Alert on problems]
 Traces --> Slow[Find slow service/request]

In simple words:

Mon, 01 Jan 0001 00:00:00 +0000

Cloudflare Tunnel + other tunnels — practical developer notes#

A tunnel is used when your app is running on your laptop/server, but someone outside your network needs to access it. Example: webhook testing, demo to professor, mobile app testing, temporary API sharing, or exposing a home server safely.

flowchart LR
 Dev[Your laptop] --> App[FastAPI app<br/>localhost:8000]
 Friend[Friend / webhook / professor] -.cannot directly reach.-> App
 Friend --> PublicURL[Public tunnel URL]
 PublicURL --> TunnelProvider[Cloudflare / ngrok / Tailscale]
 TunnelProvider --> App

First understand localhost, 127.0.0.1, 0.0.0.0, LAN IP, public IP#

When you run FastAPI normally:

Mon, 01 Jan 0001 00:00:00 +0000

Local LLM Tools — Practical Notes#

One-page summary of all tools#

Tool	Best use	Interface	Best user
LM Studio	GUI-based local LLM usage, model search, chat, local API, document chat	GUI + CLI + API	Beginner, AI user, developer testing
Ollama	Fast local model running, CLI, local REST API, coding-tool integration	CLI + API + Docker	Developer prototyping
llama.cpp	Deep control over GGUF models, quantization, CPU/GPU tuning, offline inference	CLI + server + C/C++ backend	Systems-minded developer
vLLM	Production LLM serving, batching, high-throughput GPU inference	Python + server + Docker	Backend/ML engineer
MLX / mlx-lm	Apple Silicon-native inference/fine-tuning	Python + CLI + Apple stack	Mac / Apple developer

Simple choice:

Mon, 01 Jan 0001 00:00:00 +0000

6. LM Studio: best beginner GUI and local API tool#

6.1 What it is#

LM Studio is a desktop app for downloading, managing, chatting with, and serving local models. It supports local model search/download, chat UI, MCP servers, OpenAI-like endpoints, prompt/config management, and document chat. It runs on macOS, Windows, and Linux.

Use it when students need to see local AI working quickly.

6.2 Installation#

Download the app from the official LM Studio website.

Mon, 01 Jan 0001 00:00:00 +0000

8. llama.cpp: best low-level control tool#

8.1 What it is#

llama.cpp is a C/C++ inference engine for running LLMs locally, especially GGUF models. It supports many backends including Metal, CUDA, HIP/ROCm, Vulkan, SYCL, OpenCL, WebGPU, CPU backends, and more.

Use llama.cpp when you want to understand what is really happening under the hood.

8.2 Installation#

macOS/Linux with Homebrew:

brew install llama.cpp

Windows:

winget install llama.cpp

Build from source:

Mon, 01 Jan 0001 00:00:00 +0000

10. MLX / mlx-lm: Apple Silicon-native path#

10.1 What it is#

MLX is Apple’s machine-learning framework for Apple Silicon. It is designed for efficient and flexible machine learning on Apple Silicon, has Python/C++/Swift/C support, uses a NumPy-like API, supports CPU/GPU, and is designed around unified memory.

mlx-lm is the LLM package for generating text and fine-tuning LLMs on Apple Silicon. It supports Hugging Face Hub integration, quantization, upload, low-rank/full fine-tuning, and distributed inference/fine-tuning.

Mon, 01 Jan 0001 00:00:00 +0000

Prompt Engineering — Chapter 1: Foundations#

Prompt engineering is not magic wording. It is the habit of giving an AI model a clear task, useful context, strong boundaries, and a verifiable output format.

A good beginner rule:

Bad prompt = vague wish
Good prompt = task + context + constraints + output format + examples when needed

flowchart LR
 A[Your goal] --> B[Prompt]
 B --> C[LLM]
 C --> D[Output]
 D --> E{Good enough?}
 E -->|No| F[Add missing context / format / examples]
 F --> B
 E -->|Yes| G[Use or save prompt]

Official docs from OpenAI and Anthropic agree on the core idea: be clear, specific, structured, and iterative. Claude additionally recommends XML tags heavily for complex prompts. OpenAI recommends putting important instructions first, separating context clearly, and using examples/output formats for reliability.

Mon, 01 Jan 0001 00:00:00 +0000

Prompt Engineering — Chapter 2: Reliable Reasoning and Output Control#

Chapter 1 gave the basic prompt parts: task, context, constraints, examples, and output format. Chapter 2 is about making the model reason better, avoid vague answers, and return output you can actually use.

A good mental model:

Beginner prompt = ask the model to answer
Reliable prompt = ask the model to solve + check + format

flowchart LR
 A[Task] --> B[Prompt with context]
 B --> C[Model reasons internally]
 C --> D[Draft answer]
 D --> E[Self-check against criteria]
 E --> F[Final structured answer]

Do not chase hundreds of fancy prompting tricks. Most reliable prompting comes from five habits:

Mon, 01 Jan 0001 00:00:00 +0000

Prompt Engineering — Chapter 3: Prompted Applications and Production Practice#

Chapter 1 taught prompt basics. Chapter 2 taught reliable reasoning and output control. Chapter 3 is where prompting becomes a real application: the model uses documents, calls tools, follows safety boundaries, and is tested before changes go live.

A normal prompt answers from model memory. A production prompt should answer from fresh context, tools, and rules your application controls.

Chat prompt = user asks → model replies
AI application = user asks → retrieve/use tools/check rules → model replies

flowchart LR
 A[User question] --> B[App layer]
 B --> C{Need knowledge?}
 C -->|Yes| D[Retrieve docs / search / DB]
 C -->|No| E[Use prompt only]
 B --> F{Need action?}
 F -->|Yes| G[Tool / API call]
 F -->|No| H[No tool]
 D --> I[Final prompt with evidence]
 G --> I
 E --> I
 H --> I
 I --> J[Model response]
 J --> K[Validate + log + return]

The simple rule:

Mon, 01 Jan 0001 00:00:00 +0000

AI Coding Assistants#

AI coding assistants are not just autocomplete. Used well, they can write, test, refactor, and explain entire codebases. Used poorly, they produce confident-sounding but wrong code. This page is about using them well.

Claude Code#

Claude Code is Anthropic’s agentic CLI coding tool. It reads your project files, writes code, runs commands, and iterates — all from the terminal.

Mon, 01 Jan 0001 00:00:00 +0000

Context Engineering#

Context engineering is the discipline of designing what goes in the context window — the system prompt, background documents, instructions, tools, and conversation history — to make an LLM behave consistently and correctly across all inputs.

Prompt Engineering vs Context Engineering

Prompt Engineering: How to phrase one query to get a good answer
Context Engineering: How to architect the entire context so the model behaves reliably on every query

The Context Window Budget#

Every LLM has a finite context window (measured in tokens). You decide how to spend it:

Mon, 01 Jan 0001 00:00:00 +0000

LangSmith & LiteLLM#

Two tools that give you visibility and control over your LLM API usage:

LangSmith — trace every LLM call: see inputs, outputs, latency, cost per call
LiteLLM — one API to call 100+ LLM providers; built-in spend tracking and budget limits

LangSmith#

LangSmith is an observability platform for LLM applications. Every call is logged as a trace — you see exactly what prompt went in, what came out, how long it took, and what it cost.

Mon, 01 Jan 0001 00:00:00 +0000

LLM Architecture Survey#

Understanding how LLMs work under the hood helps you use them better: why does temperature affect creativity? Why do longer prompts cost more? Why does CoT work? This topic gives you the mental models.

The Transformer (2017 — The Foundation of Everything)#

Every modern LLM is built on the Transformer architecture introduced in “Attention Is All You Need” (Vaswani et al., 2017).

Core Components#

Input text: "The cat sat on"
 ↓
[Tokenization] → [1192, 4690, 6654, 319] (token IDs)
 ↓
[Token Embedding] → 4 vectors of 768 dims each
 ↓
[Positional Encoding] → adds position information
 ↓
[Transformer Blocks] × N layers:
 │
 ├── [Self-Attention] — tokens attend to each other
 │ "sat" pays attention to "cat" (subject) and "on" (position)
 │
 ├── [Feed-Forward Network] — per-token transformation
 │ applies learned patterns to each position
 │
 └── [Layer Norm + Residual] — stability and gradient flow
 ↓
[Output Head] → probability over 50,000+ tokens
 ↓
Sample: "mat" (with probability 0.73)

Self-Attention: The Core Mechanism#

import numpy as np

def scaled_dot_product_attention(Q, K, V, mask=None):
 """
 Q: queries (seq_len × d_k)
 K: keys (seq_len × d_k)
 V: values (seq_len × d_v)
 """
 d_k = Q.shape[-1]

 # Step 1: compute attention scores
 scores = Q @ K.T / np.sqrt(d_k) # scale to prevent vanishing gradients

 # Step 2: optional masking (for decoder — can't look at future tokens)
 if mask is not None:
 scores = scores + mask * -1e9 # -inf → 0 after softmax

 # Step 3: softmax to get attention weights (sum to 1)
 weights = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True)

 # Step 4: weighted sum of values
 output = weights @ V

 return output, weights

# Intuition: for each token, how much should I "attend" to other tokens?
# High weight = "this token is very relevant to understanding me"

Why attention is powerful: “The animal didn’t cross the street because it was too tired.” What does “it” refer to? Attention lets the model look back at all previous tokens and learn that “it” → “animal” has high relevance.

Mon, 01 Jan 0001 00:00:00 +0000

LLM CLI Tools#

CLI tools for LLMs bring AI into your existing shell workflows. Pipe text in, get structured answers out. No Python script needed for quick tasks.

Simon Willison’s `llm` CLI#

llm is the most popular CLI for querying language models from the terminal. It supports 50+ models via plugins and works seamlessly with Unix pipes.

Installation#

# Install with UV (recommended)
uv tool install llm

# Or with pip
pip install llm

# Verify
llm --version

Setting Up API Keys#

# OpenAI (default model)
llm keys set openai
# → Paste your sk-... key

# Anthropic
llm install llm-anthropic
llm keys set anthropic
# → Paste your sk-ant-... key

# Google Gemini
llm install llm-gemini
llm keys set gemini
# → Paste your AIza... key

# Ollama (local — no key needed)
llm install llm-ollama
# Ollama must be running: ollama serve

Basic Usage#

# Simple query (uses default model)
llm "What is the difference between TCP and UDP?"

# Specify a model
llm -m claude-sonnet-4-6 "Explain Docker in one sentence."
llm -m gpt-4o-mini "Summarize this: $(cat readme.md)"
llm -m gemini-2.0-flash "What is 847 × 293?"

# Ollama (local, free)
llm -m ollama/llama3.2 "Write a haiku about Python."

# List all available models
llm models list

The Power: Unix Pipes#

# Summarize a file
cat long_document.txt | llm "Summarize this in 5 bullet points"

# Explain code
cat main.py | llm "Explain what this code does and find any bugs"

# Fix a Python error
python script.py 2>&1 | llm "This is a Python error. What is the fix?"

# Translate a file
cat README.md | llm "Translate this to Hindi"

# Generate commit messages
git diff --staged | llm "Write a concise git commit message for these changes"

# Review PR diffs
gh pr diff 42 | llm -m claude-sonnet-4-6 "Review this PR. Focus on bugs and security issues."

# Summarize logs
tail -n 100 /var/log/app.log | llm "Summarize these logs. Flag any errors or anomalies."

# Convert formats
cat data.csv | llm "Convert this CSV to a Markdown table"

# Extract structured data
cat invoice.txt | llm "Extract: vendor name, amount, date. Output as JSON."

System Prompts and Templates#

# Use a system prompt
llm -s "You are a senior Python code reviewer. Be direct and critical." \
 "$(cat main.py)"

# Save a template
llm templates edit code-reviewer

# Template format (YAML in your editor)
system: |
 You are an expert code reviewer. For each issue you find, provide:
 1. The line number
 2. The problem
 3. The fix
model: claude-sonnet-4-6

# Use the template
cat main.py | llm -t code-reviewer

# Create a "commit message" template
llm templates edit commit-msg
# system: "Write a concise, imperative git commit message. Max 72 chars for subject line."

Conversations#

# Start a conversation (maintains history in session)
llm chat -m claude-sonnet-4-6

# Or continue the last conversation
llm -c "What was the last thing I asked about?"

# Named conversation
llm -s "You are a Python tutor" --conversation my-python-session "What is a generator?"
llm -c --conversation my-python-session "Give me an example."

Logging and History#

# All queries are logged by default
llm logs # show recent queries
llm logs -n 20 # last 20
llm logs --json # as JSON
llm logs --json | jq '.[] | {prompt, response}' # pipe to jq

# The logs database is at:
# ~/.config/io.datasette.llm/logs.db

# Query it with datasette
datasette ~/.config/io.datasette.llm/logs.db

aichat#

aichat is a feature-rich terminal AI client with a built-in shell integration mode.

Mon, 01 Jan 0001 00:00:00 +0000

Multimodal Inputs#

Modern LLMs aren’t just text-in, text-out. You can send images, PDFs, and audio directly in API requests and get back rich, structured analysis. This is the foundation of document processing pipelines, visual QA, and OCR-free data extraction.

How Multimodal APIs Work#

The API accepts a content list instead of a plain string. Each item in the list is a content block — either text, image, or document:

Mon, 01 Jan 0001 00:00:00 +0000

Prompt Caching#

Every time you call an LLM API, you pay to process the entire prompt — even if the system prompt, the 50-page document, or the tool definitions haven’t changed since the last call. Prompt caching fixes this: process once, read from cache for a fraction of the cost.

Real numbers (Anthropic Claude Sonnet 4.6)

Standard input: $3.00 / MTok
Cache write: $3.75 / MTok (1.25×)
Cache read: $0.30 / MTok (0.10×)

A 10,000-token system prompt read 100 times:

Mon, 01 Jan 0001 00:00:00 +0000

Similarity Search#

You have a million embeddings. A user queries arrive. Brute-force cosine similarity over 1M × 1536-dim vectors takes seconds. Approximate Nearest Neighbor (ANN) indices make it milliseconds — with only a small loss in recall.

The Problem: Brute-Force Doesn’t Scale#

import numpy as np
import time

n_docs = 1_000_000
dim = 1536

# Simulate 1M document embeddings
corpus = np.random.randn(n_docs, dim).astype(np.float32)
corpus /= np.linalg.norm(corpus, axis=1, keepdims=True) # normalize

query = np.random.randn(dim).astype(np.float32)
query /= np.linalg.norm(query)

# Brute force
start = time.time()
similarities = corpus @ query
top5 = np.argsort(similarities)[::-1][:5]
elapsed = time.time() - start

print(f"Brute force: {elapsed*1000:.0f}ms for {n_docs:,} docs")
# → ~1,200ms on CPU — too slow for a web API

ANN indices trade a small amount of accuracy for 100–1000x speed:

Mon, 01 Jan 0001 00:00:00 +0000

Pydantic & Structured Output#

LLMs return free-form text. Your application needs typed, validated data. Instructor is the bridge — it wraps any LLM client and guarantees structured, validated Pydantic output with automatic retry on failure.

Why this matters

Without structured output, you write brittle regex parsers that break on every model update. With Instructor + Pydantic, you define a schema once and the LLM fills it.

Mon, 01 Jan 0001 00:00:00 +0000

Vector Embeddings#

An embedding is a fixed-length list of numbers (a vector) that represents the meaning of a text. Texts with similar meanings have vectors that are close to each other in high-dimensional space. This is the mathematical foundation of semantic search, RAG, and recommendation systems.

Intuition “King” − “Man” + “Woman” ≈ “Queen”

This isn’t magic — embeddings place semantically related words in similar geometric regions of the vector space.

Mon, 01 Jan 0001 00:00:00 +0000

Chunking Strategies#

The most underrated step in RAG. Better chunking → better retrieval → better answers. No amount of fancy reranking rescues bad chunks.

Why Chunking Matters#

Your embedding model has a token limit (typically 512–8192 tokens). You can’t embed an entire PDF at once. But chunks that are too small lose context; chunks too large dilute the signal.

The goal: chunks that are semantically coherent and self-contained enough to answer a question on their own.

Mon, 01 Jan 0001 00:00:00 +0000

Contextual Retrieval#

The problem: chunks lose context when extracted. The fix: prepend a custom context sentence to each chunk before embedding.

The Core Idea#

Anthropic published this technique in September 2024. The insight is deceptively simple:

Before embedding a chunk, prepend a short context sentence that describes where the chunk comes from and what it’s about — generated by a fast LLM using the full document as input.

Mon, 01 Jan 0001 00:00:00 +0000

GraphRAG#

Vector search finds similar passages. GraphRAG finds patterns, relationships, and themes across the entire corpus.

Why Vector Search Fails at Global Questions#

Standard RAG answers “local” questions well — questions answered by one or a few passages.

It struggles with:

“What are the main themes across all documents?”
“How do these companies relate to each other?”
“What is the overall sentiment trend in these reports?”

These require synthesizing information across the entire corpus — not just finding a relevant chunk.

Mon, 01 Jan 0001 00:00:00 +0000

Hybrid Search#

Dense search finds synonyms. Sparse search finds exact terms. Hybrid search wins.

The Problem with Pure Vector Search#

Vector (dense) search is powerful but has blind spots:

Keyword mismatch — “myocardial infarction” ≠ “heart attack” in dense space without enough training data
Rare terms — product codes, model numbers, names get diluted
Exact match needs — legal terms, codes, IDs should match exactly

BM25 (sparse) search has the opposite profile: great for exact terms, poor for semantic similarity.

Mon, 01 Jan 0001 00:00:00 +0000

Late Chunking#

Traditional chunking: chunk → embed. Late chunking: embed → chunk. That order reversal is everything.

The Problem with Early Chunking#

When you chunk a document before embedding, each chunk loses context from the rest of the document.

Example:

Document: "The Eiffel Tower was built in 1889. It is located in Paris.
 This famous landmark attracts millions of visitors annually."

Chunk 1: "The Eiffel Tower was built in 1889."
Chunk 2: "It is located in Paris." ← "It" has no referent!
Chunk 3: "This famous landmark attracts..." ← What landmark?

Chunk 2’s embedding doesn’t know “It” refers to the Eiffel Tower. The embedding is incomplete.

Mon, 01 Jan 0001 00:00:00 +0000

LLM Grounding & Citations#

An answer without a source is just an opinion. Ground your LLM to the documents it retrieved.

What Is Grounding?#

Grounding means forcing the LLM to:

Only use retrieved context — no hallucinated facts
Cite every claim — map each sentence to a source chunk
Admit uncertainty — say “I don’t know” when context is insufficient

System Prompt for Grounded Answers#

The simplest form of grounding is a strict system prompt:

Mon, 01 Jan 0001 00:00:00 +0000

Multimodal Embeddings#

Text-only RAG misses 80% of real-world documents — PDFs with charts, tables, figures. Multimodal embeddings fix that.

The Problem with Text-Only RAG#

Standard RAG pipeline:

Parse PDF → extract text → chunk text → embed text
Lost: tables, diagrams, headers in images, scanned pages

Multimodal embedding treats the document as an image — no text extraction needed.

CLIP — Connecting Text and Images#

CLIP (OpenAI) embeds images and text into the same vector space. You can query with text and retrieve images, or vice versa.

Mon, 01 Jan 0001 00:00:00 +0000

Query Augmentation#

The query the user types is rarely the best query for retrieval. Fix it before it hits your vector database.

Why Augment Queries?#

Users ask vague, short, or jargon-heavy questions. Your retrieval system works best with precise, information-rich queries.

User Query	What They Mean	Better Retrieval Query
“How does it work?”	(no context)	Depends on the document!
“Python error in loop”	IndexError in for loop	“Python IndexError list index out of range for loop”
“Fast RAG”	Low-latency RAG pipelines	“Techniques to reduce latency in retrieval-augmented generation”

Technique 1: HyDE — Hypothetical Document Embeddings#

Idea: Instead of embedding the query, generate a hypothetical answer document, then embed that.

Mon, 01 Jan 0001 00:00:00 +0000

RAGAS Evaluation#

“You can’t improve what you can’t measure.” RAGAS gives you numbers, not vibes.

Why Evaluate?#

RAG systems fail in subtle ways:

Retrieved the right documents but hallucinated the answer
Gave a correct-looking answer that wasn’t supported by context
Retrieved irrelevant chunks despite having good documents in the corpus

RAGAS catches all three failure modes with four key metrics.

Mon, 01 Jan 0001 00:00:00 +0000

Reranking#

Retrieve many, rerank few, pass the best to the LLM.

The Two-Stage Retrieval Pattern#

First-stage retrieval (vector search) is fast but approximate. Rerankers are slow but accurate. The trick: run the fast stage to get 20-50 candidates, then run the accurate stage on just those candidates.

graph LR
 A[Query] --> B[Stage 1: ANN Search]
 B --> C["Top-50 Candidates\n(fast, approximate)"]
 C --> D[Stage 2: Reranker]
 D --> E["Top-5 Results\n(slow, accurate)"]
 E --> F[LLM]

Cost: vector search on 1M docs + reranking on 50 docs » vector search alone on 1M docs.

Mon, 01 Jan 0001 00:00:00 +0000

Semantic Caching#

If two questions mean the same thing, why call the LLM twice?

The Problem#

In production RAG, many user questions are semantically identical:

“What is RAG?” / “Explain RAG” / “How does RAG work?”
“Who is the CEO?” / “Who leads the company?” / “Who runs this org?”

Standard caching (exact string match) misses all these. Semantic caching catches them.

How Semantic Caching Works#

graph LR
 A[New Query] --> B[Embed Query]
 B --> C{Check Cache\nCosine Similarity}
 C -->|Hit: sim > threshold| D[Return Cached Answer]
 C -->|Miss: sim < threshold| E[Run RAG Pipeline]
 E --> F[Store in Cache\nquery embedding + answer]
 F --> G[Return Fresh Answer]

Embed the incoming query
Search the cache for similar past queries
If similarity > threshold (e.g., 0.92) → return cached answer
If miss → run full RAG → cache the result

Build a Semantic Cache from Scratch#

import numpy as np
import json
import time
from dataclasses import dataclass, asdict
from openai import OpenAI

client = OpenAI()

@dataclass
class CacheEntry:
 query: str
 answer: str
 embedding: list[float]
 timestamp: float
 hit_count: int = 0

class SemanticCache:
 """
 In-memory semantic cache with cosine similarity lookup.
 For production: replace the list with a vector database.
 """
 
 def __init__(
 self,
 similarity_threshold: float = 0.92,
 embedding_model: str = "text-embedding-3-small",
 max_size: int = 1000,
 ttl_seconds: float = 3600, # 1 hour TTL
 ):
 self.threshold = similarity_threshold
 self.model = embedding_model
 self.max_size = max_size
 self.ttl = ttl_seconds
 self.cache: list[CacheEntry] = []
 
 def _embed(self, text: str) -> list[float]:
 response = client.embeddings.create(input=[text], model=self.model)
 return response.data[0].embedding
 
 def _cosine_similarity(self, a: list[float], b: list[float]) -> float:
 a, b = np.array(a), np.array(b)
 return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
 
 def _is_expired(self, entry: CacheEntry) -> bool:
 return (time.time() - entry.timestamp) > self.ttl
 
 def get(self, query: str) -> tuple[str | None, float]:
 """
 Look up query in cache.
 Returns (answer, similarity) or (None, 0.0) on miss.
 """
 query_embedding = self._embed(query)
 
 best_similarity = 0.0
 best_entry = None
 
 now = time.time()
 for entry in self.cache:
 # Skip expired entries
 if (now - entry.timestamp) > self.ttl:
 continue
 
 sim = self._cosine_similarity(query_embedding, entry.embedding)
 if sim > best_similarity:
 best_similarity = sim
 best_entry = entry
 
 if best_entry and best_similarity >= self.threshold:
 best_entry.hit_count += 1
 print(f"🎯 Cache HIT (sim={best_similarity:.4f}): '{best_entry.query}'")
 return best_entry.answer, best_similarity
 
 print(f"❌ Cache MISS (best sim={best_similarity:.4f})")
 return None, best_similarity
 
 def set(self, query: str, answer: str) -> None:
 """Store a query-answer pair in the cache."""
 if len(self.cache) >= self.max_size:
 # Evict oldest entry (LRU-style)
 self.cache.pop(0)
 
 embedding = self._embed(query)
 entry = CacheEntry(
 query=query,
 answer=answer,
 embedding=embedding,
 timestamp=time.time(),
 )
 self.cache.append(entry)
 print(f"✅ Cached: '{query[:50]}...'")
 
 def stats(self) -> dict:
 now = time.time()
 active = [e for e in self.cache if (now - e.timestamp) <= self.ttl]
 total_hits = sum(e.hit_count for e in active)
 return {
 "total_entries": len(self.cache),
 "active_entries": len(active),
 "total_cache_hits": total_hits,
 "threshold": self.threshold,
 }
 
 def clear(self) -> None:
 self.cache.clear()

# --- Usage with RAG Pipeline ---

def rag_with_cache(
 query: str,
 cache: SemanticCache,
 rag_fn, # your actual RAG function
) -> dict:
 """
 Wrap any RAG pipeline with semantic caching.
 Returns answer + metadata about cache status.
 """
 start = time.time()
 
 # Check cache first
 cached_answer, similarity = cache.get(query)
 
 if cached_answer:
 return {
 "answer": cached_answer,
 "from_cache": True,
 "similarity": similarity,
 "latency_ms": (time.time() - start) * 1000,
 }
 
 # Cache miss — run full RAG
 answer = rag_fn(query)
 
 # Store result in cache
 cache.set(query, answer)
 
 return {
 "answer": answer,
 "from_cache": False,
 "similarity": similarity,
 "latency_ms": (time.time() - start) * 1000,
 }

# Demo
def dummy_rag(query: str) -> str:
 """Simulates an expensive RAG call (sleep 2 seconds)."""
 time.sleep(2) # Simulate LLM latency
 return f"Answer to: {query}"

cache = SemanticCache(similarity_threshold=0.92)

queries = [
 "What is retrieval-augmented generation?",
 "Explain RAG to me", # Should hit cache
 "How does retrieval augmented generation work?", # Should hit cache
 "What is a vector database?", # Cache miss
 "Tell me about vector databases", # Should hit cache
]

for q in queries:
 result = rag_with_cache(q, cache, dummy_rag)
 status = "HIT" if result["from_cache"] else "MISS"
 print(f"[{status}] {q[:50]} — {result['latency_ms']:.0f}ms")

print("\nCache stats:", cache.stats())

Production: Redis + Vector Cache#

For production, store embeddings in Redis with vector search:

Mon, 01 Jan 0001 00:00:00 +0000

Vector Databases#

Embeddings without a vector database are like data without a database — unusable at scale.

What Is a Vector Database?#

A vector database stores high-dimensional vectors (embeddings) and answers the question: “Which vectors are most similar to this query vector?”

This is the “Retrieve” in Retrieval-Augmented Generation.

graph LR
 A[Query: 'What is RAG?'] --> B[Embed Query]
 B --> C{Vector DB}
 C --> D["[0.12, -0.34, 0.87, ...]"]
 D --> E[ANN Search: Top-K similar chunks]
 E --> F[Retrieved Context]

The Four You’ll Use#

DB	Type	Best For	Scale
FAISS	Library	Research, GPU search	Millions (single machine)
ChromaDB	Embedded DB	Prototyping, local dev	< 10M vectors
Qdrant	Production DB	Production RAG	Billions, distributed
PGVector	Postgres ext.	Existing Postgres stack	Millions

FAISS — Facebook AI Similarity Search#

FAISS is a library, not a server. It runs embedded in your Python process. Fastest for exact or approximate nearest-neighbor search.

Mon, 01 Jan 0001 00:00:00 +0000

Agent Benchmarking and Evaluation#

Benchmarking compares agents on the same repeatable tasks. Evaluation decides whether one is good enough for your real task.

Start with a benchmark when you want to explore a model or agent framework. Then create a small evaluation set from your own work before trusting it with users, tools, or data. An agent is not reliable because its answer sounds convincing; evaluate the completed task, tool calls, artifacts, and failure behaviour.

Mon, 01 Jan 0001 00:00:00 +0000

Agent Fundamentals#

An AI agent is a model that can choose and use tools repeatedly to finish a goal.

Short video#

Agent vs chatbot#

Chatbot	Agent
Usually produces one response	Can take several steps
Mainly returns text	Can search, calculate, edit, or call APIs
User guides each turn	Agent chooses the next step
Low autonomy	Controlled autonomy

The agent loop#

flowchart LR
 G[Goal] --> D[Decide next step]
 D --> T[Use a tool]
 T --> O[Observe result]
 O --> D
 D -->|Goal complete| F[Final answer]

An agent repeats three simple actions:

Mon, 01 Jan 0001 00:00:00 +0000

Agent Memory Systems#

Agent memory is useful information saved now so the agent can use it later.

Saving an entire chat is history. Memory means selecting the small parts that will help a future task.

Short video#

Types of memory#

Type	What it remembers	Example
Working memory	Current task and recent messages	Current order number
Semantic memory	Facts and preferences	User prefers Celsius
Episodic memory	Past events and outcomes	Last deployment failed
Procedural memory	How to perform a task	Release checklist

Simple memory flow#

flowchart LR
 C[Conversation or event] --> S[Select useful fact]
 S --> V[Check permission and truth]
 V --> M[(Store memory)]
 M --> R[Retrieve when relevant]
 R --> A[Use in current task]

Where memory is stored#

Storage	Best use
Conversation state	Current session
SQL/document database	Exact facts, profiles, and updates
Vector database	Finding semantically similar memories
Object storage	Large files, audio, images, and reports

A normal database should usually be the source of truth. Add vector search only when fuzzy recall is useful.

Mon, 01 Jan 0001 00:00:00 +0000

Async and Parallelism in Agent Systems#

Async lets one program work on other tasks while it waits for a network, model, database, or file operation.

Agent systems spend a lot of time waiting, so async can reduce total waiting time.

Short video#

Key terms#

Term	Simple meaning
Synchronous	Finish one task before starting the next.
Asynchronous	Work on another task while waiting.
Concurrency	Several tasks make progress during the same period.
Parallelism	Several tasks run at the exact same time.
Coroutine	Function declared with `async def`.
`await`	Pause this coroutine until an operation finishes.
Event loop	Schedules coroutines that are ready to run.

Sequential vs concurrent#

flowchart LR
 subgraph Sequential
 A1[Call A] --> B1[Call B] --> C1[Call C]
 end
 subgraph Concurrent
 S[Start] --> A2[Call A]
 S --> B2[Call B]
 S --> C2[Call C]
 end

Small example#

import asyncio

async def call_api(name):
 await asyncio.sleep(1) # represents network waiting
 return f"{name} finished"

async def main():
 results = await asyncio.gather(call_api("A"), call_api("B"))
 print(results)

asyncio.run(main())

Both calls wait together, so this takes about one second instead of two.

Mon, 01 Jan 0001 00:00:00 +0000

Loop Engineering#

Loop engineering is the practice of designing how an AI agent repeatedly finds work, acts, checks the result, remembers progress, and stops safely.

It is an emerging term from 2026. The underlying ideas—feedback loops, verification, state, and stop conditions—are established software and agent-design practices.

Short video#

The loop#

flowchart LR
 T[Trigger or new task] --> P[Pick next work item]
 P --> A[Agent acts]
 A --> V[Verify with evidence]
 V -->|Failed but recoverable| F[Give useful feedback]
 F --> A
 V -->|Passed| S[Save progress]
 S --> D{More work?}
 D -->|Yes| P
 D -->|No| H[Human review or stop]

Four engineering layers#

Layer	Main question
Prompt engineering	What instruction should the model receive?
Context engineering	What files, facts, and history should it see?
Harness engineering	Which tools, permissions, and checks support one run?
Loop engineering	What triggers repeated runs, preserves progress, and stops them?

These layers work together. Loop engineering does not replace good prompts or good code.

Mon, 01 Jan 0001 00:00:00 +0000

Model Context Protocol (MCP)#

MCP is a standard way for AI applications to connect to external tools and data sources.

Instead of writing a different integration for every AI application, a developer can build one MCP server that compatible applications can use. MCP standardizes the connection; it does not replace access control, product design, or ordinary API design.

Videos#

Mon, 01 Jan 0001 00:00:00 +0000

Multi-Agent Systems#

A multi-agent system uses several agents with different roles to complete one larger goal.

More agents do not automatically mean better results. Use them only when tasks can be separated or require different tools and permissions.

Short video#

Supervisor pattern#

flowchart TD
 U[User goal] --> S[Supervisor]
 S --> R[Research agent]
 S --> C[Coding agent]
 R --> S
 C --> S
 S --> V[Check and combine]
 V --> U

The supervisor divides the work, gives each worker a clear task, and combines the results.

Mon, 01 Jan 0001 00:00:00 +0000

Sandboxing Agent Code#

A sandbox is an isolated environment that limits what agent-generated code can access and damage.

Sandboxing is a security boundary, not a particular product. Containers, application kernels, microVMs, virtual machines, and managed code-execution services can all be useful. The right choice depends on what the code can do, which data it can reach, and the cost of a breakout or mistake.

Videos#

Mon, 01 Jan 0001 00:00:00 +0000

Specialized Agents#

A specialized agent is designed for one type of work and receives only the tools, data, and permissions needed for that work.

Specialization is more than giving the model a role in a prompt. The tools and checks must also match the role.

Short video#

Basic design#

flowchart LR
 G[Clear goal] --> A[Specialized agent]
 D[Relevant data] --> A
 A --> T[Limited tools]
 T --> R[Result]
 R --> V[Domain-specific check]

Useful specializations#

Agent	Tools	How to check it
Browser agent	Browser automation	Page state or screenshot
Coding agent	Editor, shell, tests	Tests, types, and diff review
Data agent	SQL and notebooks	Constraints and totals
Research agent	Search and document tools	Primary-source citations
Document agent	PDF/Office parsers	Rendered-page review
Operations agent	Logs and runbooks	Health check and rollback state

Browser agents#

Browser agents can read pages, fill forms, and click controls. Prefer page roles and labels such as “Search” or “Submit” over screen coordinates because they are more stable.

Mon, 01 Jan 0001 00:00:00 +0000

Function / Tool Calling#

Tool calling lets a model ask an application to run a function.

The model does not execute the function. It returns the tool name and arguments; the application validates and runs them.

Videos#

Mon, 01 Jan 0001 00:00:00 +0000

Anti-bot Patterns#

Most blocks aren’t Cloudflare-grade. Learn the short ladder of defences sites use — and the honest response to each rung.

⏱ ~8 min read · ~12 min hands-on 🔗 needs: Legal & Ethical Scraping · Hidden JSON APIs

Before you reach for stealth browsers, know the ladder. Nine times out of ten a “block” is something simple — a missing header, too many requests, or a session cookie you didn’t carry. Each rung has a correct response and an arms-race response; prefer the correct one.

Mon, 01 Jan 0001 00:00:00 +0000

Authenticated Scraping#

Carry a login session in code — CSRF tokens, cookies, saved browser state — and recognise when the login is the line you shouldn’t cross.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: Hidden JSON APIs · Playwright & Selenium · Legal & Ethical Scraping

Plenty of data sits behind a login. Sometimes you’re clearly entitled to it — your own account, your own app, or an API that issues you a token. Sometimes the login is precisely the access control you must not defeat.

Mon, 01 Jan 0001 00:00:00 +0000

Change Detection & Dedup#

A scheduled scraper that re-saves the same rows every night is just an expensive clock. Store state, hash content, and record only what actually changed.

⏱ ~8 min read · ~15 min hands-on 🔗 needs: DuckDB + Parquet · Scheduled Scraping

The first run of a scraper is the easy one. The interesting question is the second: what’s new? Answer it with two cheap ideas — a stable ID for every record, and a content hash to detect edits.

Mon, 01 Jan 0001 00:00:00 +0000

Cloudflare Bot Protection#

Understand exactly how Cloudflare decides you’re a bot — then use the legitimate ways through, and know what it costs to fetch a page anyway.

⏱ ~10 min read · ~15 min hands-on 🔗 needs: Anti-bot Patterns · Hidden JSON APIs · Legal & Ethical Scraping

Cloudflare sits in front of a large share of the web. A plain httpx or requests call often gets a 403 or an endless “checking your browser” loop, while Chrome loads the same page instantly. That gap isn’t magic — it’s four specific signals. Once you can name them, you know your options.

Mon, 01 Jan 0001 00:00:00 +0000

Document Parsing#

Most of the world’s data is trapped in PDFs, Word files, and scans. Getting it out cleanly — with tables intact — is its own skill.

⏱ ~8 min read · ~15 min hands-on 🔗 needs: HTML → Markdown · Vision Models for Scraping

A PDF isn’t a document format so much as a set of drawing instructions. There’s often no “table” in there at all — just text positioned at coordinates that look like a table. That’s why naive extraction produces scrambled columns, and why picking the right tool matters.

Mon, 01 Jan 0001 00:00:00 +0000

DuckDB + Parquet (and SQLite for state)#

Two databases, two jobs: SQLite remembers what your scraper has seen; DuckDB answers questions about what it collected — straight off Parquet files, no server.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: SQLite · Hidden JSON APIs

Scraped data has two very different access patterns, and using one tool for both is why people end up with a 4 GB CSV they can’t open.

Mon, 01 Jan 0001 00:00:00 +0000

Google Dorking#

Turn a search box into a precision data-sourcing tool — find the exact files and datasets you need, then automate it into a reproducible pipeline.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: Legal & Ethical Scraping · Hidden JSON APIs

“Dorking” is just using search operators well. The same operators that find a public dataset in seconds also reveal what an organisation has accidentally left indexed — so this is both a sourcing skill and the first move in a footprint audit. Here we focus on sourcing + automation; the defensive exposure-hunting side is Week 7 → Dorking for Recon.

Mon, 01 Jan 0001 00:00:00 +0000

Hidden JSON APIs#

The page you want to scrape has already done the work for you. Find the JSON endpoint its own JavaScript calls, and skip the HTML entirely.

⏱ ~8 min read · ~15 min hands-on 🔗 needs: HTTP clients · browser DevTools (Network tab)

Most modern sites load a nearly-empty HTML shell, then fetch their real data as JSON from a backend API. If you find that request, you get clean, structured data — no brittle CSS selectors, no headless browser, often 100× faster. This is the first thing to try on any dynamic site, before you reach for Playwright.

Mon, 01 Jan 0001 00:00:00 +0000

HTML → Markdown for LLMs#

Raw HTML is 90% navigation, scripts, and cookie banners. Strip it to clean Markdown and you cut your token bill while improving the model’s answers.

⏱ ~8 min read · ~12 min hands-on 🔗 needs: Hidden JSON APIs · Document Parsing

Feeding raw HTML to an LLM wastes tokens on markup the model doesn’t need and buries the actual content in boilerplate. Converting to Markdown first is one of the highest-leverage steps in any scrape-to-LLM pipeline.

Mon, 01 Jan 0001 00:00:00 +0000

Image Processing Pipeline#

Scraped images are rarely usable as-is. Deduplicate, normalise, and crop them before they reach a model or a database.

⏱ ~7 min read · ~12 min hands-on 🔗 needs: Vision Models for Scraping

Scrape a few thousand images and you’ll have duplicates at different resolutions, EXIF-rotated photos that appear sideways, and 8 MB PNGs where a 200 KB JPEG would do. Fix that in a pipeline, once.

Mon, 01 Jan 0001 00:00:00 +0000

Legal & Ethical Scraping#

“It’s public” does not mean “you’re allowed.” Learn to answer “can I scrape this?” before you write a line of code — and how to stay on the right side of the line.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: nothing · read this before every other Week 6 page

ℹ️ This is a practical engineering guide, not legal advice. Laws differ by country and change often. When money, personal data, or a company’s core assets are involved, ask a lawyer.

Mon, 01 Jan 0001 00:00:00 +0000

OSINT — Infrastructure & Public Records#

Build a picture of an organisation’s public footprint — domains, certificates, code, filings — from open sources, and know exactly how far a single source can be trusted.

⏱ ~10 min read · ~15 min hands-on 🔗 needs: Legal & Ethical Scraping · Wayback & Common Crawl

Open-source intelligence (OSINT) is assembling a verified picture from public sources. This page scopes to infrastructure and public records — the material that matters for data science and security analysis. Profiling people is a different discipline with a much heavier duty of care; that lives in Week 7 → Person & Social OSINT.

Mon, 01 Jan 0001 00:00:00 +0000

Pagination & Infinite Scroll#

“Load more” is a button wired to a loop. Find what it increments — a page number or a cursor token — and the signal that says stop, and you can fetch every result without clicking it.

⏱ ~6 min read · ~15 min hands-on 🔗 needs: Hidden JSON APIs · HTTP clients

Any list longer than one screen — search results, a catalog, a feed — is paginated somehow: page numbers, a “Load more” button, or an endless scrollbar. Reach for this the moment you see one; skip it if the first response already returns everything (check for a total/count field first).

Mon, 01 Jan 0001 00:00:00 +0000

Playwright Advanced#

Intercept the network, reuse a saved login, and block the junk — the difference between a browser script that crawls and one that flies.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: Playwright & Selenium · Authenticated Scraping

Once basic automation works, three techniques make it production-grade: request interception, saved authentication state, and tracing. Together they typically cut runtime by 3–5× and eliminate most flakiness.

Try it in 5 minutes — make it 5× faster#

Images, fonts, ads, and analytics are pure overhead when you only want text. Block them at the network layer:

Mon, 01 Jan 0001 00:00:00 +0000

Playwright & Selenium#

When there’s genuinely no API behind the page, drive a real browser — and drive it so it waits for content instead of guessing.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: Hidden JSON APIs · HTTP clients

Browser automation is the heavyweight option: it renders JavaScript, executes the page’s own code, and sees exactly what a user sees. It’s also 10–100× slower than an HTTP request and far more fragile. Use it after you’ve checked for a hidden JSON API, not before.

Mon, 01 Jan 0001 00:00:00 +0000

Rate Limits, Retries & Caching#

Scrape so politely you never get banned, and so efficiently you never fetch the same page twice.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: HTTP clients · Legal & Ethical Scraping

The difference between a scraper that runs for months and one that’s blocked on day one is rarely cleverness — it’s restraint. Three habits do almost all the work: cap your concurrency, back off when told to, and cache everything.

Mon, 01 Jan 0001 00:00:00 +0000

Scheduled Scraping#

Data is only useful if it’s fresh. Put your scraper on a free cron, make it idempotent, and let it build a time-series while you sleep.

⏱ ~8 min read · ~15 min hands-on 🔗 needs: Change Detection & Dedup · GitHub Actions

A one-off scrape is a snapshot. A scheduled scrape is a dataset that gets more valuable every day — and GitHub Actions will run it for free.

Mon, 01 Jan 0001 00:00:00 +0000

Sitemaps, RSS & Structured Data#

Before you write a single selector, check whether the site is already handing out its data — as a URL index, a change feed, or clean JSON embedded in the page.

⏱ ~8 min read · ~15 min hands-on 🔗 needs: Legal & Ethical Scraping · Hidden JSON APIs

Sites publish structured data on purpose — for search engines, feed readers, and social previews. It’s stable, it’s meant to be machine-read, and almost nobody checks for it before writing a fragile HTML parser.

Mon, 01 Jan 0001 00:00:00 +0000

Speech AI#

Audio is a data source you can query. Transcribe it, timestamp it, and it becomes searchable text like anything else you scraped.

⏱ ~8 min read · ~12 min hands-on 🔗 needs: Video Understanding · Local LLMs

Podcasts, lectures, earnings calls, support recordings — enormous amounts of information exist only as speech. Speech-to-text (STT) turns it into text you can search, chunk, and feed to an LLM.

Try it in 5 minutes — transcribe with timestamps#

faster-whisper runs Whisper on a CTranslate2 backend — several times quicker than the reference implementation and comfortable on CPU with a small model:

Mon, 01 Jan 0001 00:00:00 +0000

Video Understanding#

A video is frames plus audio plus time. Split it into those three, and a problem that looked impossible becomes three you already know how to solve.

⏱ ~8 min read · ~12 min hands-on 🔗 needs: Speech AI · Vision Models for Scraping

Never treat video as an opaque blob. Decompose it: audio → transcript; frames → vision models; time → the index that ties them together.

Try it in 5 minutes — ffmpeg is the whole toolkit#

# One frame per second, numbered
ffmpeg -i video.mp4 -vf fps=1 frame_%04d.jpg

# Audio only, for transcription
ffmpeg -i video.mp4 -vn -acodec libmp3lame audio.mp3

# A single frame at 01:23
ffmpeg -ss 00:01:23 -i video.mp4 -frames:v 1 shot.jpg

# Duration and stream info as JSON
ffprobe -v quiet -print_format json -show_format -show_streams video.mp4

✅ Extraction, sampling, and metadata — four commands cover most of what a video pipeline needs.

Mon, 01 Jan 0001 00:00:00 +0000

Vision Models for Scraping#

When the data is a chart, a scanned table, or a UI built to defeat parsing — screenshot it and ask a vision model for JSON.

⏱ ~8 min read · ~15 min hands-on 🔗 needs: Playwright & Selenium · Structured Output

Some data simply isn’t in the DOM: values baked into an image, a canvas-rendered chart, a scanned PDF page, or a deliberately obfuscated layout. A vision-language model (VLM) reads the rendered pixels the way a person would.

Mon, 01 Jan 0001 00:00:00 +0000

Wayback Machine & Common Crawl#

Someone already crawled the web for you. Get the page — and its entire history — without sending the target a single request.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: Legal & Ethical Scraping · Hidden JSON APIs

Two public archives cover a large share of the web: the Internet Archive’s Wayback Machine (snapshots of individual URLs over time) and Common Crawl (petabyte-scale crawls released free for research). Reach for them when a site blocks you, when you need history rather than the current page, or when you need breadth no polite scraper could achieve.

Mon, 01 Jan 0001 00:00:00 +0000

GitHub Actions — Advanced#

Once a workflow works, make it fast, safe, and reusable: cache the slow parts, matrix the repetitive parts, and stop handing every job write access to your repo.

⏱ ~10 min read · ~20 min hands-on 🔗 needs: Git & GitHub · Scheduled Scraping

You’ve used Actions to run a scheduled scraper. Production CI/CD adds three demands: it must be fast (nobody waits 20 minutes), safe (a workflow is code with access to your secrets), and reusable (don’t copy-paste YAML across ten repos).

Mon, 01 Jan 0001 00:00:00 +0000

Advanced Docker#

A 1.2 GB image that rebuilds from scratch on every code change is a build-system bug. Multi-stage builds and correct layer order fix both size and speed.

⏱ ~10 min read · ~20 min hands-on 🔗 needs: Docker & Compose · GitHub Actions Advanced

You can already write a Dockerfile. Production adds three requirements: small (fast pulls, less to attack), cached (rebuild in seconds), and safe (no root, no secrets baked in).

Mon, 01 Jan 0001 00:00:00 +0000

LLM Security — Offensive#

Attack your own LLM app before a stranger does. Prompt injection is not a hypothetical — it’s the default behaviour of a system that can’t tell instructions from data.

⏱ ~10 min read · ~20 min hands-on 🔗 needs: OWASP LLM Top 10 · Prompt Engineering

An LLM receives one flat stream of text. Your careful system prompt and a hostile sentence inside a scraped PDF arrive in the same channel. That’s the root cause of nearly every LLM attack — and why “just tell it to ignore malicious instructions” doesn’t work.

Mon, 01 Jan 0001 00:00:00 +0000

LLM Safety — Defensive#

You cannot make a model immune to prompt injection. You can make a successful injection worthless — by limiting what the system is able to do.

⏱ ~10 min read · ~20 min hands-on 🔗 needs: LLM Security — Offensive · Sandboxing Agent Code

The defensive mindset is assume the model will be compromised. Design so that a model doing the worst possible thing still can’t cause serious harm. Filtering inputs helps at the margin; architecture is what actually saves you.

Mon, 01 Jan 0001 00:00:00 +0000

OWASP Top 10 for LLM Applications#

The industry’s shared checklist of how LLM applications actually get broken — and the one to run your own project against before it ships.

⏱ ~10 min read · ~20 min hands-on 🔗 needs: LLM Security — Offensive · LLM Safety — Defensive

OWASP maintains the reference list of LLM-specific risks. It’s the vocabulary security teams use, so knowing the codes is genuinely useful in a review: “that’s LLM06” lands faster than a paragraph.

Mon, 01 Jan 0001 00:00:00 +0000

VMs & SSH#

Sometimes you just need a machine. Rent one, reach it safely with keys, and keep your work alive after you disconnect.

⏱ ~9 min read · ~20 min hands-on 🔗 needs: Bash Scripting · Deployment Platforms

Serverless covers most workloads, but some jobs want a persistent box: a long scrape, a GPU fine-tune, a database you control. That means a VM — and SSH is how you live on it.

Mon, 01 Jan 0001 00:00:00 +0000

Serverless Functions#

Deploy code without owning a server, pay only while it runs, and scale to zero when nobody’s looking — provided you design for a process that can vanish at any moment.

⏱ ~9 min read · ~20 min hands-on 🔗 needs: Deployment Platforms · Advanced Docker

Serverless is the natural home for the workloads this course produces: a scheduled scraper, a webhook receiver, an inference endpoint used a few hundred times a day. You ship a function or container; the platform handles machines, scaling, and idle cost.

Mon, 01 Jan 0001 00:00:00 +0000

Terraform & Infrastructure as Code#

Clicking through a cloud console is undocumented, unrepeatable, and unreviewable. Declare your infrastructure in files, and git log becomes your change history.

⏱ ~10 min read · ~20 min hands-on 🔗 needs: Serverless Functions · Git & GitHub

Infrastructure as Code (IaC) means your cloud resources are declared in version-controlled files. You describe the desired end state; Terraform computes the diff and applies it. The wins are reproducibility, code review for infrastructure, and one command to tear a whole environment down.

Mon, 01 Jan 0001 00:00:00 +0000

Cost Alerting & Budgets#

The cloud bill nobody checks is the one that ruins a month. Cap what you can, alert on the rest, and never let an agent loop spend money unattended.

⏱ ~8 min read · ~20 min hands-on 🔗 needs: Serverless Functions · Loop Engineering

Students in this course run LLM APIs, autoscaling services, and agent loops — three of the best ways ever invented to spend money by accident. A retry loop against a paid API can burn a semester’s budget overnight.

Mon, 01 Jan 0001 00:00:00 +0000

Pub/Sub & Event-Driven Architecture#

Stop making the caller wait. Drop a message on a queue, return immediately, and let workers do the slow part — with retries you didn’t have to write.

⏱ ~9 min read · ~20 min hands-on 🔗 needs: Serverless Functions · Async & Parallelism

Your scraper takes 90 seconds. Your serverless request times out at 60. The fix isn’t a bigger timeout — it’s splitting the work: accept the job, queue it, respond instantly, and process it elsewhere.

Mon, 01 Jan 0001 00:00:00 +0000

Cloudflare — The Defender’s Side#

You’ve spent Week 6 getting past bot protection. Now put it in front of your own API and watch the traffic from the other side of the glass.

⏱ ~9 min read · ~20 min hands-on 🔗 needs: Cloudflare Bot Protection · Cloudflare Tunnels · FastAPI

Every API you ship will be scraped, credential-stuffed, and scanned. The defences you studied as obstacles in Week 6 are the ones you now have to configure — and the interesting part is that the goal is never “block all bots.” It’s to let the right ones through cheaply while making the wrong ones expensive.

Mon, 01 Jan 0001 00:00:00 +0000

Dorking for Recon & Exposure#

Search engines have already indexed your mistakes. Find them before someone else does — on domains you own.

⏱ ~9 min read · ~20 min hands-on 🔗 needs: Google Dorking · OWASP LLM Top 10

Week 6 taught operators as a sourcing skill. The same operators are the first tool in an attacker’s kit — and therefore the first in a defender’s. This page is about auditing your own external footprint.

Mon, 01 Jan 0001 00:00:00 +0000

The same techniques that profile a person will show you what the internet already knows about you. Learn both halves — and practise only on yourself.

⏱ ~10 min read · ~25 min hands-on 🔗 needs: OSINT — Infrastructure & Records · Legal & Ethical Scraping

Week 6 scoped OSINT to domains, certificates, and corporate records. This page covers the part aimed at people — because you cannot defend against a technique you don’t understand, and because as a data scientist you will be handed personal datasets and asked what’s safe to publish.

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Storage for ML#

A model is not a single file. It is a chain of data, code, weights, metrics, and predictions. Cloud storage gives that chain a home that is not your laptop.

⏱ ~10 min read · ~25 min hands-on 🔗 needs: Config Management · Cost Alerting & Budgets

The mental model#

Google Cloud Storage (GCS) is object storage. A bucket is a globally named container; an object is a file inside it. gs:// is GCS’s equivalent of a filesystem path:

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery ML#

If your data is already in a warehouse, copying it into a notebook just to train a baseline is often the slowest and riskiest part of the job. BigQuery ML lets SQL create the first model where the table already lives.

⏱ ~12 min read · ~30 min hands-on 🔗 needs: SQL basics · Cloud Storage for ML · Cost Alerting & Budgets

Mon, 01 Jan 0001 00:00:00 +0000

MLflow#

“Best model” means nothing if nobody can answer: best compared with which run, trained on which data, with which parameters, and where is the model file? MLflow turns that scavenger hunt into an experiment record.

⏱ ~12 min read · ~25 min hands-on 🔗 needs: Python + uv · Logging & Testing · Cloud Storage for ML

The mental model#

MLflow is an open-source platform for the ML lifecycle. Start with Tracking: a local API and web UI that record what happened in each training attempt.

Mon, 01 Jan 0001 00:00:00 +0000

Fine-Tuning Strategy#

Fine-tuning is not a button that makes an AI “know your business.” It is a costly way to change a model’s behaviour. Before you train, prove that prompting, retrieval, or tools cannot solve the real problem more safely and cheaply.

⏱ ~12 min read · ~30 min hands-on 🔗 needs: Prompt Engineering · RAGAS Evaluation · MLflow

What fine-tuning changes#

A pretrained model has learned broad patterns from its original training. Fine-tuning continues training it on examples for a narrower task. In supervised fine-tuning (SFT), each example demonstrates the input and the desired output.

Mon, 01 Jan 0001 00:00:00 +0000

Hugging Face Ecosystem#

Hugging Face is not “a website to download models.” It is an ecosystem for finding, running, evaluating, adapting, documenting, and sharing ML artefacts. The model page is part of the model. Read it before you run it.

⏱ ~12 min read · ~30 min hands-on 🔗 needs: Python + uv · Fine-Tuning Strategy · MLflow

The map of the ecosystem#

Hub ── stores/version-controls ── models, datasets, Spaces
 │
 ├── Transformers ── load models, tokenize, infer, train
 ├── Datasets ── load, clean, split, stream datasets
 ├── Evaluate ── calculate and share metrics
 ├── PEFT ── train small adapters such as LoRA
 ├── TRL ── supervised / preference / RL training helpers
 ├── Accelerate ── place training across available hardware
 └── Gradio / Spaces ── make a shareable demo

You do not need every library for a first project. A beginner path is:

Mon, 01 Jan 0001 00:00:00 +0000

Fine-Tuning Techniques#

The important question is not “which fine-tuning buzzword should I use?” It is “what minimum change to the model, with what evidence, improves my held-out task?” Start with an adapter. Earn the right to do anything more expensive.

⏱ ~14 min read · ~35 min hands-on 🔗 needs: Fine-Tuning Strategy · Hugging Face Ecosystem · Quantization

The techniques, from least to most invasive#

Technique	What changes	Best first use	Main risk
Prompt / few-shot	no weights	establish a baseline	prompt becomes long/fragile
SFT (supervised fine-tuning)	weights learn input → desired output examples	stable formatting, classification, extraction, task behaviour	copies mistakes in labels
LoRA	trains small low-rank adapter matrices; base model stays frozen	almost every first open-model adaptation	wrong base model/template still fails
QLoRA	LoRA while the frozen base model is loaded in 4-bit	GPU/RAM-constrained first run	hardware/software compatibility and quality trade-off
Preference tuning (DPO etc.)	learns chosen output over rejected output	you can rank alternatives more easily than write a perfect one	unclear/inconsistent preferences
Full fine-tuning	updates all/most base weights	specialised, well-funded work with strong data/evals	cost, catastrophic forgetting, hard rollback
Continued pretraining	learns from raw domain text before task tuning	a large, licensed domain corpus materially differs from base knowledge	costly; raw text is often a bad substitute for RAG

For this course, the practical default is instruction-tuned base model + supervised fine-tuning + LoRA, usually with 4-bit QLoRA. It is cheap enough to learn, produces a small adapter to share, and is reversible: remove the adapter to return to the base model.

Mon, 01 Jan 0001 00:00:00 +0000

Quantization#

Quantization is a compression trade-off: use fewer bits to store model numbers so the model fits on available hardware. It makes a model cheaper to load; it does not make the model smaller in judgement, safer, or better at your task.

⏱ ~12 min read · ~25 min hands-on 🔗 needs: Fine-Tuning Techniques · basic Python

From precision to practical memory#

Model weights are large arrays of numbers. Full precision uses many bits for each number; quantization represents them with fewer bits plus scales/metadata that approximately reconstruct the original values.

Mon, 01 Jan 0001 00:00:00 +0000

Gemma 4 Fine-Tuning#

Your first successful fine-tune should be small, boring, and measurable: a small instruction model, a narrow custom task, a LoRA adapter, and a held-out comparison. A giant model and a spectacular demo are not the same as evidence.

⏱ ~15 min read · ~45 min hands-on 🔗 needs: Fine-Tuning Strategy · Fine-Tuning Techniques · Quantization

Why Gemma 4 + Unsloth is a good first lab#

Gemma 4 is a family of Google open-weight models with instruction-tuned variants. Unsloth provides guided notebooks and Studio workflows for loading compatible models, applying a chat template, performing LoRA/QLoRA fine-tuning, testing, and exporting an adapter.

Mon, 01 Jan 0001 00:00:00 +0000

Model Publishing & Model Cards#

Uploading weights is not publishing a model. Publishing means another person can discover what it is, load the correct artefact, understand what it was tested for, avoid known harms, and decide whether they are allowed to use it. The model card is the interface for that decision.

⏱ ~12 min read · ~30 min hands-on 🔗 needs: Hugging Face Ecosystem · Gemma 4 Fine-Tuning · MLflow

Publish an artefact, not a surprise#

A release answers five questions before someone downloads it:

Mon, 01 Jan 0001 00:00:00 +0000

Wikipedia Data with Python#

You’ll learn how to scrape data from Wikipedia using the wikipedia Python library, covering:

Installing and Importing: Use pip install to get the Wikipedia library and import it with import wikipedia as wk.
Keyword Search: Use the search function to find Wikipedia pages containing a specific keyword, limiting results with the results argument.
Fetching Summaries: Use the summary function to get a concise summary of a Wikipedia page, limiting sentences with the sentences argument.
Retrieving Full Pages: Use the page function to obtain the full content of a Wikipedia page, including sections and references.
Accessing URLs: Retrieve the URL of a Wikipedia page using the url attribute of the page object.
Extracting References: Use the references attribute to get all reference links from a Wikipedia page.
Fetching Images: Access all images on a Wikipedia page via the images attribute, which returns a list of image URLs.
Extracting Tables: Use the pandas.read_html function to extract tables from the HTML content of a Wikipedia page, being mindful of table indices.

Here are links and references:

Tools in Data Science

Actor Network Visualization#

AI Coding in the CLI#

Claude Code CLI#

AI Coding Context Engineering#

AI Coding in IDEs#

Copy-paste workflows#

AI Coding Online#

Artifacts: Inline AI coding sandboxes#

AI Coding Strategies#

AI Coding Tests#

Build evaluation loops#

AI Coding Tools#

AI Coding#

Archived content#

Videos#

IITM BS Degree - Diploma Level Orientation#

Tools in Data Science Orientation#

Tools in Data Science Live Sessions#

Tools in Data Science Playlist#

Optional: Parse & clean PDF files with Tabula#

Base 64 Encoding#

Terminal: Bash#

BBC Weather location ID with Python#

TDS Bridge Bootcamp — Combined Checklist/Syllabus#

Day 1 Post-Session Checklist#

Setup Day#

Day 2 Post-Session Checklist#

Linux & Shell Essentials#

Day 3 Post-Session Checklist#

VS Code + Python Projects with uv#

Day 4 Post-Session Checklist#

HTTP, APIs & Chrome DevTools#

Day 5 Post-Session Checklist#

Git & GitHub Workflow#

Day 1 — File Structure: Windows vs Linux#

The Big Picture#

Day 1 — Installation Guide#

What You Need Installed by the End#

Option A — Native Ubuntu Linux#

Day 1 — Setup Day: Complete Notes#

TDS Bridge Bootcamp#

Table of Contents#

1. Linux vs Windows — The Mental Model#

1.1 Why This Matters#

Day 1 — Path Reading#

What Is a Path?#

Day 1 Quiz & Exercises#

TDS Bridge Bootcamp — Setup Day#

Part A — 30 Multiple Choice Questions#

Day 2 — Basic Script Writing#

What Is a Shell Script?#

Your First Script#

Step 1: Create the file#

Step 2: Write the script#

Step 3: Make it executable#

Step 4: Run it#

The Shebang Line — #!/bin/bash#

Day 2 — Linux & Shell Essentials: The Survival Toolkit#

TDS Bridge Bootcamp#

Table of Contents#

1. Navigating the Filesystem#

1.1 pwd — Where Am I?#

Day 2 Quiz & Exercises#

TDS Bridge Bootcamp — Linux & Shell Essentials#

Part A — 20 Multiple Choice Questions#

Day 2 — Terminal Navigation#

Where Am I? — pwd#

Day 2 — File Operations: touch, mkdir, rm, cp, mv#

Creating Files — touch#

Day 3 Quiz & Exercises#

TDS Bridge Bootcamp — VS Code + Python Projects with uv#

Part A — 20 Multiple Choice Questions#

Day 3 — UV Workflow#

What Is UV?#

Day 3 — Virtual Environments#

The Problem: Why Virtual Environments?#

Day 3 — VS Code Setup#

What Is VS Code?#

Day 4 — API Testing with curl#

The Shebang Line — `#!/bin/bash`#

1.1 `pwd` — Where Am I?#

Where Am I? — `pwd`#

Creating Files — `touch`#

`git init` — Create a New Repository#

What `git init` creates#