Capstone — Job Posting Scraper & Tracker#

Build a scraper that runs itself, notices what changed, and answers questions about the market.

⏱ ~6–8 hours 🔗 needs: Hidden JSON APIs · Change Detection & Dedup · DuckDB + Parquet · Scheduled Scraping

Anyone can scrape a page once. This capstone is about the harder, more valuable thing: a pipeline that runs unattended for weeks, doesn’t duplicate data, doesn’t get banned, and produces a dataset worth querying.

⚖️ Choose your targets before you write code. Read each site’s robots.txt and Terms. Prefer sites with a public API or feed, and job boards that permit it. Job posts contain personal data (recruiter names, emails) — collect only the fields your questions need, and never publish personal contact details. Your submission must include the decision record for each source. A source you can’t justify is a source you don’t scrape.

What you’re building#

flowchart LR
    S1["Source A"] --> A["Acquire<br/>(hidden API > HTML)"]
    S2["Source B"] --> A
    A --> N["Normalise<br/>+ validate schema"]
    N --> ST["SQLite state<br/>(IDs, hashes, last_seen)"]
    ST -->|"new / changed only"| P["Parquet<br/>(dated partitions)"]
    P --> Q["DuckDB queries"]
    Q --> D["Dashboard"]
    CR["GitHub Actions cron"] -.->|daily| A

Requirements#

1. Acquire — two different sources. At least two job sources, each justified in your decision record.

• Try hardest for a hidden JSON API; most job boards have one. Document how you found it (or why there isn’t one).
• Fall back to Playwright only where necessary, and say why.
• Handle pagination with an explicit stop condition and a MAX_PAGES cap.
• Be polite: rate limits, backoff, and caching.

2. Normalise — a validated schema. Every record conforms to a Pydantic model; rows that fail are quarantined with the reason, not silently dropped.

class JobPosting(BaseModel):
    job_id: str            # stable ID, derived from the canonical URL
    title: str
    company: str
    location: str | None
    salary_min: int | None  # parse "₹12–18 LPA" into numbers where you can
    salary_max: int | None
    posted_date: date | None
    url: str
    source: str
    content_hash: str      # over the meaningful fields only

3. Track state incrementally. SQLite holds job_id, content_hash, first_seen, last_seen. A second run on unchanged data must report 0 new, 0 changed. Detect removed postings via last_seen — a job disappearing is a signal (filled or expired).

4. Store as dated Parquet. data/date=YYYY-MM-DD/postings.parquet, so DuckDB can glob the history.

5. Schedule it. A GitHub Actions cron running daily that commits only when data changed, and fails loudly if it collects suspiciously few rows.

6. Answer questions. A DuckDB-backed dashboard (Streamlit, or static HTML + a generated JSON) showing at least:

• Postings per day, and the trend over your collection window
• Top companies and locations hiring
• Salary distribution where parseable
• Churn: how many postings appeared and disappeared this week

Deliverables#

Grading#

Stretch goals#

• Fuzzy-match the same role posted across both sources (entity resolution).
• Extract required skills from descriptions with an LLM into a validated schema — then measure how often it’s wrong.
• Alert (email/Discord) when a posting matching your criteria appears.
• Backfill history from the Wayback Machine to extend your window past your start date.

Checklist before you submit#

• Running twice in a row reports 0 new, 0 changed.
• A deliberately broken selector makes the run fail, not write empty data.
• No API keys or auth.json in the repo.
• No personal contact details in the dataset.
• Every source has a written decision record.
• A stranger can clone the repo and run it from the README alone.