Bot & System Automation
Open Source

YÖK Atlas University Data Scraper & Normalizer

View Source Code
PythonSeleniumPandasBeautifulSoup4RegexBSONChromeDriver

An automated web scraping pipeline built to extract, normalize, and serialize dynamic university placement data from official government endpoints into ML-ready datasets.

Tech Stack

PythonSeleniumPandasBeautifulSoup4RegexBSONChromeDriver

System Metrics

▹Offloading DOM traversal from Selenium to `BeautifulSoup` eliminates recursive WebDriver network calls, vastly improving row-iteration throughput
▹Vectorized DataFrame operations via `pandas` execute row deduplication and column mutations natively in C, avoiding slow Python `for` loops during the transformation phase

Why Did I Build This?

"Extracting historical placement data from the official YÖK Atlas platform manually is highly inefficient due to its reliance on dynamic JavaScript pagination and highly polluted string nomenclature (e.g., scholarship types and program durations hardcoded into titles). I engineered this pipeline to autonomously traverse the endpoints, sanitize the raw text via aggressive Regex constraints, and compile the data into structured formats suitable for database seeding or relational analysis."

Architecture & Decisions

The extraction layer utilizes Selenium WebDriver strictly for state mutation and pagination, handling AJAX load states via explicit `WebDriverWait` and `ExpectedConditions` to prevent DOM desync. To completely bypass the massive IPC (Inter-Process Communication) overhead typical of Selenium's recursive element finding, the actual DOM parsing is offloaded to BeautifulSoup via static source snapshots. The transformation layer relies on Python's `re` module for string sanitization, while `pandas` executes vectorized deduplication and multi-format serialization (CSV, Excel, JSON, BSON).

Key Features

  • 01.Hybrid extraction architecture fusing Selenium's JS execution capabilities with BeautifulSoup's ultra-fast in-memory DOM traversal
  • 02.Aggressive Regex pipeline that autonomously strips metadata anomalies (scholarship flags, durations) to yield normalized program nomenclature
  • 03.Automated pagination traversal protected by strict `staleness_of` state waits to prevent race conditions during asynchronous table redraws
  • 04.Simultaneous multi-format dataset generation (JSON, BSON, CSV, Excel) structurally isolated into raw and sanitized output directories