YÖK Atlas University Data Scraper & Normalizer
An automated web scraping pipeline built to extract, normalize, and serialize dynamic university placement data from official government endpoints into ML-ready datasets.
Tech stack
- Languages
Performance & metrics
- Offloading DOM traversal from Selenium to
BeautifulSoupeliminates recursive WebDriver network calls, vastly improving row-iteration throughput - Vectorized DataFrame operations via
pandasexecute row deduplication and column mutations natively in C, avoiding slow Pythonforloops during the transformation phase
Why I built it
Extracting historical placement data from the official YÖK Atlas platform manually is highly inefficient due to its reliance on dynamic JavaScript pagination and highly polluted string nomenclature (e.g., scholarship types and program durations hardcoded into titles). I engineered this pipeline to autonomously traverse the endpoints, sanitize the raw text via aggressive Regex constraints, and compile the data into structured formats suitable for database seeding or relational analysis.
Architecture & decisions
The extraction layer utilizes Selenium WebDriver strictly for state mutation and pagination, handling AJAX load states via explicit WebDriverWait and ExpectedConditions to prevent DOM desync. To completely bypass the massive IPC (Inter-Process Communication) overhead typical of Selenium's recursive element finding, the actual DOM parsing is offloaded to BeautifulSoup via static source snapshots. The transformation layer relies on Python's re module for string sanitization, while pandas executes vectorized deduplication and multi-format serialization (CSV, Excel, JSON, BSON).
Key features
- 01Hybrid extraction architecture fusing Selenium's JS execution capabilities with BeautifulSoup's ultra-fast in-memory DOM traversal
- 02Aggressive Regex pipeline that autonomously strips metadata anomalies (scholarship flags, durations) to yield normalized program nomenclature
- 03Automated pagination traversal protected by strict
staleness_ofstate waits to prevent race conditions during asynchronous table redraws - 04Simultaneous multi-format dataset generation (JSON, BSON, CSV, Excel) structurally isolated into raw and sanitized output directories