Jikan API Scraper: Fault-Tolerant ML Dataset Generator

PythonPandasRequestsREST APIOS I/O

An autonomous, fault-tolerant data harvesting engine designed to extract MyAnimeList datasets for machine learning models, featuring exponential backoff and low-level disk I/O flushing.

Tech Stack

PythonPandasRequestsREST APIOS I/O

System Metrics

▹Elimination of memory ballooning by employing a chunked append strategy, bypassing the need to load the entire dataset into RAM at once

▹Stable memory footprint maintained by ensuring temporal HTTP response objects are immediately released for garbage collection post-processing

▹Dynamic calculation of thread sleep intervals based strictly on the API's current throttling state to optimize execution time

Why Did I Build This?

"Machine learning pipelines require massive, highly structured datasets, but public APIs like Jikan enforce strict rate limits and suffer from unpredictable downtime. I engineered this robust scraping infrastructure to autonomously harvest raw data for future model training, ensuring absolutely zero data loss during multi-hour execution cycles."

Architecture & Decisions

The script operates entirely on Python, utilizing `requests` for network I/O and `pandas` for DataFrame manipulation. The core execution loop implements a resilient retry mechanism equipped with exponential backoff and randomized jitter to seamlessly absorb HTTP 429 (Too Many Requests) and 5xx server errors. To guarantee data integrity during potential OS crashes or exhaustive long-running tasks, file descriptors are forcibly flushed directly to disk using the low-level `os.fsync()` system call after every pagination cycle. Standard output (stdout) is dynamically intercepted and routed into a multi-channel structured logging architecture.

Key Features

01.Aggressive HTTP 429 rate-limit mitigation via an exponential backoff algorithm integrated with randomized jitter
02.Absolute persistence of incremental DataFrame writes guaranteed by OS-level file syncing (`os.fsync`)
03.Dual-depth CSV exporting strategy (core metrics vs. complex entity relationships) pre-optimized for ML feature engineering
04.Autonomous error-handling and dead-letter logging mechanisms to isolate network disconnects and malformed JSON payloads

This project is not just a standard web scraper; it is a resilient data ingestion pipeline. By implementing OS-level file synchronization (`fsync`) and a custom exception hierarchy, the engine survives API timeouts, connection resets, and rigid throttling limits entirely without manual intervention. The dual-table export strategy—separating core metrics from heavy relational nodes—is a calculated data modeling decision that optimizes the payload for immediate ingestion into Pandas/Polars training environments with minimal preprocessing overhead.