Yt-dlp & FFmpeg Telegram Media Archiver Bot
An asynchronous Rust-based Telegram bot engine that orchestrates yt-dlp and FFmpeg subprocesses to securely download, compress, and serve media from X (Twitter) and YouTube.
Tech stack
Performance & metrics
- Lock contention is aggressively minimized by extracting memory snapshots prior to executing disk I/O, ensuring the critical path remains unblocked
- The Tokio event loop remains strictly non-blocking during heavy media transcoding pipelines by utilizing proper asynchronous subprocess spawning
- Memory footprint is optimized by reading process standard outputs asynchronously in chunked buffers
Why I built it
Relying on ad-ridden, heavily rate-limited third-party websites to archive media from X and YouTube disrupts workflow efficiency. I engineered this isolated bot architecture to directly orchestrate CLI tools (yt-dlp, FFmpeg) as asynchronous subprocesses over Telegram, integrating a strict authorization layer to prevent abuse and ensure absolute ownership of the media pipeline.
Architecture & decisions
The core operates on the teloxide framework utilizing the tokio asynchronous runtime. While network I/O is handled asynchronously, the actual video extraction is offloaded to yt-dlp via tokio::process::Command. The subprocess's stdout is piped and parsed in real-time using Regex to project a dynamic progress bar onto the Telegram UI. If the downloaded payload exceeds predefined environment size constraints, ffmpeg is conditionally invoked to aggressively compress the media utilizing libx264 and aac codecs with faststart headers. In-memory state (users and audit logs) is strictly protected via Arc<RwLock>, while physical persistence is guaranteed through an atomic I/O write-and-rename mechanism to completely eliminate JSON file corruption.
Key features
- 01Real-time asynchronous progress reporting achieved by regex-parsing yt-dlp piped stdout buffers
- 02Autonomous FFmpeg H.264/AAC downsampling triggered by environment-based file size thresholds
- 03Native compatibility with Local Telegram API servers via custom endpoint routing to bypass default 50MB upload restrictions
- 04Atomic file writing (rename syscall over .tmp) to enforce zero-corruption data persistence
- 05Thread-safe authorization and metric tracking utilizing RwLock synchronization primitives
Technical review
This project effectively demonstrates the robust integration of the Rust asynchronous ecosystem with native OS subprocesses. Pinging the tokio::process::Command stdout buffer in a non-blocking loop to update a UI progress bar requires precise stream handling. Additionally, to avoid the overhead of a dedicated database, I opted for local JSON storage. To counter the inherent risks of concurrent file I/O corruption, I implemented a strict atomic write pattern leveraging temporary files and OS-level atomic rename operations, guaranteeing absolute data integrity.