2026 – present · private
Proximal
A continuously running pipeline that streams, enriches, and exports GitHub pull request data as structured training sets — with a Rust-specific subset feeding the code-generation work.
Training data for verified code generation
Detail
- Ingests from the public GitHub Archive rather than the API, so it runs continuously at no API cost.
- Enriches merged pull requests with diffs, reviews, and commits, then exports JSONL datasets.
- Language filters are refreshed on a schedule to keep the Rust subset current.
- Deduplicated on GitHub event IDs; merged pull requests only, as the higher-signal population.
Stack
TypeScriptPostgreSQLDockerGH ArchiveJSONL
This work is in a private repository. Happy to talk through the architecture.