2026
COBOLEval
A substantial rebuild of bloop's COBOL code-generation benchmark, adding JCL static analysis, an agentic orchestrator, and local inference.
Measuring generated COBOL
146
Benchmark tasks
3.8×
Rust vs. upstream
Detail
- Built on bloop.ai's transpilation of OpenAI's HumanEval into COBOL — the benchmark data and original transpiler are theirs, and the dataset is unchanged.
- Added a JCL parser, validator, and linter for mainframe job control — static analysis only; it does not execute or submit jobs.
- Added a Rig-based agentic evaluation orchestrator in Rust, with per-task JSONL conversation logging.
- Added Ollama support so evaluation runs against local models without an API round-trip.
Stack
Underlined entries link to the writing on that topic.