Project Overview
An advanced autonomous Model Context Protocol (MCP) agent framework engineered to govern the entire evaluation, triage, and repair lifecycle of complex AI coding benchmark tasks. Featuring an intelligent 10-phase automated workflow, the platform performs early Invalid-first triage, tracks repository code drift across atomic revisions, executes single Oracle/NOP verification via isolated containers, and conducts two-model semantic reviewer rehearsals to ensure benchmark stability, prescriptiveness, and packaging hygiene prior to platform admission.
Challenges & Pain Points
High Compute Waste on Invalid Tasks
Attempting deep Docker builds, benchmark runs, and model evaluations on tasks with irremediable environmental flaws or invalid issue statements consumed massive GPU/CPU compute without viable return.
Uncontrolled Workspace Code Drift
Iterative fixes by different engineers frequently mutated baseline source trees, altering test contracts unintentionally and invalidating previous verification evidence.
Credential Leakage in Execution Artifacts
Autonomous agent logs, environment variables, and tool invocation traces often contained temporary tokens, sensitive URLs, and API keys that required bulletproof sanitization before submission.
Subjective Peer Review Bottlenecks
Human reviewer feedback often varied in quality expectations and prescriptiveness scores, causing unexpected revision loops and slowing down benchmark throughput.
Our Solution
Invalid-First Triage Engine
Engineered a pre-execution heuristic pipeline analyzing source scope feasibility, difficulty history, and static contracts, eliminating up to 80% of unnecessary container runs on fundamentally unfixable tasks.
Monotonic Revision Ledger & Drift Tracking
Maintained an immutable revision ledger tracking file manifests, added/modified/deleted deltas, and SHA-256 hashes, invalidating stale evidence automatically upon any source modification.
10-Source Secret Sanitization Protocol
Implemented automated cryptographic scrubbers filtering bearer tokens, cloud API keys, and private URLs across all logs, tool call traces, and evidence bundles prior to packaging.
Two-Model Semantic Reviewer Rehearsal Panel
Constructed an isolated two-model rehearsal sandbox predicting quality, prescriptiveness, and core architectural pillars, achieving consistent dashboard submissions on the first pass.
Technology Stack
LLM Evaluation & Agentic Systems Expertise
Agentic Code Repair & Triage
Pioneering systems for automated AST-based patch generation, test minimisation, regression prevention, and safe repair application within ephemeral execution sandboxes.
Automated AI Security & Compliance
Deep implementation of secret redaction, safe human-in-the-loop gates, tamper-evident evidence manifests, and double-build archive integrity checks.
Semantic Evaluation Governance
Expertise in synthetic evaluation benchmarking, peer-review simulation, rubric standardization, and deterministic metric extraction for LLM leaderboards.
Project Highlights
- 10-phase automated evaluation lifecycle from task intake to atomic bundle
- Invalid-first triage reducing compute waste by up to 80%
- Two-model semantic rehearsal panel ensuring first-pass acceptance
- Automated credential redaction across 10 distinct log sources
- Immutable revision tracking with SHA-256 evidence binding