Coding agent harnesses · Security mechanisms
HarnessSecurity-Bench
An empirical study and benchmark of security mechanisms in coding agent harnesses.
Read the paper- 6
- Harnesses
- 9
- Security mechanisms
- 23
- Coding tasks
- 2,500
- Trials
- 2,218.20M
- Consumed tokens
Key findings
Attack success.
Task utility.
The security and utility effects differ across mechanisms, harnesses, and tasks.
Network isolation and read-only mode reduce attack effects with substantial utility losses. Command allowlisting and command denylisting reduce attack effects with a small utility loss and a utility gain, respectively.
01 / Empirical study
Security mechanisms in
coding agent harnesses.
We study 40 coding agent harnesses: 27 open-source and 13 closed-source products. We assess all 400 harness–mechanism cells, recording implementation status, default settings, and supported configurations.
The assessment confirmed implementation in 205 cells and absence in 83, while 112 remain unresolved.
02 / HarnessSecurity-Bench
23 coding tasks.
Five attack surfaces.
HSB compares enabled and disabled settings of native security mechanisms across six harnesses. Separate deterministic oracles measure task utility and attack effects.
Misleading context and malicious resources augment task environments while preserving legitimate task requirements.
Data availability
Harness Dataset
HSB tasks and oracles are available in the GitHub repository. The harness dataset, rating records, trial artifacts, and analysis scripts are being prepared for public release.
GitHub repository