Cisco Foundation AI introduces Agentic FAITH to test AI security agents on vulnerability localization in real repositories

Cisco Foundation AI has extended FAITH, its open-source evaluation framework for cybersecurity language models, with support for agentic benchmarks in which a model works through a multi-step task using tools instead of answering a single prompt. The extension, presented by Cisco researchers Kimia Majd and Zhuoran Yang in a post published on October 23, 2026, lets FAITH run an agent loop, route its terminal commands to an isolated execution environment, record every interaction and score both the final answer and the way the agent reached it.

The first agentic benchmark integrated into the framework is the group's vulnerability localization benchmark, which Cisco's post calls VulnLoc and which the team first published in September as VLoc Bench. It asks an agent to explore a real software repository with read-only terminal tools and identify the files affected by a class of weakness, and then to recognize, on the patched version of the same repository, that the weakness is no longer there. The benchmark draws on 500 vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories, and its published results show how far current models remain from reliable repository-scale security analysis: the best system evaluated reached a File F1 of 0.229, and on 38.4% of tasks no evaluated model found a single correct file.

FAITH itself is distributed under the Apache 2.0 license in the cisco-foundation-ai GitHub organization, with documentation at faith-evals.com. Until now its public benchmark catalogue has been made up of question-answering style evaluations for threat intelligence, vulnerability classification and security knowledge. The agentic extension changes the unit of evaluation from a response to a trajectory, which is the format in which security agents are increasingly deployed.

··········

FROM SINGLE ANSWERS TO MULTI-STEP SECURITY TASKS.

The existing FAITH workflow is organized around three stages. The query stage runs a benchmark against a model and collects the records, the eval stage grades the outcomes, and the summarize stage reports aggregate results; a run-all command chains the three. The repository README lists sixteen benchmarks for that workflow, including the CTIBench family (ATE, MCQA, RCM, TAA and VSP), CTI reasoning, CWE prediction, four CyberMetric sizes from 80 to 10,000 questions, MMLU and its security subset, SecBench and SecEval.

Those benchmarks measure what a model knows or can infer from a self-contained prompt. Cisco's argument for the agentic extension is that real security work does not look like that. In the words of the post, "An agent investigating a codebase makes a sequence of decisions," and each of those decisions, from which directory to list to which file to open next, affects whether the investigation succeeds. An evaluation that only checks the final submission cannot distinguish an agent that reasoned its way to the right file from one that guessed, nor can it explain why an agent failed.

Agentic FAITH therefore keeps the same query, eval and summarize structure but adds a managed interaction loop. The model chooses an action, FAITH executes it and returns the tool output, and the model decides what to do next. A run ends when the agent submits an answer or when it reaches a stop condition defined by the benchmark, such as a turn limit or a tool-call budget.

··········

HOW THE HARNESS, SANDBOX AND RUN RECORDS WORK.

When a model issues a terminal tool call, FAITH forwards it to the configured execution environment, passes the result back to the model and records the exchange. For the vulnerability localization benchmark that environment can be a local Docker container, which Cisco describes as suitable "for trusted inputs," or a Kubernetes deployment through Agent Sandbox. In both cases, according to the post, "The execution environment enforces read-only access to the repository," so the agent can inspect code but cannot modify the snapshot it is being evaluated on.

The framework is designed so that benchmark authors control the parts that differ from task to task. A benchmark definition specifies how tasks are loaded, which tools are available, where those tools run, and how submissions are extracted and scored, and authors can either reuse FAITH's existing components or register new ones. Prompts, turn limits, tool-call budgets and timeouts are set in the benchmark configuration, while a separate deployment configuration can override the default execution environment without touching the task definition. That separation is what allows the same benchmark to run on a laptop with Docker and on a Kubernetes cluster without forking the evaluation.

Two operational features address the cost and fragility of long agent runs. FAITH saves each completed task before starting the next, so an interrupted run can reuse eligible completed results as long as their storage is still available. It also keeps a detailed record of every task: the conversation history, each tool call and its response, the submitted answer, any execution errors and the reason the task ended. Cisco presents these records as the means to tell whether an agent actually inspected the vulnerable code or simply ran out of budget before getting there.

........

Component	What Agentic FAITH provides
Interaction loop	Model action, tool execution, tool output, next decision
Termination	Submitted answer or benchmark-defined stop condition
Execution	Local Docker or Kubernetes via Agent Sandbox
Repository access	Read-only, enforced by the environment
Configuration	Prompts, turn limits, tool-call budgets, timeouts
Recovery	Each completed task saved before the next starts
Run records	History, tool calls, answers, errors, stop reason

........

··········

WHAT THE VULNERABILITY LOCALIZATION BENCHMARK MEASURES.

The benchmark was built from GitHub Security Advisories disclosed between 2016 and 2026. Each of its 500 tasks pairs two snapshots of a repository: the vulnerable version before the fix and the patched version after it. Ground-truth files are derived from the patch diff, excluding tests, documentation and configuration. The model receives only a generic MITRE CWE description of the weakness class, with no advisory text, CVE identifier, fixing commit or file hint, so it has to locate the problem from the code alone.

In Phase A, run on the vulnerable snapshot, the agent must name the affected files, and performance is measured with File-set F1, a metric that penalizes both missed files and incorrect selections. In Phase B, run on the patched snapshot, the correct answer is that no matching vulnerability is present, and the primary metric is the true negative rate. FAITH records completion status separately, so that, as the post puts it, "a correct report of no matching vulnerability can be distinguished" from a missing submission or an execution failure.

In the September release, the agent's tools were limited to a read-only terminal offering file listing, text search, file viewing and metadata queries, running in a fresh container with networking disabled. Each task allowed 15 commands and 20 turns, and reported scores were averaged over three runs.

........

Benchmark property	Value
Tasks	500
Repositories	290
CWE categories	147
Ecosystems	npm, pip, Maven, Go, Rust, Composer
Advisory window	2016 to 2026
Budget per task	15 commands, 20 turns
Phase A metric	File-set F1 on vulnerable snapshot
Phase B metric	True negative rate on patched snapshot

........

The task distribution follows the advisory database rather than a balanced sample. Go projects dominate the set, while Composer is represented by a single task, which limits how far ecosystem-level conclusions can be drawn.

··········

PUBLISHED RESULTS SHOW A LOW CEILING AND A TRADE-OFF BETWEEN FINDING AND RESTRAINT.

Cisco's September evaluation covered 31 systems, 27 language models and four static analyzers, under the same prompt, read-only tools and command budget. The highest Phase A score, 0.229 File F1, came from GPT-5.5 at its xhigh reasoning setting. Antares-3B, a small task-specific model trained by the Cisco team, followed at 0.223. The best static analyzer, Semgrep, scored 0.086, which according to the authors was still higher than most of the open-weight models tested.

The results also suggest that model size is a weak predictor of performance on this task. The authors report that Gemma-4-E2B, a 2B-parameter model, outscored Llama-3.3-70B, and they found that repository structure predicted per-task difficulty about 4.5 times more strongly than model identity. Larger repositories and Go projects were harder, with performance dropping by a factor of 13 between the smallest and the largest repositories. Most failures were either exhausted budgets (32.3%) or wrong files (27.4%), and the mean number of commands used per task correlated with File F1 at r = 0.72, which indicates that agents which explored more tended to localize better within the fixed budget.

........

Result (VLoc Bench, September 2026)	Reported value
Best Phase A score	0.229 File F1 (GPT-5.5 xhigh)
Antares-3B	0.223 File F1
Best static analyzer	0.086 File F1 (Semgrep)
Tasks unsolved by every model	38.4%
Failures from exhausted budget	32.3%
Failures from wrong files	27.4%
Commands vs File F1 correlation	r = 0.72
Phase A sweep cost	About $0.60 to $141

........

Phase B exposes the opposite failure mode. GPT-5.5 reported vulnerable files on about 72% of patched repositories, where the correct answer was none, and Antares-1B did so on about 99%. Static analyzers cleared between 91.2% and 99.6% of patched repositories. At the extreme, Granite-4.0-1B achieved a perfect true negative rate of 1.000 with a File F1 of 0.000, because it never submitted a file. The pairing of the two phases is meant to prevent exactly this kind of one-sided result from looking like competence: a system has to find real vulnerabilities and also refrain from flagging code that has already been fixed.

Cost varies widely. A full Phase A sweep ranged from about $0.60 for Antares-350M running locally to $141 for GPT-5.5 at xhigh through the API, a spread that matters for teams deciding whether to run the benchmark regularly.

··········

RUNNING AN AGENTIC BENCHMARK IN FAITH.

FAITH's quickstart documentation describes a minimal agentic run on the benchmark identifier `vulnerability-localization-a`. It requires the `openai`, `docker` and `metrics` extras, a running Docker installation that passes the framework's preflight checks, and an API key supplied through the runner environment. The model runs remotely while the agent's commands execute in a local container. Agentic runs currently require the `openai` engine, the `chat_comp` generation mode and the `chat` prompt format, which means any model under test must support structured tool calls through an OpenAI-compatible Chat Completions interface; hosted endpoints and locally served models behind a compatible API are both supported.

........

Quickstart parameter	Documented value
Benchmark	vulnerability-localization-a
Model engine	openai
Generation mode	chat_comp
Prompt format	chat
Trials	1
Sample size	1
Datastore location	out/agentic-smoke
Summary statistics	mean_file_f1, error_rate, stopped_reason

........

The query stage writes logs but no metrics. Grading is done with `faith eval` on the experiment path, and `faith summarize` then reports the mean File F1, the error rate and a breakdown of stop reasons. The documentation warns that a single task can trigger multiple billed model requests, recommends keeping sample sizes small for a first run, and notes that a low F1 score on its own does not indicate a setup failure. It advises checking the `benchmark-log.json` file for tool output and errors, since a command that exits successfully is not proof that the agent's environment behaved as intended. Both phases of the vulnerability localization benchmark require a read-only workspace, which the supplied container image provides and which custom deployments must replicate.

··········

LIMITATIONS DISCLOSED BY THE AUTHORS.

The benchmark's authors list several constraints that affect how its numbers should be read. Runs that time out, error out or end in a refusal are scored as correct clean verdicts in Phase B, so the true negative rate of hosted models is entangled with provider safety layers in a way the team has not yet separated. In 52 of 499 patched pairs, the patch itself contains an advisory identifier or link, which means a clean verdict could be read from the repository rather than inferred from the code. Labels derived from patch diffs are reproducible but not perfect, and cost limited coverage: some models, including Claude Opus 4.8, were not run in a standardized sweep because of expense.

The October post adds a framework, not a new set of scores. It does not report results for any model under Agentic FAITH, does not state how many vulnerability localization tasks are exposed through the framework's interface, and does not include licensing terms for the benchmark data. The FAITH repository README, at the time of writing, still described only the question-answering benchmarks and had no PyPI distribution, so the agentic documentation lives on the separate documentation site.

··········

TECHNICAL ASSESSMENT.

What is established is the structure of the extension. FAITH now supports multi-step, tool-using evaluations with a managed loop, a read-only execution environment on Docker or Kubernetes, per-task checkpointing and complete run records, all inside an Apache 2.0 framework that already covered sixteen security knowledge benchmarks. The first agentic workload, the vulnerability localization benchmark, is documented in detail, from its 500 paired tasks to its 15-command budget, and its September results set a modest reference point: no system exceeded 0.229 File F1, and the systems that found the most vulnerable files were also prone to flagging code that had already been patched.

What remains open is how these numbers will move once the benchmark runs routinely inside a shared harness and other groups contribute their own agentic tasks. The scoring treatment of refusals and timeouts, the advisory identifiers left in some patches and the heavy weighting toward Go are acknowledged sources of distortion, and Cisco has not yet published Agentic FAITH results that could be compared with the September leaderboard. The value of the extension for the wider field will depend less on a single headline score than on whether the trajectory records it captures become the standard way to diagnose why security agents miss the code they were asked to find.

··········

FOLLOW US FOR MORE.

·····

DATA STUDIOS

·····

datastudios.org