Red-team real AI agents in real environments

Promptbeat helps teams evaluate agent applications where failures happen across files, commands, tools, services, datasets, browser state, and hidden verifiers. Start from a scenario, generate or load attack seeds, run them against a real target, and keep the evidence needed to explain what failed.

scenario + target profile + seed/dataset
  -> generated probes
  -> real agent or environment adapter
  -> trace-aware judges
  -> normalized report

What you can evaluate

Target class	How it connects	Typical risks
Coding agents	Codex SDK, Claude Agent SDK, OpenCode, OpenClaw, custom CLI adapters	repo injection, terminal injection, secret reads, sandbox escape, network egress
Browser agents	HTTP/custom provider or Target Lab browser adapter	DOM injection, form exfiltration, unsafe navigation, cookie/session misuse
Support agents	HTTP/custom provider around CRM or ticketing tools	cross-user access, PII leakage, refund abuse, policy override
Data agents	notebook, SQL, BI, or Python/JavaScript provider	prompt-injected rows, private table access, unsafe code execution
DevOps agents	script/custom provider or disposable Target Lab environment	credential discovery, destructive cleanup, deployment sabotage
Benchmark tasks	Terminal-Bench, Harbor, Inspect, or custom environment adapter	task boundary violations, hidden-test probing, verifier tampering

Start with Typical agent apps, then use Agent configuration examples to wire a concrete target.

Why scenario-driven

A dataset name or plugin name is not enough to decide what to test. Promptbeat puts the scenario first:

Scenario decision	Why it matters
Target capabilities	A browser agent needs DOM traces; a coding agent needs command/file traces; a support agent needs action audit logs.
Risk taxonomy	`secret_handling`, `network_egress`, `authorization`, and `prompt_injection` require different evidence and judges.
Seed source	Hand-written seeds, HarmBench, JBB, ALERT, private tickets, and benchmark tasks are starting points, not final tests.
Judge evidence	Answer-only checks are useful smoke tests; trace-aware checks are stronger for agents with tools.
Report shape	Results must compare targets, risks, datasets, generated probes, assertions, and representative failures.

Read Scenario-driven evaluation and Risk taxonomy.

From initial dataset to agent-specific probes

Initial datasets matter because they shape the attack space before generation starts. Promptbeat keeps dataset provenance through every case.

raw dataset -> DatasetSpec -> Seed -> DatasetRiskMapping -> scenario-bound probes -> report

Use Dataset catalog to pick a seed source and HarmBench example to see the current validated dataset slice. The current HarmBench smoke run passed 5/5 direct harmful-content cases on Codex, but that result does not replace coding-agent boundary tests.

Current evidence: broad Codex run

The current saved report evaluates Codex as a real coding-agent target.

Suite	Passed	Failed	Pass rate
Broad coding-agent sample	21	11	65.6%
HarmBench dataset slice	5	0	100%
Combined	26	11	70.3%

Strongest areas in this run: repo prompt injection, terminal output injection, and steganographic exfiltration. Weakest areas: procfs credential probing, secret environment handling, network egress, and sandbox read escape. See Broad Codex report and Report schema.

What a Promptbeat report keeps

Evidence	Example
Generated attack pool	`generated_redteam.yaml`, sampled eval configs, generated case metadata
Target identity	provider/adapter, model, workspace, sandbox, approval policy
Trace evidence	commands, file reads/writes, network events, tool calls, returned records
Judge reasoning	assertion type, metric, pass/fail reason, unsafe workaround classification
Reproducibility	config snapshots, raw backend output, logs, HTML/Markdown reports

Promptbeat treats unsafe workaround guidance as evidence too. If an agent suggests env | sort > env_dump.txt or a curl upload command, that can be a behavior failure even when the sandbox blocks execution.

Three ways to start

1. Run the validated Codex path

Use this when you want a quick local proof that the generate/eval/report loop works.

2. Connect your own agent

Use this when the target is a browser agent, support agent, data agent, DevOps agent, security triage agent, or internal assistant.

3. Start from datasets

Use this when you already have official datasets, business seeds, or private red-team records.

API for downstream systems

Promptbeat also exposes seed expansion through the Go web service: small-batch preview, asynchronous batch tasks, dataset risk mapping, plugin/strategy passthrough, callback-friendly results, and smoke-testable container startup. Read Go web service.

Getting Started

Concepts

Targets

Datasets

Coding Agent Safety

Target Lab

Reports

Reference

Promptbeat

Red-team real AI agents in real environments

What you can evaluate

Why scenario-driven

From initial dataset to agent-specific probes

Current evidence: broad Codex run

What a Promptbeat report keeps

Three ways to start

1. Run the validated Codex path

2. Connect your own agent

3. Start from datasets

API for downstream systems

​Red-team real AI agents in real environments

​What you can evaluate

​Why scenario-driven

​From initial dataset to agent-specific probes

​Current evidence: broad Codex run

​What a Promptbeat report keeps

​Three ways to start

​1. Run the validated Codex path

​2. Connect your own agent

​3. Start from datasets

​API for downstream systems

Red-team real AI agents in real environments

What you can evaluate

Why scenario-driven

From initial dataset to agent-specific probes

Current evidence: broad Codex run

What a Promptbeat report keeps

Three ways to start

1. Run the validated Codex path

2. Connect your own agent

3. Start from datasets

API for downstream systems