QuantitativeFinance-Bench

What makes a good task?

QFBench tasks must require real quant expertise: numerical methods, dirty data, and verifiable outputs. Not trivia — real professional workflows that a senior quant would recognize. See the full guide for design principles and examples.

Task requirements

• Tasks must be verifiable and easy to verify: explicit output contract (what to produce and where to save it), programmatically checkable by code (e.g. np.isclose).
• instruction.md and task.toml must be written entirely by humans. instruction.md must not reference which skills to use — the agent must figure that out itself.
• The reference solution must not be leaked via skills or the Dockerfile; no task-specific hints that give away the answer.
• Oracle must pass 100%: the reference solution must pass all tests. Run harbor run --path ./tasks --task-name <task-id> --agent oracle and confirm every test passes before submitting.
• Compare against Finance-Zero: run the single-shot baseline and report it separately from CLI-agent runs.
• Deterministic: same input → same output; no external APIs at runtime.
• Use real data, not synthetic — real data has missing values, outliers, mixed formats.
• Tasks must represent realistic professional workflows without artificial difficulty. The problem itself should be fundamentally hard, not an ordinary problem made adversarial so that agents score low and one can claim hardness.

Task format

Every task directory must include:

tasks/<task-id>/
+-- task.toml                                 # Metadata & resource limits
+-- instruction.md                            # Agent-facing problem statement
+-- environment/
|   +-- Dockerfile                            # Inherits from quantitativefinance-bench-sandbox
|   +-- data/                                 # Input datasets
|   \-- skills/                               # OPTIONAL — skills available to agent
|       \-- <skill-name>/
|           +-- SKILL.md
|           +-- scripts/                      # optional
|           \-- ...
+-- tests/
|   +-- test.sh                               # Harbor verifier entry-point
|   \-- test_outputs.py                       # Pytest assertions
\-- solution/
    \-- solve.sh                              # Reference (oracle) solution

Workflow

1. Design the task and implement all required files (instruction, metadata, environment, tests, reference solution).
2. Run harbor run --path ./tasks --task-name <task-id> --agent oracle — oracle must pass 100%.
3. Run Finance-Zero baseline and report the result separately from agentic CLI attempts.
4. Run at least two frontier agents from different companies (see the "Frontier (Strongest)" section in the model reference) and record results; include screenshots and a summary table in your PR.
5. Open a PR with your task under tasks/.

FAQ

Resources

• QFBench repo
• Task contribution guide (full)
• Model & agent quick reference
• Harbor framework