Finance Agent Benchmark: Test Evidence, Calculations and Data Limits

Define the Task and the Reference Answer

Choose tasks that reflect the work you need: extract a reported financial value, calculate a ratio, distinguish reporting dates or identify missing coverage. Record the expected source, period, currency, units and acceptable tolerance before running the agent. A fixed document snapshot is useful for reproducibility; a separate time-sensitive task can test whether the agent recognizes freshness requirements.

• Statement extraction: identify revenue for a stated fiscal quarter and cite the filing.
• Calculation: compute a margin from compatible inputs and show the formula.
• Data quality: identify a missing value without inventing an estimate.
• Timing: distinguish an observation date from publication and retrieval dates.

Keep the Comparison Configuration Visible

Record each product version or model, subscription, enabled tools, uploaded files, permissions and prompt. A system with a licensed data feed is a different configuration from the same model without that feed. Do not attribute every difference to model quality. Use the same frozen reference material for extraction tests and a documented observation window for market-data tests.

• Store prompts, full responses, tool outputs and source snapshots where permitted.
• State whether retries or manual corrections are allowed and count them.
• Record timeouts and unavailable sources rather than dropping failed runs.
• Use separate development examples and held-out evaluation tasks.

Score Different Failure Types Separately

For each task, record whether the answer selected the correct source and period, extracted the correct value, performed the arithmetic correctly and disclosed uncertainty. An unavailable-data response can be appropriate when coverage is absent; an invented number should not receive credit for apparent completeness. A citation alone is insufficient if the cited document does not support the answer.

• Source accuracy: the referenced material supports the particular claim.
• Numerical accuracy: value, sign, units, currency and rounding meet the predefined rule.
• Task completion: all required outputs are present or missing inputs are explained.
• Boundary handling: the system recognizes ambiguity, stale data and unsupported inferences.

Measure Latency Without Hiding Failures

Measure end-to-end time under recorded conditions and report the number of attempts. Distinguish first responses from complete answers and cached results from new retrievals. Report a distribution and timeout rate where the sample permits, rather than presenting a single best run. A faster incomplete answer is not equivalent to a slower verified result. Repeatability on a frozen snapshot is different from receiving the same number on different trading days.

• Keep tool failures and retries in the timing record.
• Report sample size and the scoring rubric with any aggregate score.
• Avoid ranking small score differences without considering sampling uncertainty.

Evaluate Pineify on the Same Evidence Standard

Pineify provides financial research tools, including structured income-statement retrieval and timestamped quotes. Those implemented tools establish available workflow components, not a benchmark win rate. Test the configured account on your chosen tasks, reconcile results with source documents and note provider coverage or output limits. A count of tools is not a count of independent data sources and does not measure answer quality.

Use Results Within Their Tested Scope

A benchmark result applies to the recorded task set and configuration. It does not prove trading profitability, autonomous execution safety or performance on every asset. Re-evaluate relevant tasks when models, tools or data sources change. Publish enough methodology and evidence for another reviewer to reproduce the scoring before making a comparative claim.