KEY RESULT

Deterministic verification improves numerical accuracy by 42× while reasoning-based methods (CoT, RAG) consistently degrade performance. This challenges the dominant assumption that scaling up reasoning is the primary path to improvement in financial LLMs.

RESEARCH METRICS

Modular Verification Outperforms Chain-of-Thought Reasoning in Small Financial LLMs

Aaditya Thokal · Universal College of Engineering · FinNLP @ EMNLP 2026 (Submitted)

HuggingFace Model ↗GitHub ↗Paper 🔒

0.00%

FINAL ACCURACY

n=873 FinQA dev set

IMPROVEMENT

over unaugmented baseline

DVL CORRECTIONS

parameter-free, zero training cost

EXTRACTION FAILURES

after QLoRA fine-tuning

0.00%

REASONING ERRORS

dominant remaining failure mode

ABLATION STUDYProgressive pipeline

Configuration	Accuracy	95% CI	Delta
Baseline	1.00%	[0.4, 1.9]	—
+Doc Context	24.00%	[21.2, 26.9]	+23.0pp
+DVL v1	32.00%	[29.0, 35.1]	+8.0pp
+QLoRA FT	38.50%	[35.4, 41.7]	+6.5pp
+DVL v2	42.61%	[39.5, 45.7]	+4.1pp

NEGATIVE RESULTSApproaches that degraded accuracy

Configuration	Accuracy	95% CI	Delta
+CoT zero-shot	29.50%	[26.6, 32.6]	-9.0pp
+CoT FT v1 (3K)	26.50%	[23.7, 29.5]	-12.0pp
+Cross-doc RAG	31.00%	[28.0, 34.1]	-7.5pp

ACCURACY PROGRESSION

ERROR TAXONOMYn=539 errors analyzed

539

total errors

ERROR BREAKDOWN

Key Insight

73.1% of failures are multi-step reasoning errors — where the model locates the correct figures but loses numerical precision across sequential computation steps.

Reasoning close (<50% rel.)21039%

Reasoning far (>50% rel.)18434.1%

Magnitude (>2 orders)6612.2%

Order-of-magnitude (1-2 ord.)6211.4%

Sign error91.6%

Scale error40.8%

Unknown40.8%

ROBUSTNESS ANALYSISDVL under simulated real-world noise

Condition	FT Only	FT+DVL	FT Drop	DVL Benefit
Clean data	38.5%	42.6%	—	+4.1pp
Distractor numbers	30.6%	30.7%	-7.9pp	+0.1pp
Missing values	32.1%	42.6%	-6.4pp	+10.5pp
Conflicting signs	34.1%	42.6%	-4.4pp	+8.5pp

HOW IT WORKS

Scale Correction

0.1240→ scale_mul100 →12.40%

When financial ratios are expressed as decimals (0.12) but the question asks for a percentage, or vice versa. Triggered by keywords: margin, return, yield, growth, change, ratio.

Paper case: Operating margin — predicted 0.1240 → scale_mul100 → 12.40%
(value <1 with ratio keyword → clearly a decimal needing % conversion)

Sign Correction

-0.3411→ |negate| →+0.3411

When the model computes the correct magnitude but inverts the sign. Common with directional questions. Applied only when magnitude matches within 5% tolerance.

Paper case: HTM securities decrease — sign flipped from negative to positive

Magnitude Correction

104.0→ ×10 →1040.0≈ 995

When figures appear in wrong unit denominations. Tries multipliers [×10, ×100, ×1000, ×0.1, ×0.01, ×0.001] and applies the first match within 5% tolerance.

Paper case: Class A shares — magnitude adjusted by ×10