What we were training for
We run a 10-agent analyst “Room” for a trading-education product (simulation only). The agent that matters most reads a fundamentals brief and writes an analyst report. We wanted it better at the things a careful analyst does and a general model doesn’t:
- Basis discipline — don’t compare an annual figure to a quarterly one, or gross debt to net debt; when a brief is ambiguous about basis, say so
- Don’t invent conflicts — two sources differing by definition (cash vs cash+equivalents) is not a data conflict, and calling it one is a detectable failure
- Ratio and trend reasoning grounded in the statements, not plausible-sounding prose
- Earnings-quality flags — accruals vs cash flow, receivables outrunning revenue
- As-of-date discipline, and refusing when the brief can’t answer
All programmatically checkable, which is why we picked them.
What we did
BASE
fastino/Fastino-Nemotron-3.5-Lightning-Finance — 31.6B hybrid Mamba-MoE. Note it is already a finance fine-tune; it beat a general 35B model 43/44 vs 21/43 on our basis rubric, which is why we built on it.
LORA
r=32, α=64, lr 1e-4, cosine, 2 epochs, bs=1 × grad-accum 16, bf16, seq len 4096
TARGETS
attention q/k/v/o + Mamba in_proj + shared-expert up/down — 93 modules, 22.7M trainable. Routed experts frozen.
HARDWARE
single DGX Spark GB10, 121GB unified memory, ~17h for the successful run
EVAL
468 held-out prompts, 8 surfaces, scored by the same programmatic checks that generated the training labels; held-out tickers disjoint from training
| DATA SOURCE | SHARE | WHAT |
|---|---|---|
| Our generators | 51% | Briefs from real filings where the label is computed, not judged |
| Public finance SFT | 41% | FinQA, TAT-QA, financial-RLVR, Finance-Instruct |
| Chat replay | 8% | UltraChat slice, anti-forgetting |
Output formatting was one recipe of sixteen — 8% of the mix, and not the point.
What we wanted
Measurable movement on those capabilities — fewer basis errors, fewer invented conflicts, more correct refusals. We’d have accepted a few points on any one of them.
What we got
| CAPABILITY TARGETED | BASE | FINE-TUNED |
|---|---|---|
| Analyst report correctness | 43/44 | 43/44 |
| False conflicts invented (lower better) | 0/44 | 2/44 |
| Correct refusals | 45% | 50% |
Parity on quality, slightly worse on invented conflicts, refusals inside the noise.
The only thing that moved was output formatting — the 8%. And when we finally ran the control, decode-time schema constraint got the untrained model to the same place in ~50 minutes:
| FORMATTING CHECK | BASE | BASE + GRAMMAR | FINE-TUNED |
|---|---|---|---|
| Size-ladder compliance | 57% | 100% | 100% |
| Trader block parses | 26% | 100% | 100% |
So the one gain didn’t need training either. n=69 and n=68.
Two process failures of ours, in case they invalidate the result for you.
37% of our first eval set was verbatim training data — three surfaces were generated by calling the training-data generators. We caught it and rebuilt them; every number above is from the clean set. We also found six defects in our scoring harness, in both directions, all fixed before these numbers.
WHAT WE’RE ASKING
-
Is this expected?
Should ~10k programmatically-labelled analytical examples move this class of capability with SFT at all? Everything we taught is verifiable, but we taught it as next-token prediction over a correct answer — which may simply be the wrong lever.
-
Is it the base?
Stacking LoRA on a model already fine-tuned for finance may leave no headroom, or fight the existing adaptation. Should we train from the vanilla instruct model and accept losing its finance skills?
-
Is it capacity?
r=32 / 22.7M trainable on a 31.6B MoE with routed experts frozen. Too small a surface to move reasoning, as opposed to style?
-
Is RL the honest answer?
Every label we have is verifiable — the RLVR/GRPO setup. Is SFT just the wrong tool for “stop making this class of error”?
-
Should we do anything at all?
Genuine question. The formatting win is free via grammars, and the base is already decent at the analysis. Is there a version of this worth another run, or is the right answer to stop and spend the effort on retrieval and tooling instead?