Just a quick note for anyone looking to do some light recursive improvement, I made a light customisation for the spark : Here
I had to remove flash attention and replace with Pytorch’s scaled_dot_product_attention, but it works. It was a five minute job so there’s probably lots of optimisations I missed, but i guess that’s half the point.
Enjoy
We will test. I think i like the sleeping part of it.
Would love to see some output.
Nice, when i get a mo i’m going to turn autoresearch itself into an RL gym, so ppl can train the research model also, akin to that recent Alibaba paper where it went off to crypto mine.
One spark for the research model and one for the autoresearch subject. Will do a solo node version for those with one spark as well.
Probs in the next week
I was reading when i get amo. Thinking of more sparks.
Just swap the pubkey or add multisig when it does not look, so it has to ask for its own money.
I had to do the same thing on my Gigabyte GB10, and SDPA works - I wonder how much faster the flashAttention would’ve been…
Also, would love to know when FA3 CUDA kernels will be built for GB10 architecture (sm_121a binaries)
Here’s my benchmark for comparison:
36 steps (vs 952 on H100) O.o
Training Time = 310.6s
Total Time = 617.4s (first run was slow due to compilation)
27 experiments on GB10, val_bpb = 1.174 on TinyStories (8K vocab, 5-min budget)
Key findings: FP8 via torchao unlocks tensor cores (~15x over BF16 CUDA cores). Optimal config is much shallower than H100 — depth=3, 384-dim (17.9M params), batch 2^16, warmup 2%, warmdown 70%. Gets ~1,290
steps in 5 minutes at 281K tok/sec, using only 6 GB VRAM.
Hardware determines optimal architecture — H100 priors (depth 8-9, wider models) don’t transfer to GB10. Smaller model + more optimizer steps wins under fixed time budget.
Fun times
@grindstone Great suggestion!
The FP8 optimized config is a massive improvement: