William Eliot

I don't think it's possible to build a frontier LLM, or image / video / world model for that matter, without using unconsented training data. Even if you invent an algorithm many times more efficient than transformers, there’s just not enough ethical data around to beat SOTA. And any company which is happy using unethically-sourced training data will be able to draw from a much larger, broader universe of data. So I’ll say the quiet part out loud. If you want to be at the frontier, with an all-purpose model, in a race against dopers, you can't do it on clean data alone.

I know because I tried training a model (mindforge-r1) on said data. I took Apertus-8B-2509, a pre-trained base model from a Swiss supercomputing project built on 100% compliant tokens, and I collected all the post-training datasets that were open source, ethical, and easily accessible online. There’s only about ten of them. Then I fine-tuned Apertus, using all the snazziest post-training techniques like OPSD, SDPO, and STaR. After around $1,000 in Modal compute, the 8B model had already been through several epochs of the same data and had reached diminishing returns.

The final model performs somewhere between GPT-3 and GPT-3.5. Not great, but also not that bad given the data provenance, the budget, and the number of params (GPT-3 had 175B). To me these results indicate the power of self-distillation; with a more powerful instruction-follower I’m sure you could get even better results. Self-distillation feels like an exciting new way to bootstrap your model with much less data… using a decent verifier, you can go surprisingly far. But it's not good enough. The world is currently on GPT-5.6, about six years on from GPT-3. I assume a lot of OpenAI's data is ethical today, in that it's collected from their chat users who've given consent, or bought from other companies, including RL env startups, or generated in fantastic synthetic data pipelines that can convert one clean article into hundreds or thousands of training sequences. Most of their RL nowadays is probably from clean sources, but I also know that they didn't get to GPT-3 without books3 (ahem I mean “books2”).

mindforge-r1 wouldn’t have many customers over API, and that’s kinda my point. I exhausted the internet’s easily accessible compliant datasets and trained the strongest model I could, but it’s many generations behind in performance. As far as I know, I did train the world's first reasoning model trained on compliant post-training data, but its capabilities won’t win over general purpose API customers. Anyway, the base Apertus training data wasn’t filtered for opt-ins (only opt-outs), so even in one of the most careful LLM training runs ever, I can’t reach ethical perfection.

The UK’s copyright law seems to require that if you train a model in the UK, all the training data must be commercially licensed or otherwise compliant. That’s one reason why no significant LLM has been trained here yet. But there is a workaround, which is you simply train your model on compute in a different country and deploy it in the UK. So it'll be interesting to see how the UK's Sovereign AI frontier wannabes end up performing. Perhaps they have access to a lot more commercial data that I can’t afford. Perhaps they’ll use synthetic data for most of it. Only training on compliant data is a bit like putting on the brakes — we actively hold ourselves back by banning taboo methods (which is admirable). If we maintain our principles here, we’ll basically be saying we're okay with losing the AI race since we care more about the fairness of the training data than we do about producing a seriously competitive model.

You can produce somewhat commercially viable models from clean data, but being truly web-scale means you get access to billions of data samples rather than thousands or millions. As any algorithmic efficiency compounds with more data, it follows that being frontier in music, image generation, video generation, and so on, requires you to get there off the back of dodgy datasets, since we live in a world where the frontier is defined by people doing exactly that.