Add the OOM's and they can outpaces humans - see here.
Given OpenBSD, one of the, if not THE most secure operating system written by humans.
The latest Claude model found critical vulnerabilities in the operating system - dating back 20 YEARS.
People on their Macbook Pro's will have this capability within < 1 year.
Anthropic's decision not to release the weights of their most capable model, reportedly because it's considered too powerful to release openly [5] shows the bind perfectly. Release it and lose control entirely. Withhold it and the open-source community replicates it within a year regardless.
tech-insider.org/anthropic-claude-mythos-zero-day-project-glasswing-2026/
The second is more unsettling. Those guardrails only exist because the companies control more compute than anyone else. Once equivalent capability is available open-source - and we're clearly there now, with models like Llama and Mistral reaching near-parity with first-gen frontier models [2] [3] - they simply don't apply. Anyone can remove or ignore them.
Or more specifically, Gemma 4.
I read this when it came out, 2023/23? and I thought this was nonsense.
And yet we are in 2026/27.
The OOM's he described have been met (later than thought, but barely). And We have Microsoft, Google, OpenAI not only buying Gigawatss of power but building their own power stations to their datacentres.
Now at this point I must provide a contrary voice - The Enshittifinancial Crisis
Again, a very long read. Which I believe to be entirely true.
But even based on the above, I think the forward motion genie cannot be put back in the bottle.
I think the next 10 years will be the most turbulent in history.
using Claude
I recall @Bozza talking about Claude in the tribe for this year's April ool's joke, and @Vermillion-Rx talking abut not trusting it in the same tribe.
I don't think I'll be trusting any of those things with anything important, ever. Maybe for speeding up research, but I still would want to verify results myself.
Fuck, I remember reading about some law firm using AI (forget which one) to research a case, and it just made shit up. Cost them millions to fix things after they trusted it and ran with its results.
I'd highly recommend reading through Situational Awareness
It's a very long read. And while he may have been a tad optimistic, his prediction aren't really that far out from what has been achieved.
I sit somewhere in the middle on this. I see a strong use case for LLMs but I don't fully trust them. Both in terms of output quality, and more critically, who controls the input.
The input side worries me more. LLMs have a potential for data farming on a scale beyond anything we've previously seen. Everything you share is in most cases logged, retained, and used. The T&Cs are long; almost no one reads them.
Several years ago I wrote a fairly detailed analysis predicting that LLMs would eventually "break free" of the monopoly held by a small number of well-capitalised organisations. In the early days, the compute and CapEx required to train and serve these models was enormous. Only the likes of OpenAI, Microsoft, and Google backed by billions in VC could afford it. [1]
But hardware improves. Algorithmic efficiency improves. What required a data centre in 2021 requires an enthusiast's desk in 2025.
The only guardrails that currently exist are those imposed by gatekeepers who have the capital. As open-source compute catches up, those guardrails become increasingly meaningless.
This matters for two reasons. The first is access, ordinary people can now use genuinely powerful AI without surrendering their data to a corporation. Tools like LM Studio let you run models comparable to GPT-4 locally, on consumer hardware, with no data leaving your machine and no corporate filters applied.
The second is more unsettling. Those guardrails only exist because the companies control more compute than anyone else. Once equivalent capability is available open-source - and we're clearly there now, with models like Llama and Mistral reaching near-parity with first-gen frontier models [2] [3] - they simply don't apply. Anyone can remove or ignore them.
The time delay between a frontier model dropping and an open-source equivalent reaching normies has gone from years to months, and it's still shrinking. [4]
I highly suspect within the next year or two, Open Source compute capacity will maybe even outnumber what the likes of OpenAI or Anthropic can attain, even with hundreds of billions of CapEx.
Anthropic's decision not to release the weights of their most capable model, reportedly because it's considered too powerful to release openly [5] shows the bind perfectly. Release it and lose control entirely. Withhold it and the open-source community replicates it within a year regardless.
Neither path leads somewhere obviously safe. The genie is out of the bottle.
Read MoreAs for hallucinating, it is pretty real. In your use case I'd try to search eg 'is Claude hallucinating on only user supplied data'
Ah yeah you’re right on that. I do a cycle between each input session, I vet the output for inconsistencies or hallucinations.
In one case, it had applied the model I defined for one rule set to another. I caught it, because it had introduced information that didn’t share context. Well anyway, I tightened the rule set and repeated the cycle!
That's the only news last moth that I had to reread a header and go to verify
