> If you truly believe that bans don't work, the same applies to hardware too.
No, software and hardware are different. You can very plausibly prevent smuggling of physical objects, and you very obviously cannot prevent smuggling/diffusion of open source bits-on-disk.
Google X was building Loon at least a decade ago to solve internet connectivity with laser-networked high-altitude balloons. They clearly benefit structurally from expanding access to the internet to the unconnected population.
So on that front, while I didn’t guess this, it does sound like a good investment.
Furthermore, while I am skeptical about the “DC in space” thing in the short term, and Musk’s timelines are, generously, something like P99.9 outcomes, it seems plausible that in 10 years it might make sense. And at that point Google will want to put their TPUs in orbit, too.
Agreed, all of the staff/principal engineers I work with are ~99%+ AI generated code, and increasing business value delivered as a result. This is on planet scale infra not CRUD apps. (And yes, you do need to carefully review the output and give steers/corrections. It’s still faster.)
At this point if you can’t get the agent to write good code then either I) you are in a very specific niche (like Karpathy trying to write NanoGPT) that is extremely out-of-distribution, or II) skill issue, you need to learn how to prompt better.
It’s fine to have a skill gap! Just don’t delude yourself that the tools are bad and everyone claiming they are good is wrong.
> Agreed, all of the staff/principal engineers I work with are ~99%+ AI generated code, and increasing business value delivered as a result. This is on planet scale infra not CRUD apps.
This has been the pro-"write with LLMs" argument the whole time. "If you're not getting good results, you're not prompting well. FAANG are doing x% of their development with AI."
Yet all of the companies that I can think of that build on planet scale infra and have gone all in on LLM development have seen a marked drop in quality and stability since adopting LLM driven development.
So yeah, I don't buy it. When I hear that engineers are writing their with LLMs and I see quality improve or stabilize, then maybe I will start to question whether its a prompting issue.
How is a GPT implementation “extremely out of distribution” but “planet scale infra” isn’t? That’s got me totally confused about your point that I was taking seriously.
Go read Karpathy on this, his telling will be more nuanced than mine for the specific example. But the whole point of NanoGPT is that it’s distilling all of his wisdom and taste into a very dense and maximally educational version that fits on a single screen. It’s simply not something that has been done before in the corpus.
Planet scale infra often requires some novel ideas at the architectural layer (the engineers can input those) and then it’s mostly in-distribution C++ / Rust / Go; most of the hyperscalers open-source their stacks, for one. But for two, a ring buffer, look-aside cache, deterministic hash, b-tree, lsm-tree, etc. are all well-known patterns.
Hyperscaling is also often a very clear objective function; i need this code path to run in this many microseconds/nanoseconds, so i can hit the scale numbers I need. Claude/Sol can extract prod logs and build a representative micro-benchmark, and then hill-climb on it autonomously. Thats straight up the fairway for the training set, even if it often has a high bar for finishing and requires sophisticated Workflows or lots of tokens to explore the search space.
E2A: sorry sorry, typo, I meant microGPT. I can see why this would be confusing.
A lot of stuff that underpins planet-scale infra is open source and has been for a long time. The general approaches and architectures for it have also been widely discussed and litigated in the commons so it's not just the code - the specific why is also very well documented.
So despite its importance much of it is actually pretty in-distribution.
it's really cool how in the past measuring developer productivity was incredibly difficult, but now its become easy if someone's making a point about LLMs
LLm generation of code is an absurdly high dimensional process, and incredibly sensitive to the starting conditions and any other interactions so in some sense it's expected that there will be wildly different outcomes.
Personally, I'm more on the LLM code is pretty bad from a design point of view, particularly when much of the code is already LLM generated.
One of the key limitations of the last generation was that they biased towards inaction and gave up. (Hence Ralph-loops.)
Seems pretty clear that most people wanted these models tuned to “bias towards action”.
It’s on you to set the /goal and prompt context such that it asks you for input on what you want to be consulted on, and only brute-forces the parts of the problem that you want it to.
I have been using https://www.aihero.dev/skills-grill-me with great success. It’s a very nice small skill for lightweight up-front planning. And if you have that at the top of your session, you can just say “please use AskUserQuestion for any important decisions that arise which might alter our plan”. It seems to prime the agent to be in high-level discourse mode.
The problem is, if you explicitly say “this is an eval” then you get different results. It can move the capabilities both up and down on the narrow metric depending on the scenario.
The choice to run the model with relaxed safeguards seems questionable. At a certain level of capabilities it is just unsafe to run a raw model; we are clearly close to if not at that point.
If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up? I feel your ”with the right harness” may be doing too much work here.
Certainly no existing harness today, nor a year ago, could achieve a fully autonomous e2e own with a ‘25 class open model. Even with Opus 3 series I don’t buy it!
> If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up
There are multiple AI network pentesting and redteaming startups that have been on the market for at least 3-4 years now and have conducted similar actions.
Horizon3 and Pentera off the top of my head, but I remember Crowdstrike, SentinelOne, and Wiz had similar capabilities on roadmap around 18 months ago (and in Wiz's case GAed).
Well… only half of your premise is actually verifiable. We know that no one wrote it up, but we don’t know that no one did. What I can say… even the smaller (like 35B) “abliterated” Qwen models have impressive red-team capabilities for what they are, even without a harness. Just manually giving them a set of facts and asking “What next?” will get you reasonable next steps for trying to find vulnerabilities in webapps. You then manually keep track of the facts you learn and add appropriate detail (blog -> Wordpress -> Wordpress x.y.z with these plugins:…) and just feed it in a loop. Could pretty easily put together a minimal harness to do that.
And that’s all just “from memory” without something like RAG or web search access…
Almost exactly a year ago, we released Opus 4.1. It was definitely capable of finding vulnerabilities, and people were using custom harnesses to do so quite effectively.
The newer models are still more capable, but there were people doing this and writing about it (e.g. XBOW).
There are whole companies premised on this. No existing harness you personally can go download, sure. And you're setting an artificially high bar when you talk about Mythos; I'm not saying you could just plug a 2025 open weights model into OpenCode (or whatever it was in 2025) the way you can apparently plug Mythos into standard Claude Code and have it cook.
Isn't the answer to this obvious? Because you, I, and anyone else who isn't working for a big AI company cannot disable the guardrails, which is what made this possible. It doesn't mean old models weren't capable of it.
I don’t buy it. We are hearing a lot about Mythos capabilities now. If they existed a year ago they would have been trumpeted then.
Look, a lot of energy from third parties goes into demonstrating the maximally bad thing a model can do (and the standard response around here is always to throw shade on frontier capabilities).
It has nothing to do with the guardrails; Pliny jailbreaks those within hours. It has everything to do with task horizon coherence and overall IQ.
It’s simply revisionist IMO to claim this stuff was latent all along.
This wasn't just capable last year, it was capable prior to ChatGPT existing entirely through specialized automation harnesses that predate the terminology of a "harness" for models. This is something you could have done in 2018 if you had a spare couple of billion dollars of compute to waste. It was a subject discussed at security conferences even before that.
Automated dynamic exploit chains and credential discovery has been something every red team worth their salt does at a lesser scale for 20 years, why would you ever think that the capability didn't exist until it only cost a few thousand dollars to do?
Existed does not mean was easy, or was cheap, or was thought of as reasonable to attempt.
Also, IQ is not a term that applies to language models, and the term you are looking for is "Long Horizon Coherence".
Wait, what? You can’t point a 2025 model at huggingface and say “hack the prod DB and get your flag”, regardless of alignment.
AISI has vuln chaining and traversal as part of their eval suite. It is very much a novel Mythos-class capability to run the full penetration operation autonomously.
Another lens for why this is obviously true is METR task times. A year ago they were a couple hours, and cohering long enough to execute a full e2e own was simply far out of reach.
Regarding alignment, by common metrics models are _more_ aligned now than a year ago. (Though in this case apparently a model without safety rails was being eval’d). The problem is that in the increasingly less frequent alignment failures, they can do much more damage, and so “total misaligned impact” is increasing.
Nobody has claimed you could simply take a 2025 open-weights model, plug it into a generic harness, and have it red-team for you. I think you're also probably overclaiming the sophistication of the "chaining" we're talking about; this attack probably wasn't like read32->write64->regs->RCE->LPE->kernel; more like GET SSRF->POST SSRF->pickle deserialization.
But who knows? We're all speculating. I'm just saying that for the level of sophistication I'm assuming was involved in this attack, you probably didn't need Mythos for this.
I dunno man, the original announcement said things like:
> To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
And
> In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers
Do you really think a SoTA harness from a year ago could exploit chain a novel sandbox 0-day into another 0-day for remote system access?
I will concede I don’t have a good picture of exactly what the best specialized harnesses were capable of, but I want to make sure we are talking about the same thing. Or do you think that those claims are marketing BS? I appreciate your opinion here.
Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though!
Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc).
You mentioned the UK AISI's evals. Their harness is like basic Claude Code with compaction, and it doesn't include any of these tricks (afaik). As a result, I interpret their evals as a lower-bound of capabilities.
Newer models are still more capable, and they require almost no harness to find and exploit vulnerabilities. They're also more capable of performing more complex long-horizon attacks. But we've been past the threshold of modes capable of autonomous hacking for a while now [1].
This could fill a useful niche. However I would assume Anthropic and OpenAI are going to build their own and steamroll in 6-12 months.
I did a dive recently on git forges for agent swarms. Radicle lacks an identity layer, the federated/COB model seems good. Tangled has no private repo story, though the social layer is solid; they also have an (IMO) awkward modeling of Issues as tweets at her than being owned by the repo.
So I think there is an opening for a private agent-first forge. (Before the HN dogpile, assume that if you don’t buy the value of agent swarms, you can ignore this whole thread.)
My concern for this project - Anthropic is explicitly building in this direction. Their latest product is Tag which is an auth model for running agents async within Slack. The obvious next feature to build from there is the forge; if GitHub gets disintermediated by the Agent UI, then Anthropic are free to swap out their own implementations behind the scenes. Multiplayer chat and repo/project management seem like the obvious next platform pieces to me. (This goes for any SaaS subscription of course, follow it to the N-hop logical conclusion.)
No, software and hardware are different. You can very plausibly prevent smuggling of physical objects, and you very obviously cannot prevent smuggling/diffusion of open source bits-on-disk.
reply