Who is this vendor that is consistently providing high quality inference for all families of open weight models at a competitive cost?
That's a serious questions that I really interested in the answer too. I have 25 providers included into my coding agent harness not because I care about vendor neutrality, but because I have to keep adding new ones as inference quality degrades at the providers I was using. Its quite tiresome.
It's more than just quantization. The middleware the provider is running matters a lot even to the point of exactly which version they are running due to defects being introduced / resolved.
In my coding agent harness I've included 25 open weight providers mainly because I keep having to find new ones when what was previously a great combination of model and provider becomes pretty bad. vllm has defect that causes reasoning to get dropped much of the time for the GLM family of models. sglang has a defect that causes the elements of array args to get dropped for the deepseek family of models. Some providers need some very specific additional config passed through for reasoning to make it back to the model.
I've not tried OpenRouter as adding yet another layer will just make it that much more difficult to get a model and provider combination working well.
I suspect people's bad experiences with open weight models have a lot to do with these headaches. Finding a good model and provider combination is pretty tedious and so far its been a never ending process. I'd really like to host my own models but it isn't economically feasible for one person for the open weight models that work well (i.e. the 300B+ ones).
They aren't grown/evolved from data, they are fit to the data. The fitting process can be fully deterministic although its fairly easy to screw things up such that it isn't deterministic, but that just a defect not some fundamental shift.
This assumes the functionality of brains can be fully captured as a deterministic mathematical function, but the function of the brain may well depend on nondeterministic quantum states that can't be reduced to stateless functions: https://en.wikipedia.org/wiki/Quantum_mind
The problem with this line of thinking is that modern computers are nothing like the brain. LLMs don't stand on their own, they have to be run on these modern computers, but doing so does not change the physical properties of the computer.
The simulation you propose of the brain is likely impossible due to quantum mechanics making it impossible to fully simulate: https://en.wikipedia.org/wiki/Quantum_mind
Perhaps we'll be able to build an artificial brain that includes the same quantum properties as biological brains, but this won't be a simulation of a brain it will be a synthetic brain.
At first I was inclined to agree with you, but then I realized that the brain requires this whole complicated contraption (the body) to run and, really, do anything at all. And while I'm not familiar with the notion of 'quantum mind' I do think that biological processes aren't deterministic (at a cellular level).
And I think this does mirror the situation with LLMs -- you need this whole computer contraption and GPU, also running on electricity, to support the LLM's "thought" processes. And that if we model the brain's neurology sufficiently (which it seems we've done) we can achieve results that appear to be like thinking, even if it is an emergent behavior from "relatively" simple math.
Which actually makes me wonder the opposite -- are we, as humans, not much better than these LLMs? Suppose the body is just that super complicated computer contraption, honed by thousands/millions of years of evolution to achieve some semblance of homeostasis? If you reject the idea that we have a soul, we start to look very similar to the machines we build. "You are a brain inside a skull cockpit, piloting a bone mech covered in meat armor and skin" feels more and more relevant. That I'm just a meat circuit running brain chips and once you pull the plug on the source of electricity it all just... stops
Take a computer that can run the biggest LLM available today. It can also run any smaller LLM as well. It can also run software that isn't an LLM at all. Brains and LLMs are not at all equivalent as LLMs lack a stateful physical form while brains are very stateful. As you said once the brain is no longer maintained properly by the body it stops and transitions to a non-functional state that can be reversed. That isn't true for LLMs at all. You can copy them and run the same one on many computer, run different ones on the same computer, you turn that computer offs for long periods of time and then turn them back on keep on running the same LLMs as before on them.
If computers degraded the way flesh does once you stop fueling it I feel like that would defeat a lot of your argument. And yes the brain has inherent statefulness (you're referring to memories, I'm guessing?), we have also jerry-rigged some degree of statefulness into LLMs. Mechanically it is very different and inferior, and there is a notion of separation that probably doesn't map to brains, but I would argue that LLMs, when you look at how inference is used in situ, are not necessarily stateless.
One thing to keep in mind is that your brain's hardware heavily influences your experience and thus your brain's development.
Eg. whether you are male or female, tall or short, your limbs can make you run fast or not, your eyes can see well or not... all of these influence your experiences, your brain development, and who do you feel "you are". Try really removing all of your sensory inputs from your past, your body ability and disability, and do you think you end up the same person?
I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.
Claude Code may simply be best used with Anthropic's models and quite bad with Kimi. An alternate solution is to remove Claude Code from the diagram if its so far off from the others that it causes scaling problems.
I was looking at the results JSON and it looks like there is only one run of each task with each harness. Since these are disparate tasks using the median means that the headline cost is the cost of one specific task for each harness (or the mean of two it looks like in the case of Exo Harness; I couldn't spot the one task that lined up with the headline cost), but the same task isn't used for the headline cost for each harness. It's not the same as picking a task at random to use as the headline task, but its in the ballpark.
I've been using Brave for years and its great. Having JS off by default (i.e. shields) is excellent for security. They provide MV2 versions of AdGuard and uBlock Origin that you can enable directly from the browser settings. Those are also both excellent for security since the main threat is malicious scripts served up by ad networks.
reply