> They clear their mind and let the solution come.

They work on a problem until their brain is full of problem-related concepts. Then something comes. After validation it might be a solution.

16GB VRAM is probably not quite enough. With 32GB, or maybe even 24GB, you can do serious coding work with local models.

For decode, memory bandwidth is the main bottleneck, so these machines will likely perform well even without a ton of GPU horsepower. Not as well as Blackwell, but I expect they will be a reasonable choice in terms of price/performance if you want to run large models with a lot of context.

The main place they are a bit behind is in the number formats they support natively. Iirc M5 doesn't have native FP8 support, so you will take a speed penalty on quants where other architectures get better acceleration.

As someone who does a lot of work with local LLM's, today's systems feel woefully under-powered. I'm looking forward to a future where my laptop has 10x the memory, 100x the memory bandwidth, and optimized cores to make inference workflows that currently take minutes or hours go down to seconds or milliseconds.

While we're at the point where traditional software is pretty much fast enough for all but extreme use-cases, with LLM's it feels like we're back to the days where you press compile and go have a coffee or chat to your colleague.

Right now there isn’t really a convincing use case for local LLMs outside of enthusiast or specific niches.

It takes a lot of expensive hardware, and most people don’t have extreme enough requirements to recoup that investment.

That might change, but it would require either that hardware gets much cheaper or everyone’s demands for LLMs increase significantly.

That's what I mean though. The comment I was responding to was talking about "forever laptops" - my point is there's plenty of room for new capabilities which will make current hardware obsolete. Just like how GPU's didn't exist at all, and became a standard part of computing.

And given how fast the hardware and software is evolving, I can easily imagine a future where we all have very capable models running on our own devices for an embedded intelligence layer that's doing most of the day-to-day tasks, and only have to outsource to a super-smart cloud model for specific things.

That's the most basic version of a harness. But a harness is really about automating context delivery to the LLM based on your use-case.

Is that the harness' job? It seems to me the best place for sandboxing is at the OS level (i.e. running the harness inside a container with correct access configured).

I would argue that it is the harness's job... Certainly for any harness which is targeting an end user.

They don't sell cars without seat belts, and it is the car manufacturer who has to do it.

Also, small nitpick but technically a container doesn't give full isolation compared to something like a VM.

Ok that's fair if it's targeting a non-technical audience.

But I think this will eventually be a problem solved at the OS level in a more streamlined way. I.e. there will be fine-grained permissions you need to approve to give an agent access to the system.

You want both. The harness and the sandbox do different things.

The harness says "You have access to tool X, Y, and Z, but not A, B, C".

The sandbox says "If you try to use X to access a forbidden resource, I'll prevent you from reaching it".

And despite the enormous capital expenditure, Chinese models are nipping at their heels at what must be a fraction of the cost. Sometimes constraints are healthy for inducing creative solutions.

Is huggingface profitable?

I'm not a fan of big-tech acquisition results either, but one benefit can be that a product continues to exist when it would otherwise become insolvent.

That could be a factor, but the optimistic interpretation would be that they want to support the open ecosystem because it sells more chips.

Models are already largely hardware agnostic. It would be pretty hard to put that cat back in the bag.

I could imagine them building value-added services on top of HF to advantage Nvidia products (i.e. "run this model on NVIDIA cloud" with one-click), but in this moment it's hard to imagine how they could actively disadvantage models built to run on other platforms.

They didn't, but their price increases haven't kept pace with general memory price increases (yet) so currently their pricing seems reasonable.