It's a problem to the economic model, too, but of course no one cares about more than a couple fiscal quarters in the future. The looming ecological disasters will irreversibly destroy the global economy in some of our lifetimes if nothing happens to stop it pretty soon.
Are you...ignorant of how many of the refugee crises are because of climate change? Like literally right now? In 2024 alone, about 40M people were displaced due to climate change, and that's a doubling from ten years ago.
I think it's just more about market incentive. At it's core, LLMs are bad for google's previous business model, which was to send you to as many sites 'good enough' for what you were looking for and plant Ad land mines along the way, in the search results and in the websites themselves.
The new paradigm is you ask the llm a question, get the answer and cutout the middle man. (yes the answer may or may not be as good as the old google result, but for the sake of the argument lets say it is), Google was in danger of simply getting their arm cut off. so they focused on scaling so they could add LLMs to the search, which they largely have. You can't offer an opus like model on something as big as search (and which is offered for 'free'), so they focused on that model, and the infrastructure to run it, because they cannot afford to lose search.
Meanwhile, they know the power of frontier models, they are working to have the infrastructure to be a huge player in them and I'm sure they will have a frontier capable model, eventually. They are playing a longer game, because they can, and I think it's going to work out very well for them.
LLM coding isn't even close to plateauing. Right now, the major players are in a consolidation step, focusing more on economic efficiency but still not at the point where we're ready to start burning models to hardware and freezing the line.
They are straddling the line between pushing it forward, and justifying the business case. It's hard to do both at the same time.
LLMs are hungry for tokens. Every day I see more startups claiming token usage at 50-100k per month. You can always brute-force your way in - just see HuggingFace's recent attacks.
If throwing more money at inference while accumulating compounding technical debt is the new norm, then we are not solving the problem, and the solution space is already covered.
Perhaps there are marginal gains at the expense of quadrillion-params LLM models with 10x the cost and energy. We are simply making inefficiency more expensive, camouflaged by VC money and great marketing.
If that is not plateauing, then I guess I will have to reconsider what plateauing means.
This is such unbelievable revisionism! Can you imagine in 2022 saying "Oh of course you can brute force your way to AGI if you spend enough money per month". Nobody thought that! Come on!
It's quite bad at role play in my (rather large) experience.
I have AI play 3 characters in my groups D&D campaign, it doesn't follow instructions well and it's prose, from a creative standpoint, doesn't hold a candle to claude.
