I think we will go through the same process as we did with electric cars. First “terrible, the work if satan.”, “this is no good”, “it is worse than the old” … resist resist. A few will adopt, the tech gets better, more will adopt, then finally almost everyone agrees it is actually quite good, we want this.
Smart use of local LLM is good :)
Producing funny pics, using big llm as search engine, or worse as an encyclopaedia, etc is bad.
The technology cannot actually improve. Increasing model scale increases power costs and compute time in return for higher accuracy. The returns from increasing model scale at optimal compute times diminish at a rate which will stop producing better results when about 94% accuracy is reached with literally infinite training data and compute time. There is no evidence that this approach will ever be improved upon in a way that approximates human output such as an AGI.
Even if it were free it creates liability and it’s still also free to simply just not to use it and since it isn’t necessary for literally any task: the costs will never be justified under any circumstance.
in response to your first point: 24B models from this year are far better than 24B models from 3 years ago. Same model size, similar energy consumption, far better results.
30 years ago there was a lot of doubt that Moore’s law could continue the pace for so long. There was no evidence that PCs would continue improving, and yet they did.
So it’s really anybody’s guess whether or not AI will continue improving. But with the amount of money being poured into it I’m willing to bet it will.
Okay, first of all, Moore’s law continuing was caused by a different approach to memory technology which allowed vertical stacking of silicon. Secondly, you literally just gave an example of diminishing returns making steady improvement physically impossible in spite of public expectations, unless a new and different technology is developed, so if anything that’s a great argument against AI not for it. Thirdly, Moore’s Law is still fucked because right now we’re etching silicon with gamma rays and a near-perfect lense and mirror which means we pretty much hit the physical constraints of etching smaller circuits.
Moore’s law continuing was due to a ton of different advancements and innovations, not just one. And yes it’s slowing down but it still went for 30+ years. If AI continues to improve at this rate for 30 years, hard to imagine how good it could get.
There’s been a ton of innovation in the space right now. Like MoE, which was only introduced like 2 years ago and now it’s everywhere. It’s hard to say what can happen when you have millions of engineers working on something.
My argument is that the shitbots will always get more than 1/20 tokens wrong, have no contextual understanding, and have no morality to speak of such that it will never be improved upon past its current limited and useless form.
Unless a new so-far unheard of technology changes it, AI is currently worthless.
Well if we use vague metrics like those, then anybody can claim anything. “more than 1/20 tokens wrong” what does wrong mean? One view is that a computer program is never wrong, it does exactly what the code says. Another view is that if the AI ends up at a verifiably incorrect answer (for example if you prompted it with a math question), then all the tokens it gave out were wrong. But then humans can be wrong too. Are humans 1/20 neurons wrong on average?
For comparison, Moore’s law uses well defined metrics like computations per second. That’s what made it a useful concept.
I think we will go through the same process as we did with electric cars. First “terrible, the work if satan.”, “this is no good”, “it is worse than the old” … resist resist. A few will adopt, the tech gets better, more will adopt, then finally almost everyone agrees it is actually quite good, we want this.
Smart use of local LLM is good :) Producing funny pics, using big llm as search engine, or worse as an encyclopaedia, etc is bad.
Two problems:
The technology cannot actually improve. Increasing model scale increases power costs and compute time in return for higher accuracy. The returns from increasing model scale at optimal compute times diminish at a rate which will stop producing better results when about 94% accuracy is reached with literally infinite training data and compute time. There is no evidence that this approach will ever be improved upon in a way that approximates human output such as an AGI.
Even if it were free it creates liability and it’s still also free to simply just not to use it and since it isn’t necessary for literally any task: the costs will never be justified under any circumstance.
in response to your first point: 24B models from this year are far better than 24B models from 3 years ago. Same model size, similar energy consumption, far better results.
30 years ago there was a lot of doubt that Moore’s law could continue the pace for so long. There was no evidence that PCs would continue improving, and yet they did.
So it’s really anybody’s guess whether or not AI will continue improving. But with the amount of money being poured into it I’m willing to bet it will.
Okay, first of all, Moore’s law continuing was caused by a different approach to memory technology which allowed vertical stacking of silicon. Secondly, you literally just gave an example of diminishing returns making steady improvement physically impossible in spite of public expectations, unless a new and different technology is developed, so if anything that’s a great argument against AI not for it. Thirdly, Moore’s Law is still fucked because right now we’re etching silicon with gamma rays and a near-perfect lense and mirror which means we pretty much hit the physical constraints of etching smaller circuits.
Moore’s law continuing was due to a ton of different advancements and innovations, not just one. And yes it’s slowing down but it still went for 30+ years. If AI continues to improve at this rate for 30 years, hard to imagine how good it could get.
There’s been a ton of innovation in the space right now. Like MoE, which was only introduced like 2 years ago and now it’s everywhere. It’s hard to say what can happen when you have millions of engineers working on something.
Spending fucktons more power to solve problems that don’t exist isn’t innovation, it’s waste.
My argument was that the models would get more powerful over time, even when controlling for size and energy usage. I wasn’t talking about waste
My argument is that the shitbots will always get more than 1/20 tokens wrong, have no contextual understanding, and have no morality to speak of such that it will never be improved upon past its current limited and useless form.
Unless a new so-far unheard of technology changes it, AI is currently worthless.
Well if we use vague metrics like those, then anybody can claim anything. “more than 1/20 tokens wrong” what does wrong mean? One view is that a computer program is never wrong, it does exactly what the code says. Another view is that if the AI ends up at a verifiably incorrect answer (for example if you prompted it with a math question), then all the tokens it gave out were wrong. But then humans can be wrong too. Are humans 1/20 neurons wrong on average?
For comparison, Moore’s law uses well defined metrics like computations per second. That’s what made it a useful concept.
@oyzmo @Sunshine The real driver (ahem) at the end of the day is cost. People will buy them when they are cheaper than the alternative.
…or sadly if some uneducated tiktoker with tons of followers think it is cool …😅🐑🐑