Well if we use vague metrics like those, then anybody can claim anything. “more than 1/20 tokens wrong” what does wrong mean? One view is that a computer program is never wrong, it does exactly what the code says. Another view is that if the AI ends up at a verifiably incorrect answer (for example if you prompted it with a math question), then all the tokens it gave out were wrong. But then humans can be wrong too. Are humans 1/20 neurons wrong on average?
For comparison, Moore’s law uses well defined metrics like computations per second. That’s what made it a useful concept.
Those papers are based on dense model architecture. The MoE architecture that I mentioned a few comments ago, does not follow the same laws. Architectural changes could push us past the wall.
Not to mention if we reach 90% accuracy (which was defined in those papers to be AGI level), there’s no reason we will need to keep making new models and training them. AGI is good enough. After that we improve inference performance and bring inference cost down.
Hallucinating 1 in 10 fractions of a statement is not AGI by how anybody with half a brain defines it.
A statistical model hallucinating 1 in 1000 isn’t even AGI.
AGI is the capability to solve a riddle without being trained on infinite copies of the same riddle, which these machines guessing the next word in a seque have never shown any capacity for.
Well if we use vague metrics like those, then anybody can claim anything. “more than 1/20 tokens wrong” what does wrong mean? One view is that a computer program is never wrong, it does exactly what the code says. Another view is that if the AI ends up at a verifiably incorrect answer (for example if you prompted it with a math question), then all the tokens it gave out were wrong. But then humans can be wrong too. Are humans 1/20 neurons wrong on average?
For comparison, Moore’s law uses well defined metrics like computations per second. That’s what made it a useful concept.
Remember several comments ago when I said it cannot get above 94% with approaching infinite power, compute time, and data? 5% is 1 in 20. SMH.
That number is based on the OpenAI and DeepMind papers on AI Scaling Laws which predicted the performance of every model in the last 6 years.
Those papers are based on dense model architecture. The MoE architecture that I mentioned a few comments ago, does not follow the same laws. Architectural changes could push us past the wall.
Not to mention if we reach 90% accuracy (which was defined in those papers to be AGI level), there’s no reason we will need to keep making new models and training them. AGI is good enough. After that we improve inference performance and bring inference cost down.
Hallucinating 1 in 10 fractions of a statement is not AGI by how anybody with half a brain defines it.
A statistical model hallucinating 1 in 1000 isn’t even AGI.
AGI is the capability to solve a riddle without being trained on infinite copies of the same riddle, which these machines guessing the next word in a seque have never shown any capacity for.
I know humans that hallucinate more