Excerpt:
“Even within the coding, it’s not working well,” said Smiley. “I’ll give you an example. Code can look right and pass the unit tests and still be wrong. The way you measure that is typically in benchmark tests. So a lot of these companies haven’t engaged in a proper feedback loop to see what the impact of AI coding is on the outcomes they care about. Lines of code, number of [pull requests], these are liabilities. These are not measures of engineering excellence.”
Measures of engineering excellence, said Smiley, include metrics like deployment frequency, lead time to production, change failure rate, mean time to restore, and incident severity. And we need a new set of metrics, he insists, to measure how AI affects engineering performance.
“We don’t know what those are yet,” he said.
One metric that might be helpful, he said, is measuring tokens burned to get to an approved pull request – a formally accepted change in software. That’s the kind of thing that needs to be assessed to determine whether AI helps an organization’s engineering practice.
To underscore the consequences of not having that kind of data, Smiley pointed to a recent attempt to rewrite SQLite in Rust using AI.
“It passed all the unit tests, the shape of the code looks right,” he said. It’s 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It’s a dumpster fire. Throw it away. All that money you spent on it is worthless."
All the optimism about using AI for coding, Smiley argues, comes from measuring the wrong things.
“Coding works if you measure lines of code and pull requests,” he said. “Coding does not work if you measure quality and team performance. There’s no evidence to suggest that that’s moving in a positive direction.”
Oh precious. You want me to prove to you that someone presented a viewpoint that was diametrically opposed to my own and then successfully argued me around to their way of thinking? It hasn’t happened yet, not on this platform, and I shall not be linking this profile to other platforms I comment on where I have had convincing arguments sway my point of view. But surely you will be the first, you’re better than all my other interlocuters right?
Exactly. In your own words you’re incapable of changing your mind when new evidence is presented. And so why would I want to try when I know that no matter what I say, you will fight against it because winning is more important to you than having accurate views.
Oh I am afraid misinterpeting the information provided to you doesn’t bode any better for you than any other vacuous AI shill I have talked to.
Let’s try rewording this into something you can comprehend: Lemmy is one of the avenues of discussion I use. On Lemmy I have mainly talked about LLMs (you call them AI, I don’t believe for a moment Intelligence is characterised by making lots of mistakes then trying to select which one is least wrong). On other platforms I have discussed other topics, I have been convinced by arguments in relation to other matters. I use Lemmy because it affords me a degree of seperation from my other online activity. I will not surrender that seperation simply to make you feel more comfortable that your poor arguments and shoddy data have any hope of proving adequate.
Let me provide a super clear summary of my position, the scanty insubstantial benefits of LLMs are being overhyped by shills and conmen to prop up the “AI” bubble. Enough large businesses and governments have bought into it now that number must go up is the only reason it’s having the societal impact it is. At some point someone is going to be asked to pay their bill and the whole shaky edifice will collapse, when it does we will see something closer to a true cost of this technology being exposed this will reveal that it’s not in any way sustainable. Economies around the world will take decades to recover and public trust will be effectively nil during that time. Once economies have recoveredits highly likely that productive economies will be ascendant, the vast majority of the “western” nations will be unable to compete and will further devolve into oligarchies. None of this is worth being able to generate pictures of anthropomorphic molerats with big racks, or to create incorrect PowerPoint presentations, or to vibe code an application that works 70% of the time and deletes the contents of the storage medium it occupies the other 30%.
I wonder, are you an Evangelical Christian and / or Young Earth Creationist? Actually never mind, bored of you and a bit sad now.
Oh I see. You don’t know anything about AI beyond LLMs. No wonder you have this view. Well, I can see why you have your views now. To know about it, you’d have to use it properly… and since you won’t use it properly you can’t ever know about it. 🤷♂️
lol @ “western” that explains a lot.
Oh for fucks sake, “western” yes because the Western socio-political sphere is no longer a geographical divide, it’s ideological and financial.
My views are shaped by the technology, the economic ramifications, the environmental impacts, and the geopolitical environment.
If you had a compelling use case for the technology then you would have explained it, instead you present arguments from incredulity and strawmanning, ergo your argumentative capabilities are on a level with a flat earther or young earth creationist.
I have been using technologies that have been getting described as “AI” since fuzzy logic and self modifying code were the new hotness. Every decade there is a new push towards this sort of stuff, every decade we are willing to broaden the definition of intelligence to be more loosely defined. The difference is that this decade a bunch of rich people realised they can release slop and a bunch of credulous idiots will run around declaring it as the first horseman of the coming singularity.
I was going to say a few more things but I have already wasted FAR more time on you than is warranted.
Bye 👋
Like… what technologies. Show me your work. Show me how you use AI with modern tools and still fail.
I apologise, I engaged with you in good faith and you proved to be disingenuous and dishonest, I shouldn’t have made you feel special by continuing to respond to you.
I asked you to demonstrate a claim and you turned it into this big production and you still haven’t provided any reciepts.
If you want to talk with the adults in future you need to engage in good faith and back up your claims when asked. Then you get to ask follow up questions.
So when I said “Bye 👋” that was my polite way of saying “I’m done with you.” Sorry that went over your head, let’s see if I can draw a ban from this instance by removing the polite and speaking to you in a way I am certain you are more accustomed to:
**Fuck off, you corporate boot licking waste of skin.
Go make another TikTok video about how an atmosphere can’t exist next to a vacuum you mouth breathing cretin.
I am done with you! **
Is it diagnosed? 🤣