Excerpt:
“Even within the coding, it’s not working well,” said Smiley. “I’ll give you an example. Code can look right and pass the unit tests and still be wrong. The way you measure that is typically in benchmark tests. So a lot of these companies haven’t engaged in a proper feedback loop to see what the impact of AI coding is on the outcomes they care about. Lines of code, number of [pull requests], these are liabilities. These are not measures of engineering excellence.”
Measures of engineering excellence, said Smiley, include metrics like deployment frequency, lead time to production, change failure rate, mean time to restore, and incident severity. And we need a new set of metrics, he insists, to measure how AI affects engineering performance.
“We don’t know what those are yet,” he said.
One metric that might be helpful, he said, is measuring tokens burned to get to an approved pull request – a formally accepted change in software. That’s the kind of thing that needs to be assessed to determine whether AI helps an organization’s engineering practice.
To underscore the consequences of not having that kind of data, Smiley pointed to a recent attempt to rewrite SQLite in Rust using AI.
“It passed all the unit tests, the shape of the code looks right,” he said. It’s 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It’s a dumpster fire. Throw it away. All that money you spent on it is worthless."
All the optimism about using AI for coding, Smiley argues, comes from measuring the wrong things.
“Coding works if you measure lines of code and pull requests,” he said. “Coding does not work if you measure quality and team performance. There’s no evidence to suggest that that’s moving in a positive direction.”
This is a copium post. AI works very well if you know what you’re doing with it. I’ve proven it several times already.
Not often someone outright states that their comment is copium. Well done you!
Rather than making copium posts though maybe try not doing that. I’d respect you more and I’m sure a lot of others would feel the same.
I literally wrote “post”, not “comment”. Rather than being a dumb smart fuck, actually come up with something worth while to read next time.
Oh poor baby, I was being facetious.
Comments like yours don’t even rise to the level of bait. We get it, you have drunk the coolaid and feel that because you are incapable of the act of creation unassisted the whole world should burn.
I started to look through your post history before realising I was giving you WAY too much credit and saw that you have a shower thought about how it’s a good thing oil prices are rising, I assume out of some misplaced sense that this will lower demand (I honestly couldn’t be bothered reading your drivel). You don’t tackle demand for essentials by raising input costs, you tackle demand by reducing demand through market controls, alternative technologies, and innovation. Raising prices just facilitates faster wealth transfer to the top 0.01% from the bottom 99.99%.
Which is exactly what the “AI” industry is doing. But you are simply too ignorant to understand that. Therefore you get the facetious comments going forward, you poor misguided little capitalist bootlicking sheep. Oh and I know, you don’t think this was worth reading, but there are a bunch of other people who will be having a restrained chuckle and being grateful that there was someone else who had a big enough gap in their day to slap your nose with the metaphorical rolled up newspaper and send you back to your paddock.
Bye 👋
Certainly well enough that jobs have been lost and will continue to be. Increasing the number of people applying to the smaller number of jobs that do still exist.
This will only get worse.
For people looking for jobs it will get more difficult, competition will continue to rise, and anyone not well versed in using AI will be left behind.