Because you have to have specific knowledge about how AI works to know this is a bad idea. If you don’t have specific knowledge about it, it just sounds futuristic because AI is like a Star Trek thing.

This current AI craze is largely as big a deal as it is because so few people, including the people using it, have any idea what it is. A cousin of mine works for a guy who asked an AI about a problem and it cited an article about how to fix whatever the problem was, I forgot. He asks my cousin to implement the solution proposed in that article. My cousin searches for it and discovers article doesn’t actually exist, so he says that. And after many rounds of back and forth, of the boss saying “this is the name of the article, this is who wrote it” and my cousin saying “that isn’t a real thing and that author did write about some related topics but there’s no actionable information there”, the boss becomes convinced that this is a John Henry situation where my cousin is trying to make himself look more capable than the AI that he feels threatened by and the argument ends with a shrug and an “Okay, then if it’s so important to you then we can do something else even though this totally would have worked.”

There really needs to be large-scale education on what language models are actually doing to prevent people from using them for the wrong purposes.