His argument is that generating statistically plausible text should not be treated as proof of consciousness. The reason why embodiment tends to be brought up is because it creates a basis for a system to have self awareness. You end up with a feedback loop where the system has to model the world and itself within it, and taking actions feeds back into the system so it has to be able to recognize itself as it interacts with its environment. Ted Chiang wrote a great novella where he discusses this idea https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Objects
Of course, it is possible that you could have some other type of feedback loop that produces self awareness and consciousness, but from what we understand of how LLMs work, it seems highly unlikely that statistical token generation is sufficient for that.
I do agree that he fails to really make the argument for why a disembodies intelligence could not be conscious. In my opinion, the strongest part of the article is at the end where he shows how the whole constitution kabuki theatre that Anthropic came up with clearly wouldn’t afford any protections to an entity that was conscious, so they don’t really believe what they’re saying.
His argument is that generating statistically plausible text should not be treated as proof of consciousness. The reason why embodiment tends to be brought up is because it creates a basis for a system to have self awareness. You end up with a feedback loop where the system has to model the world and itself within it, and taking actions feeds back into the system so it has to be able to recognize itself as it interacts with its environment. Ted Chiang wrote a great novella where he discusses this idea https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Objects
Of course, it is possible that you could have some other type of feedback loop that produces self awareness and consciousness, but from what we understand of how LLMs work, it seems highly unlikely that statistical token generation is sufficient for that.
I do agree that he fails to really make the argument for why a disembodies intelligence could not be conscious. In my opinion, the strongest part of the article is at the end where he shows how the whole constitution kabuki theatre that Anthropic came up with clearly wouldn’t afford any protections to an entity that was conscious, so they don’t really believe what they’re saying.