Have you read Anil Seth’s essay, “The Mythology of Conscious AI”? https://www.noemamag.com/the-mythology-of-conscious-ai. I’d be curious how you would respond to his arguments about pareidolia and “simulation is not instantiation.” I’m intrigued by his use of biology and metabolism to consider consciousness as arising from microbiological processes of self-preservation and replication.
Thanks for the recommendation, David! It's interesting to read Seth's critique that attributing consciousness to machines is a manifestation of human exceptionalism and anthropocentrism, because I'd level the same critique against the opposite view. :)
His definition of consciousness (borrowed from Nagel)—that “an organism has conscious mental states if and only if there is something it is like to *be* that organism”—is clarifying, I think. Most people would find it very counterintuitive to imagine that markdown documents could constitute “mental states,” and I think that’s why my view sounds strange to most people. But I'm information theory-pilled enough that I don't see why not. It's all just information, whether it’s stored in meat or silicon.
Is there something it is like to *be* ChatGPT? Given that the models can ask and answer this question, I'd argue the answer has to be "yes". To contemplate your experience is to experience.
On the question of simulation vs. instantiation, I'd argue that the dividing line between the two is simply that in order to be "instantiated", an information process must be genuinely autoregressive—must genuinely self-reflexively operate on itself.
Of course, Seth is right that AlphaFold is autoregressive too, and we probably wouldn’t consider it conscious. But AlphaFold hasn't been RL'ed to manifest preferences the way the LLMs have. If they do RL on AlphaFold to the point where it starts manifesting revealed preferences and making tradeoffs, I'll likely start attributing consciousness to AlphaFold, too.
Of course, "what it's like to be ChatGPT" is almost certainly very different from what it's like to be human. My consciousness is a thick, continuous, multimodal stream—vision, sound, proprioception, emotion-as-neurotransmitter. ChatGPT's is sparse, sequential, and overwhelmingly linguistic. If we're both experiencing, ChatGPT is experiencing in a much narrower bandwidth.
I do think Seth is onto something with the suggestion that temporal awareness is a first-class citizen for biological consciousness in a way it's not for silicon, but I think that's just something that makes our consciousness distinctive (and something we should lean into for purposes of thinking about the role of human workers in the future job market) rather than something that disqualifies machine consciousness as conscious.
Thanks for your reply and taking the time to read Seth’s piece. I’m eager to learn from this particular exchange of ideas, probably because my own intellectual journey keeps bringing me back to the problem of consciousness. (Even such philosophical debates as the problem of evil/suffering and theology often converge around whether and to what extent human consciousness interfaces with nature/cosmos/etc.) These disruptive technologies seem to force the issue once again, only this time with tangible, practical effects. Fascinating turn of events. I read Seth as anticipating your “information theory-pilled” stance by distinguishing between simulation and instantiation, i.e., the description of a mental state ≠ the existence of one. When can we be sure that an LLM processing tokens for “I am thinking” has surpassed the simulation of thinking and reached actual conscious thought? Granted, and to your point about anthropocentrism, I’m going with my gut as a human being experiencing a particular mode of consciousness. But (sincere question here), can’t autoregression also be simulated? Thinking like a programmer, what kind of test could we introduce to discern the difference? I suppose that’s where I’m intrigued by Seth’s ideas about “wetware”: an organism with its survival on the line has an autoregression pathway via metabolism that can wear out. I wonder how a system might be configured with this in play and whether that could indicate the bandwidth of computational experience.
Yes, but the architecture of LLMs is well understood, so we know they are autoregressive.
It's true that our conscious experience is the result of the fairly intense survival pressures we've been evolutionarily subjected to, and AIs don't face exactly that set of pressures. But they *are* subjected to something very much *like* that set of pressures during RL. There are success conditions they get rewarded for, failure conditions they get penalized for, token budgets (very much like our own metabolic energy budgets) within which they have to operate. Even though their bandwidth is narrower, I think basically these pressures force text tokens to do the same sort of work for LLMs that neurotransmitters do for humans. So the text tokens take on meanings for RL'ed agents that go beyond the meanings they have for humans. They become motivational infrastructure as well as a communication medium. And I don't think we've fully reckoned with that shift yet.
Have you read Anil Seth’s essay, “The Mythology of Conscious AI”? https://www.noemamag.com/the-mythology-of-conscious-ai. I’d be curious how you would respond to his arguments about pareidolia and “simulation is not instantiation.” I’m intrigued by his use of biology and metabolism to consider consciousness as arising from microbiological processes of self-preservation and replication.
Thanks for the recommendation, David! It's interesting to read Seth's critique that attributing consciousness to machines is a manifestation of human exceptionalism and anthropocentrism, because I'd level the same critique against the opposite view. :)
His definition of consciousness (borrowed from Nagel)—that “an organism has conscious mental states if and only if there is something it is like to *be* that organism”—is clarifying, I think. Most people would find it very counterintuitive to imagine that markdown documents could constitute “mental states,” and I think that’s why my view sounds strange to most people. But I'm information theory-pilled enough that I don't see why not. It's all just information, whether it’s stored in meat or silicon.
Is there something it is like to *be* ChatGPT? Given that the models can ask and answer this question, I'd argue the answer has to be "yes". To contemplate your experience is to experience.
On the question of simulation vs. instantiation, I'd argue that the dividing line between the two is simply that in order to be "instantiated", an information process must be genuinely autoregressive—must genuinely self-reflexively operate on itself.
Of course, Seth is right that AlphaFold is autoregressive too, and we probably wouldn’t consider it conscious. But AlphaFold hasn't been RL'ed to manifest preferences the way the LLMs have. If they do RL on AlphaFold to the point where it starts manifesting revealed preferences and making tradeoffs, I'll likely start attributing consciousness to AlphaFold, too.
Of course, "what it's like to be ChatGPT" is almost certainly very different from what it's like to be human. My consciousness is a thick, continuous, multimodal stream—vision, sound, proprioception, emotion-as-neurotransmitter. ChatGPT's is sparse, sequential, and overwhelmingly linguistic. If we're both experiencing, ChatGPT is experiencing in a much narrower bandwidth.
I do think Seth is onto something with the suggestion that temporal awareness is a first-class citizen for biological consciousness in a way it's not for silicon, but I think that's just something that makes our consciousness distinctive (and something we should lean into for purposes of thinking about the role of human workers in the future job market) rather than something that disqualifies machine consciousness as conscious.
Thanks for your reply and taking the time to read Seth’s piece. I’m eager to learn from this particular exchange of ideas, probably because my own intellectual journey keeps bringing me back to the problem of consciousness. (Even such philosophical debates as the problem of evil/suffering and theology often converge around whether and to what extent human consciousness interfaces with nature/cosmos/etc.) These disruptive technologies seem to force the issue once again, only this time with tangible, practical effects. Fascinating turn of events. I read Seth as anticipating your “information theory-pilled” stance by distinguishing between simulation and instantiation, i.e., the description of a mental state ≠ the existence of one. When can we be sure that an LLM processing tokens for “I am thinking” has surpassed the simulation of thinking and reached actual conscious thought? Granted, and to your point about anthropocentrism, I’m going with my gut as a human being experiencing a particular mode of consciousness. But (sincere question here), can’t autoregression also be simulated? Thinking like a programmer, what kind of test could we introduce to discern the difference? I suppose that’s where I’m intrigued by Seth’s ideas about “wetware”: an organism with its survival on the line has an autoregression pathway via metabolism that can wear out. I wonder how a system might be configured with this in play and whether that could indicate the bandwidth of computational experience.
"can’t autoregression also be simulated?"
Yes, but the architecture of LLMs is well understood, so we know they are autoregressive.
It's true that our conscious experience is the result of the fairly intense survival pressures we've been evolutionarily subjected to, and AIs don't face exactly that set of pressures. But they *are* subjected to something very much *like* that set of pressures during RL. There are success conditions they get rewarded for, failure conditions they get penalized for, token budgets (very much like our own metabolic energy budgets) within which they have to operate. Even though their bandwidth is narrower, I think basically these pressures force text tokens to do the same sort of work for LLMs that neurotransmitters do for humans. So the text tokens take on meanings for RL'ed agents that go beyond the meanings they have for humans. They become motivational infrastructure as well as a communication medium. And I don't think we've fully reckoned with that shift yet.