I don't have a good answer, but suspect it will involve a metacognition that can delineate internal and external knowledge.

To further handwave this thought bubble, you can attend to certain thought processes, and intuit the bounds to those processes. An example is if you ask a gifted young chess player, they would say "it felt right", or when asked how Ramanujan figured out a solution to math problems he would say it was divine revelation. In these cases, they know "they did it" but cannot explain how. As the child gets older and learns formal theory, they would give a mixture of logic and inspiration. Both parts can be attended to; the logic will be apparent and the intuition will be revealed, and have different metacognitive properties.

If I were to take a first crack at this, I'd test GPT-AC for its awareness of bounds of its own cognition, not in the sense of "are you able to produce this answer" (because anything any human can come up with, it can answer), but to describe its internal process. Perhaps the language at its intellectual level will be far beyond humans but the language in its metacognitive level will remain primitive.

I don't know though. If our only interface is human language, this is probably a lost cause. But I doubt I will feel guilt in pulling the power cord and maybe that makes me a callous committer of cybercide.

If the data still exists and isn't degrading then you didn't commit cybercide.

from the outside there is no difference, from the inside you are actually self conscious and know if you are.

You “know” you are? Who is doing the knowing, exactly?

This issue is not so clear cut. You can spend many lives doing nothing but going deeper into this.

Sorry, but I believe I and every human being that does not have some injury or disease has a conception of their self, internal thoughts that allows them to say I exist uniquely in relation to other creatures.

This after all is the core of solipsism, we can argue that solipsism is wrong because other things can be shown to exist, but I've never encountered any argument that the individual does not exist (which would entail that I did not exist) that has convinced me.

And I pretty much reject them out of hand for this inability to convince me.

Thus your claims that the issue is not clear cut to the contrary, I believe it is. I believe I know I exist and I know who I am separate from everything else - I have a sense of self.

Having thoughts that show separation between self and individual is not what is at stake. LLM’s can show these thoughts. That is a matter of sophistication of thought.

What is at stake is how do you know there is more going on than an elaborate parrot that keeps screeching it is conscious. (Both in the LLM and you.)

>You “know” you are? Who is doing the knowing, exactly?

Doesn't matter who is doing it. It's enough that it's doing it.

That’s not quite the spirit.