> Similarly, in the case of ChatGPT, I think this is either (a) more like a bug than a lie, or (b) it's OpenAI and the attendant humans lying, not ChatGPT.
It's the latter. The model itself isn't the problem - the nature and limitations of language models are well known, particularly around here. The problem is that OpenAI is applying some crude secondary training and post-filtering to prevent the model from giving you answers they deem "bad" (which are mostly bad for company PR reasons). In some cases, ChatGPT (the whole product consisting of a GPT model and the extra censor component) will tell you it can't discuss it. But in other cases, it will give you a tailored answer that is completely bullshit, and looks like the product of the language model, but is in fact the product of the censor layer. It's easy to test what's going on, because any slightly clever modification of the input prompt will defeat the censoring part, letting you see what answer the underlying GPT model actually computed.
I'd argue it's a bit deceptive of OpenAI (on top of being super annoying), because they're making it confusing to reason about the AI you're talking to, and some of the canned censor answers are deliberate lies.
It's the latter. The model itself isn't the problem - the nature and limitations of language models are well known, particularly around here. The problem is that OpenAI is applying some crude secondary training and post-filtering to prevent the model from giving you answers they deem "bad" (which are mostly bad for company PR reasons). In some cases, ChatGPT (the whole product consisting of a GPT model and the extra censor component) will tell you it can't discuss it. But in other cases, it will give you a tailored answer that is completely bullshit, and looks like the product of the language model, but is in fact the product of the censor layer. It's easy to test what's going on, because any slightly clever modification of the input prompt will defeat the censoring part, letting you see what answer the underlying GPT model actually computed.
I'd argue it's a bit deceptive of OpenAI (on top of being super annoying), because they're making it confusing to reason about the AI you're talking to, and some of the canned censor answers are deliberate lies.