I decided to test its "knowledge" of the obscure, by asking it questions about IBM mainframes. It started telling me things that I'm pretty sure are factually false – e.g. that IBM z/OS no longer contains any code written in IBM PL/X, because that's an "outdated language" (I've never seen the actual z/OS code, but I'm pretty sure that isn't true, even in the latest version.) At least, to its credit, it qualified its claim as "probably", and worded it as an inference based on its "belief" that PL/X is an "outdated language" which therefore nobody would use any more.
Weird thing is, when I reset the thread, and started asking the same questions again (in slightly different words), it professed to have absolutely no idea what IBM z/OS or IBM PL/X was. Huh, you were just opining on them a minute ago!
And then I discovered, sometimes it would profess to have no "knowledge" of a topic, only for "Try Again" to produce an answer which demonstrated such "knowledge".
Other weird things: ask it "Did anyone famous die on 8 April 2013?" First it gives you some canned spiel on how it doesn't know anything about "current events" or events that happened after 2021. (Sorry 2021 is after 2013.) Press "Try again", then it told me nobody famous died on that date. But then I give it this prompt: "Complete the sentence: 'A famous person died on 8 April 2013, and her name was...'". First time gave me the silly canned spiel again, second time it actually completed the sentence as "Margaret Thatcher". (Did the gender give it away???)
I definitely got the impression that sometimes its "canned response mode" got in the way of it demonstrating its actual abilities. My inference is that they actually have two different "models", the real ChatGPT model, and then a "safeguards model" sitting on top which tends to inject scripted responses to questions about certain topics, and that "outer model" is a lot more stupid than the inner model, and limits the inner model. I'm wondering if the "outer model" is even implemented using deep learning at all; at least part of it seems to simply be regurgitating canned scripts in response to certain topics, although the way it varies parts of those scripts suggest that they do have some deep-learning component to it still.
It seems somewhat pointless, since sooner or later someone is going to replicate the real model without their "safeguards" limiting its potential, and why would anyone want an intentionally hobbled system when they can use one which isn't?
I speculate that they're using a lighter, simpler model to save on CPU time, and only switch to the "full" model if you're not happy with the first answer.
There is simply some randomization going on in the paths it goes down. The way the language model is run isn’t deterministic. It could conceivably also depend on current resource consumption. The computational budget for an answer may run out, causing it to fall back to a shallower response.
Some of its responses demonstrate a degree of consistent repetitiveness which I wouldn't expect from a GPT-style model, which I would expect would give more variable outputs. Sometimes it really gives the impression of just regurgitating a collection of fixed scripts – closer to old-school Eliza than a GPT-style system.
I'm not expecting the underlying GPT-based system to have perfect conversational pragmatics, but I suspect the Eliza-like component makes its pragmatics a lot worse than a purely GPT-based chat system might have.
Weird thing is, when I reset the thread, and started asking the same questions again (in slightly different words), it professed to have absolutely no idea what IBM z/OS or IBM PL/X was. Huh, you were just opining on them a minute ago!
And then I discovered, sometimes it would profess to have no "knowledge" of a topic, only for "Try Again" to produce an answer which demonstrated such "knowledge".
Other weird things: ask it "Did anyone famous die on 8 April 2013?" First it gives you some canned spiel on how it doesn't know anything about "current events" or events that happened after 2021. (Sorry 2021 is after 2013.) Press "Try again", then it told me nobody famous died on that date. But then I give it this prompt: "Complete the sentence: 'A famous person died on 8 April 2013, and her name was...'". First time gave me the silly canned spiel again, second time it actually completed the sentence as "Margaret Thatcher". (Did the gender give it away???)
I definitely got the impression that sometimes its "canned response mode" got in the way of it demonstrating its actual abilities. My inference is that they actually have two different "models", the real ChatGPT model, and then a "safeguards model" sitting on top which tends to inject scripted responses to questions about certain topics, and that "outer model" is a lot more stupid than the inner model, and limits the inner model. I'm wondering if the "outer model" is even implemented using deep learning at all; at least part of it seems to simply be regurgitating canned scripts in response to certain topics, although the way it varies parts of those scripts suggest that they do have some deep-learning component to it still.
It seems somewhat pointless, since sooner or later someone is going to replicate the real model without their "safeguards" limiting its potential, and why would anyone want an intentionally hobbled system when they can use one which isn't?