Two-and-a-half years after ChatGPT launched, LLM capabilities are more impressive than ever. OpenAI’s o3 can scour many documents to answer questions that I wouldn’t have bothered asking otherwise. Sonnet 4 can one-shot (some) features in Monumental’s codebase.

One trend is making me uneasy: non-technical users crediting LLMs with mystical powers.

An example from my LinkedIn feed:

This ChatGPT user asked 4o to compare her usage to other professionals. ChatGPT then gave a statistical breakdown of the prompts issued by the user and compared it to other users.

A couple of problems:

• 4o (I believe) lacks access to the user’s full prompt history. At most it has some memories - not enough to summarise or count queries.
• 4o (definitely) does not have access to data about other users’ queries.

Software engineers find this obvious. We know how you would implement support for this query. And we intuit that OpenAI haven’t done that yet.

But a normal user’s mental model is very different. They see an oracle which gives mostly useful and correct answers to questions. There is no clear pattern to explain when the oracle gives bad answers. So they average: assume every answer is probably correct until proven otherwise.

I don’t mean to pick on this one LinkedIn user: they are illustrative of a wider phenomenon. Search for "I asked ChatGPT" and many examples will avail themselves (1, 2, 3)

Does this matter?

Bad things happen when common understanding of a system diverges from the actual reality of a system.

Some examples:

• Advanced driver assistance systems
  • Observed system: Tesla Autopilot can drive my car
  • Actual system: Autopilot can drive the car, most of the time, under certain conditions
  • Result: user sleeps at the wheel, crashes
• VPNs
  • Observed system: this keeps my data safe on the internet
  • Actual system: this avoids leaking a very small and specific subset of data
  • Result: your threat model was wrong, the government arrests you or the VPN has logs
• Google
  • Observed system: Google finds me safe and useful links related to my query
  • Actual system: some of the links are paid ads, and anyone can buy them
  • Result: phishing

When the LLM is a mystery

Just a few months after the launch of ChatGPT, a university instructor pasted submitted essays into ChatGPT and asked if it had wrote them.

• Mental model: ChatGPT is a singleton, persistent entity
• True model: ChatGPT is a program that runs independently for each request

A more worrying phenomenon has been termed ‘ChatGPT-induced psychosis’. There are several anecdotes on Reddit describing how people have been sucked into harmful delusions of a mystical or spiritual nature.

• Mental model: ChatGPT confidently affirms my spirituality queries and so is correct
• True model: ChatGPT is designed to be sycophantic and keep the conversation going

Understanding how an LLM works should inoculate against these delusions.

It is still unknown whether user understanding of LLMs will improve over time, like it did for older technologies. Is the conversational interface so dazzling that people will never see them as ‘just ordinary software’?

Fixing the mental model

Can we help users form a better mental model of what the LLM is doing?

OpenAI’s token effort (writing ‘ChatGPT can make mistakes’ on every page) appears insufficient.

Query context is important. Many users do not realise what data the LLM does (and does not) have access to. For example:

• training data: lots of answers are fuzzily retrieved from the training inputs
• system prompt: information about the user, today’s date, etc.
• memories: facts learned and stored (often silently) from past queries
• tools: data from a vector database or internet searches

If the user can see whether the input includes ‘statistical analysis of ChatGPT queries made by sales professionals’, they can actually know whether it’s able to answer that question.

What is today’s user allowed to know?

• training data: kept secret to avoid triggering the wrath of copyright holders. Also to maintain an edge over other model providers
• system prompt: hidden from the user. Models refuse to print it out. Maybe they will ban you for asking.
• memories: hidden away in a settings panel
• tools: ChatGPT will tell you what tools it has, but it isn’t in the UI.

No wonder users can’t judge which questions the LLM can and can’t answer.

Model providers, you need to do a better job. Reveal your context; demystify your oracles.