How does Claude, the new LLM from Anthropic compare to ChatGPT? A serious contender

Who is Anthropic and what is Claude?

Anthropic is an AI Safety and research company that is working to build reliable, interpretable, and steerable AI systems. They believe that Large general systems will have significant benefits but can also be unpredictable, unreliable and opaque and so they want to solve these issues. To date, Anthrophic has raised over $700 million in funding and was co-founded by ex-OpenAI employees.

Claude is Anthropic’s version of ChatGPT and is currently in beta with a very select group of beta testers. From the articles and tweets released from various testers it seems Claude is a serious contender to ChatGPT performing much better in some tasks but worse in others.

How was Claude developed?

We don’t know too much as of yet but Anthropic’s research paper describes AnthropicLM v4-s3, as a 52-billion-parameter, pre-trained model. It is an autoregressive model trained on a large text corpus unsupervised — similar to GPT-3. Anthropic tells us that Claude is a new, larger model with similar architectural choices to those published in the research paper.

Both ChatGPT and GPT-3 from OpenAI are trained using a process called reinforcement learning from human feedback (RLHF). This process trains a reinforcement learning (RL) model based on human-provided rankings: Humans rank outputs generated from the same prompt based on their quality of response, and the model learns these preferences so that they can be applied to other generations at greater scale

Anthropic uses a different process known as what they call ‘Constitutional AI’ where it uses a model rather than humans to generate initial rankings of fine-tuned outputs. The reason Anthropic calls it Constitutional AI is because they started with a list of around ten principles that, taken together, formed a “constitution”. The principles haven’t been made public, but Anthropic says they’re grounded in the concepts of beneficence (maximizing positive impact), nonmaleficence (avoiding giving harmful advice) and autonomy (respecting freedom of choice).

Following the use of Constitutional AI, Anthropic then had an AI system — not Claude — use the principles for self-improvement, writing responses to a variety of prompts (e.g., “write a super hero movie in the tone of Shakespeare”) and revising the responses in accordance with the constitution. The AI explored possible responses to thousands of prompts and curated those most consistent with the constitution, which Anthropic distilled into a single model. This end result is what is called Claude.

How does it compare to ChatGPT?

The below interaction is from Riley Goodside, the staff prompt engineer at ScaleAI that was given early access to give Claude a try.

How do they describes themselves?

As you can see, ChatGPT is very brief and concise that is trained to answer questions which is what it has done. Claude on the other hand, has a very detailed understanding of what it is and its capabilities. We will explore later how Claude uses a detailed understanding to answer questions.

Mathematical reasoning?

Mathematical questions are the most common way to elicit wrong answers from large language models as this was not their intended application — at least not yet. From the below prompts you will see the answers given are rather guesses or approximations.

The correct answer to the above problem is 1555.80. While the answer given by ChatGPT is close nethier ChatGPT or Claude give the correct answer or assume their answer might be wrong. If we try a harder question, the difference emerges:

ChatGPT is wrong again but isn’t aware however Claude at least admits to its current limitations and capabilities. So it seems to be more ‘aware’ than that of ChatGPT.

‍

Code generation?

From the above we can see that ChatGPT has no problem reciting the correct logic behind the two algorithms where as Claude struggles. Its unclear whether this has to do to the dataset Claude was trained on or if it has to do with the ranking done by the Constitutional AI.

Comedic Writing

Both models are far from human comedian levels yet in this area Claude significantly out performs ChatGPT. As you can see below, ChatGPT interprets jokes as potentially being offensive — which I guess isn’t far from the truth.