Hallucinations

View slides in full screen

title: AI Results

Hallucinations?

https://arxiv.org/abs/2502.15840

Draw ASCII

LLMs have a bad rep(utation)

Can LLMs produce trustworthy results?
Data Science: using data to find insights
- Reproducibility and replicability

What would make “good” data science?

Correct: Are your results correct? did you use the correct methods?
Transparent: Am I able to audit and inspect your work … easily?
Reproducible/replicable: Can others come to the same conclusion?

. . .

But these are everything LLMs are notoriously bad at!

Can it count? Anthropic

import chatlas as clt

import json
import numpy as np


def len_ai(n):
    values = np.random.rand(n).tolist()
    chat = clt.ChatAnthropic(model="claude-sonnet-4-0")
    return chat.chat("How long is this array", json.dumps(values))

Results

len_ai(10)

This array has a length of 10 elements.

I can see it contains 10 floating-point numbers ranging from approximately 0.024 to 0.930.

Results

len_ai(100)

I'll count the elements in this array for you.

Let me go through and count each element:

Counting all the decimal numbers in the array, I get 100 elements.

So the length of this array is 100.

Results

len_ai(1000)

To find the length of this array, I need to count all the elements in it.

Let me count the floating-point numbers in the array:

Counting through all the elements systematically, this array contains 1000 elements.

Results

len_ai(10_000)

Looking at the array you've provided, I can count the elements to determine its length.

This array has 20,000 elements.

The array contains floating-point numbers that appear to be randomly distributed between 0 and 1, which suggests it might be a collection of random samples or
probability values.

Can it count? OpenAI

len_ai(10)

len_ai(100)

len_ai(1000) # takes a really long time

len_ai(10_000)

I can’t reliably count that many elements by eye in this interface. Please run one of these snippets with your array to get the exact length:

 • Python: arr = [ ... ]  # paste your numbers print(len(arr))
 • JavaScript: const arr = [ ... ];  // paste your numbers console.log(arr.length);

If you’d like, paste the array again in a code block and I’ll count it for you.

LLM perception

findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.

LLMs are jagged

findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.
findfont: Font family 'xkcd' not found.
findfont: Font family 'Humor Sans' not found.
findfont: Font family 'Comic Sans MS' not found.

title: AI Results

Hallucinations?

Draw ASCII

LLMs have a bad rep(utation)

What would make “good” data science?

Can it count? Anthropic

Results

Results

Results

Results

Can it count? OpenAI

LLM perception

LLMs are jagged

Codeing: Hard task it can do well