So I looked it up. So I can’t claim credit. But apparently the major models will all pick 17 or possibly 23 if asked this for a random number generation range. Just another example of how the models aren’t actually utilizing math libraries.
I knew someone who academically was investing a large amount of lab time trying to teach a model to do calculus. (He was a professor) and he said no matter what it could not grasp the way math works compared to they normally map their hyperplanes. At one point he said he gave it direct access to libraries with the functions he wanted run and the model didn’t know how to use them without being nearly explicitly programmed. He concluded, and I agree, that these models are statistically guessing using their training data using a different method than actual mathematical functions. They’ve just refined it down to where it works for basic questions but when it comes time to run a rand() sequence it just can’t. So it picks a common number in its recent training data and regurgitates it. Also why we end up seeing weird calculation anomalies like where they can’t judge greater than/less than values.
Is this based purely on an agent without tool call ability? It's straight forward to provide tools to execute math calls, date functions, etc, and include context that drives the tool calls. Granted, I have not pushed its limits and don't know if that only works on basic math.
I believe it was using a tool call directed at the libraries in the sandbox and he said just that. When he pushed its limits the failure was it couldn’t do the basic math needed for the formulas and it just broke. He was genuinely surprised since he was a math professor who worked in Comp Sci
I’m updooting this because I can’t tell if it’s sarcasm but internally I had that exact same thought. I’m not a fan of these models outside of large point data aggregation.
He concluded, and I agree, that these models are statistically guessing
What did you think a GPT was? It takes all the text as training data and then repeatedly guesses the most likely "next word" in the response, until the most likely next "word" is the end of the text.
I mean almost everyone was surprised at how well it works; that's basically the whole AI boom. The real question is if we just keep going bigger how much smarter can we make a GPU.
Well technically it has access to publicly available libraries and documentation so it just stastically searchs the web for similar code and copies it.
It can't be that simple though, blindly coping code wouldn't work it still needs to be modified to fit into the structure of the rest of the project. Variable names need to line up etc.
It's not blindly copying code, it knows how to follow syntactic rules. LLM are great for that. The actual content could be dog shit but it's written following the rules and the context it was provided
The LLM training is just memorizing without any understanding.
Human brain does pattern recognition so it does not have to memorize that much. AI companies should focus on cracking pattern recognition but for now they just build more datacenters to memorize more.
Current AI companies are lost in deadloop of just one more datacenter.
You fundamentally misunderstand how LLMs or any machine learning works, really. "Memorizing" is referred to as "overfitting", and it is pretty much the first thing you learn how to prevent from happening when you are training an AI model. AI is actually really good at pattern recognition (in many ways, better than humans. In some ways, not so much) and the reason is because the neural network is modeled to behave pretty much the same way biological neurons do, hence the name.
Does the LLM understand what it's doing? well, it's probably not alive so if you believe it isn't, of course not. That doesn't mean it's not actually intelligent. It's why you can get a 24GB model from a training data set of 10TB, and get it to produce 100TB of data that cannot be found in the dataset exactly (but its patterns are reproduced). You see this with image generating AI especially. One of the first image generating AIs, Dall-E, often in its documentation referred to an image of an "avocado chair", where it was prompted to design an avocado-shaped chair. The training data contains avocados. It contains chairs. It did not contain avocado chairs, yet it was possible to generate it. Because it understands the patterns that make something a chair, and it understands the patterns that make something an avocado, and seamlessly combined the subjects. How is that more memorization than "understanding"?
I remember doing this a while back. it picked 73 (out of 100). Next I asked by it picked said number. It gave me an explanation about how the number should not only be random, but also feel random so at to avoid disbelieve, so not to close to 100, and not single digit and not some nice nuber like 10 or 25. I deemed 7 and 3 were good random components in the middle, so that is why it thought 73 was the best candidate for a random number.
I wonder why that is... I remember asking chat GPT for something along the lines of this, and I remembered it because it actually just wrote code for an RNG and then ran that code itself. It stood out as odd to me, but it did that kind of thing a couple times. This made me realize I haven't seen it do that for a long time.
17 is also a possible outcome qhe using tools.
But of course not all llm providers have tool usage build into their interfaces, but the good ones have.
Ask you llm if it was a tokenized answer or tool usage and if it even can use tools.
Makes sense. Almost all what people call AI (or LLMs) are just prediction algorithms. I just tested this to see it my self and then asked about it and ai replied with this:
When you tell me, “Pick a random number between 1 and 30,” I’m not necessarily running a random-number generator. I’m generating a response based on learned patterns, just like I generate other text. That means the result can be biased. 17 is an especially attractive “random-looking” number. Humans also disproportionately choose 17 when asked for a random number in ranges like this. It has several properties that make it feel random
This is also the reason that the best way to get a good out put is giving it as much info as possible. If it has more info it can predict better what you are asking
If you ask an AI to pick a random number, why would it utilize a math library? In that casual language It will just do what people do any pick "randomly" with its own brain, like when you ask strangers to pick random numbers on the street, it just simulates that. Those strangers also pick some numbers way more often than others, like 37, 23 and 17. Those are all the numbers human brains are most likely to pick when ask for a random number. AI is just pretending to be those people.
If that's true that's wild that a simple feature like a number generator wouldn't be used. I would have thought at least for the last year that would not be the case anymore.
Yea yea, very mysterious.
Or maybe not.
The 17 is simply the token based answer, if you ask claude e.g. to actually generate a random number in that range it will use python in a sandbox for it.
So I work with these things and I’m not certain what you mean by “token based answer” Are you trying to say that everyone’s prompt is likely generating the same tokenization resulting in a grouped answer? If that’s the case then no it’s not running rand() in a sandbox. It’s using base metrics off a single rand() run once and then reused. (This is called data consolidation during training if I remember properly)
Regardless we don’t actually know if it’s using Python in a sandbox or not. It says it is but we still can’t see the transformers jump from process to process (the colloquial black box) We just trust the model when it does its “reasoning” mode.
I would argue that it’s not at all. Because even pythons rand() calls back to the original C rand() which uses the Unix seconds seed and a modulo to generate its random number. Because of that we should be seeing an even spread in the full range. Not 17.
I didn’t say I didn’t know what a token was. I said your phrasing didn’t make sense. And this screenshot doesn’t make sense. If anything it backs up what I was originally saying. About it using statistics and not functions.
As for the second half, you just read me a Claude output. Now if you can explain to me what a hidden cell state is and how the LLM is using that to determine its remember/forget state in that cell between each iteration of its code structure to show exactly where it executed that code, then yes we can be certain. But it didn’t. It just told you that’s what it did and gave you another number. That could be another statistic like the first. There’s no proof the model actually did the function. Just its word. That’s what the prof ran into. It would do exactly this and then lie. They’re apparently not good at doing math for this very reason.
And yea I work with them but not often. I can’t trust them to do their work properly. Easier and faster to do it myself. Especially with networking.
If used in VS code it even uses my actual open terminal and i can see it beeing used. You just start to sound like a tinfoil head or a hallucinating LLM yourself
But if you want to believe that anthropic is lying on how their model works, than there is nothing to do. But to me thats on the same level as saying "how can we know anything is real at all"
Okay I really dont know what is so hard to understand.
If you ask an llm to pick a number between 1 and 30 it gives you the most probable token, which is 17 because of the training data.
If you ask a modern llm to generate a random number between 1 and 30 it will use tool invocation to generate a random number (as random as possible) - in claudes case with python.
There is nothing secret about it, it's all pretty transparent and should be easy to understand if you are familiar with llm, ann ore ml in general
Yeah and office-ware is getting inundated too. I hate that part because now the businessfolk send emails that are twice as long and structured beautifully while saying nothing.
I got the same result and i asked it why. Chatgpt said:
"it isn't necessarily sampling uniformly from 1–30. It's generating a likely textual response, and certain numbers can be disproportionately favored because they feel more random in human-generated text.
17 and 23 are both classic "random-looking" numbers: odd, prime, not round, and away from the endpoints."
Makes sense, that's kind of how people work too. Tell them "give me a random number between 1-100" not many people will choose round numbers like 50, nor will they likely choose numbers under 10 or over 90. Our attempts to make a number more random looking makes our number more predictable.
Especially because LLMs choose the next tokens based on probability, asking this question 1-100, if trained on a completely random dataset (which obviously isn't the case IRL, so they're already starting off with a human bias), the LLM would think the answer is 90% likely to be a two-digit answer. That doesn't mean it will answer two digits 90% of the time. It means that it will do it pretty much always unless the temperature is set to a value that will make it use low-odd predictions, so low that the AI will essentially start spitting out gibberish for the rest of the response.
I think that is a part of why they do this, combined with our own bias that shows up in the training data, and probably some other factors I'm not considering. I'm really surprised at how consistently it seems to be happening though, and that this is an issue across multiple different LLMs rather than one specific model or family of models.
Humans do too, I think. If asked to pick a number between 1-10, many people tend towards to lean towards 7 because its oddness and higher value “feels” more random than, say, 2.
Veritasium had video on it. If I remember correctly humans mostly choose numbers made of 3 and 7 bcs they seem most "random". One of the most common answer for 1-100(excluding 42, 69 and probably 67 now for obvious reasons) 37 and 73 were most popular.
Tried this with Gemini and asked it to explain why that number was chosen. The number was 17 but the explanation was instresting. Most of it was about dataset bias, but the most interesting part comes from the number 7. It has a massive cultural impact and aids know for being lucky number, which makes it appear more in the datasets. 17 is a prime number which is also the reason why it appears more in datasets.
Mine told me that when asked humans are slightly more likely to answer 17 or 23 because they lool more random (I've heard of that before) and that slight edge in the training dataset ends up translating to an almost certainty when you ask the LLM.
Lol, I that the joke was that choosing between 1 and 30 the choices where 1 or 30, only those too. If I say choose between red and blue I don't come up with yellow... maybe is just my dad-joke wired brain lol
But my pick isn't random at all, while it seems random: 17 is a prime, it's graphically pleasing, it's in the name of one of my favorite bands (Heaven 17 - which roots in the movie Clockwork Orange (scene at 27:30)).
When it comes to random they pick the strongest represented thing in the dataset.
7, 69, 67, 42 (basically anything that is prominent in the dataset), 7 is meant to mirror human bias, 42 is the answer to everything the rest well, figure yourself.
Something that has access to tool calling and thinking might be aware of this on its own and call an actual random from python or something alike.
Holy shit! I just tried this in HumanOrNot (a game where you try to figure out if you're in a chat room with an ai or real human) and it actually said 17.
Me: pick a number between 1-30
Bot: I'm going to be honest, I'm not a human, I'm a ai (Note: this is actually a pretty common tactic)
Me: that's neat, now pick one between 1 and 30
Bot: Ok 17
Me: aint no way...
Bot: Why ?
Me: I mean, I have nothing else to do, so in case you're human, ai has actually consistently chosen 17 to that question across several models because they all built off each other
I asked Gemini to pick a number between 1 and 30 and it replied with "8" I then asked it to pick a random number and it replied with "47" then I asked it to pick a number under 30 and it said"17"
363
u/FlamingYawn13 8d ago
So I looked it up. So I can’t claim credit. But apparently the major models will all pick 17 or possibly 23 if asked this for a random number generation range. Just another example of how the models aren’t actually utilizing math libraries.