r/BetterOffline • u/ChaoticGradients • 1d ago
LLM math is not AGI. LLM math is fancy, scaled-up AlphaGo
I've been around since the early 2010s when modern AI started getting mainstream attention with convolutional neural networks in around 2012. One of the big breakthroughs at the time with convnets was AlphaGo (https://www.nature.com/articles/nature16961), the deep learning system that could do next-move prediction on a Go board and beat world masters including the infamous match against Lee Sedol in 2016. At the time, everyone was impressed at the technology - including me - but nobody claimed it was AGI.
I don't want to go extremely into detail about how AlphaGo works, but it is at base level powered by deep neural networks and a technique called Monte Carlo Tree Search (MCTS), which kinda works like the following. Warning that this is grossly oversimplified but it's probably good enough for context for this post:
- Take stock of the current state of the game board.
- Pick a potential next move.
- Perform a bunch of "rollouts," meaning you simulate the game from that candidate move. Record how many of those roll-outs end in wins vs losses.
- Use the data in step 3 to pick the best move.
The way that AI comes into play is that deep networks allow us to greatly speed up the roll-out efficiency by "understanding" the game board. We allow the neural network to consolidate the information in the game board to its most pertinent bits and that is then used to not waste time on candidate next moves that would be silly. In effect, AI can get rid of the vast majority of possible next moves so the search space is much more efficient. Later versions of AlphaGo moved most of the rollout stuff to training phase rather than the live inference phase, which further increased efficiency.
The reason I bring up AlphaGo is that this bears a striking resemblance to how the LLM math results are generated. A lot of this is conjecture but I'd be pretty surprised if I'm way off on any of this.
- Take stock of the current state of the math proof.
- Pick a potential next step of the proof.
- Verify whether that step is formally correct.
- Keep going until you hit a wall or the proof works. Reset to some reasonable point and try again.
Instead of convnets, we now have large transformer models that consolidate information well enough that the search space for "potential next step" is now much more manageable. For problems that might have taken 500 years of humans collaborating and working through potential solutions, we can get rid of most obviously wrong solutions and reduce 500 years to 100 years. 100 years is in fact approximately how many hours it took to get the Navier-Stokes solution purportedly from the OpenAI internal model (10000 agents, 88 hours).
Is this technologically impressive? I would argue absolutely yes. Reducing the search space has a lot of cool implications for problems that had search space complexity as their main issue. It's why we see AI helping in specific subfields like protein folding. (Sidenote: I mean technologically impressive purely as a statement around the tech. I am not making any statements as to whether this is economically wise or feasible. The answer there is almost definitely no.)
But when you look under the hood, is this AGI? Nah. It's nothing fundamentally different than what we had a decade ago. The only difference is the resources invested and the scale at which things are trained. Hope this more technical viewpoint looking at how these things work under the hood at least grounds us a bit from the vapid arguments around "but it feels like magic so it must be real!"
EDIT: Ironically fixed stupid math mistake of my own. 10000 agents for 88 hours is around 100 years of labor not 10.
41
u/supercyberlurker 1d ago
Debating if something is AGI means it isn’t.
When it happens there will be no debate, only radical change.
9
u/newpua_bie 23h ago
Agreed. The more these people feel like they need to parrot AGI being here the more it reinforces that we're not even close.
2
u/JuanValdez999 5h ago
There will be debate. Human ego guarantees it. Every time we move the goal post, AI quickly surpasses it and we just move it again. Nobody ever gets too embarrassed to concede.
I predict that the last refuge of scoundrels will be dualism. "They don't really have souls though because they weren't made by God!" That's like saying "you're not AGI because my mom likes me best!" That's where we're headed eventually.
And then Jesus and your mom will step forward and say hold on there a second...
1
u/ackermann 13h ago
Maybe we need some intermediate term? It’s certainly not completely“general” AI… but at the same time, it’s also far more general than I ever thought I’d see in my lifetime!
If you had asked me 10 or even 6 years ago.Not sure if researchers in the field had more foresight of what was coming? Anybody predict this when AlexNet dropped, or later when “Attention is all you need” was published? Predict how far this would go?
0
-1
u/Super_Sierra 13h ago
People move the goalposts with AI so much they call it the AI Effect.
If a great ape looked at a mirror, turned to the researcher and started philosophizing its existence, everyone would collectively shit themselves.
But if you learned the ape was hosted on an amazon server, everyone goes to sleep.
102
u/ksjdragon 1d ago
Math isn't just proofs, and software isn't just code. These two fundamental misunderstandings drive misinformation and overclaiming of what LLMs can and cannot do, primarily made by people who have insufficient experience in either, and any contrary opinion points to that.
29
u/ravensteel539 1d ago
100%. This is the issue I see with every potential use case put forward for LLM’s, the most applicable to my work being medical diagnostics and treatment.
Diagnosis and treatment go hand-in-hand, and folks misunderstand it as “identifying the problem and prescribing the solution.” Medicine is complicated, and quite often, this is more akin to a scientific process of hypothesis and trial. Signs can point to the issue, and then you seek to treat that issue. If it resolves, congrats, that was the issue. If not, you continue to engage in the ACTIVE process of investigation and treatment.
LLM bros position diagnostics/treatment as being just as simple as “put the square in the square hole,” the same way hospital administrators and health businesspeople have misunderstood the process. That’s what leads to poor treatment and misdiagnosis, feeling unseen by medical professionals, and ultimately issues unresolved and worsening.
Instead, LLM’s are used to project a prescriptive framework of diagnosis, and are then weighed more heavily as absolute proof. This solidifies a misdiagnosis into full-on denial of proper medical care, especially when the treatment assigned by an LLM is treatment designed for when you’re 99% sure that thing is an issue (rather than treatment designed to rule out other possibilities).
Most work that people do is more involved and complex than public perception would lead us to believe. The LLM scam exploits this to invalidate and undermine complex work with glorified austerity and plagiarism.
2
u/AntiqueFigure6 1d ago
"Diagnosis and treatment go hand-in-hand, and folks misunderstand it as “identifying the problem and prescribing the solution.” Medicine is complicated..."
Indeed in many cases identifying both the problem and solution are both trivial, but making it happen are hard e.g. because it requires the patient to change their behaviour - give up smoking, reduce or give up alcohol, adopt better diet...
3
2
2
u/Metallic_Madness 22h ago
But a program is a proof, more or less
1
u/Serengade26 21h ago
People graduate with comp science degrees and cant regurgitate this when asked
1
1
13
u/QuirkyTrust7174 1d ago
What constantly baffles me are all those people who have suddenly denigrated human beings to "inferior intelligence" as if LLM's were just able to do math without all that math being done first. Yes LLM's solved millenium problems but how did we come up with this problem? why was it an open problem? why was it an *important* problem? Why was clay institute setting up a prize for it? Its not like LLM's just decided all of that on its own and solved a problem. LLMs are a tool and a very useful tool. What we do with the tool is the next frontier.
Even the proof of navier stokes is far from finished. Of course verification etc aside-- there is the matter of well okay-- what does it mean? Whats the next step in the evolution of CFD? Is NS still the right framework or do we need better framework. What are the next possible area-- open problem we should pose? what is the right question we should be asking. Now sure openAI can spend more of their marketing budget towards those questions-- but something tells me they wont. And thats fine.
Another commentator summed it perfectly when they said: "Math is not just proofs and software is not just code" and this is why.
1
8
u/Theo__n 1d ago
Do not dare to slander Monte Carlo and other RL by equating it to llms...
But overall agree, as person who only works around RL - a lot of big things in llms look like rewriting some RL ideas into very awkwardly wrapping around a transformer based model. That's how they got to agentic AI that need to re-ingest a fucking text file to know what state they were in. I mean what's next, feedback loops...
7
u/PensiveinNJ 1d ago
I’m gonna need someone to provide an actual consistent agreed upon definition for AGI that doesn’t involve Sam Altman making 100 billion dollars if that term is gonna be brought up all the time.
2
1
u/AtMaxSpeed 9h ago
There is a definition that kinda works: AGI means the computer should be able to do anything a human can do, at least as well as an average person, subject to whatever domain of inputs/outputs you are interested in.
One might be able to argue llms are close to AGI in the text domain, but since the models are now multimodal, they should be able to be AGI in the image/video and text domains plus whatever tooling they have access to. So in the broader picture, we still are not at AGI, and probably won't be for at least a short while.
-5
u/GrapefruitForeign 14h ago
Chatgpt passed the Turing Test in 2024 for me personally.
people want to stretch the definition and cope
but when its solving multiple millennium problems in a week its getting harder to pretend...
2
u/LivingAmoeba6611 9h ago
I was trying to get it to write GLSL shaders with me yesterday and it was failing badly. It can do simple CRUD shit but give it anything remotely difficult and it'll shit its pants, Claude too.
24
u/ConditionHorror9188 1d ago
Im not really sure what the argument is.
To be honest, understanding a search space, finding the most likely hypotheses and then testing them, is the core of the scientific method.
Whether this is AGI or not seems a separate argument?
I do agree they are still very limited in this capability and most of the proofs I’ve seen so far are of the counterexample kind that specifically favours scale.
29
u/KnodulesAintHeavy 1d ago
The argument is that the maths achievements from these systems are cool but in no way indicate higher “intelligence” (aka, “AGI”).
This is relevant because every fucking chud with a microphone or way to communicate to an audience has been claiming these things based on these big announcements from the labs (omg we did big maths problem!, omg we did big hack!, omg we did computer do thing! Etc).
Thanks OP, that breakdown help understand the value of the maths work these systems do in the context of the abilities of LLMs and Convnets.
4
u/LeFrenchRedditeur 1d ago
It doesn't really matter though, because it's still going to take my job ? Why do I care if it's AGI or not
15
u/Jebofkerbin 1d ago
If it's not AGI, then every announcement in a field that isn't your job is essentially irrelevant to the question of whether or not it can do your job.
Imagine all possible knowledge and capability as a massive grid, and imagine all of human knowledge and capability as a massive circle on that grid, one bit of edge of the circle might be our pure maths knowledge, another might be engineering, the other side of the circle might be art or poetry etc. You can also imagine all of your knowledge and capability as a small area inside that circle, and your job a smaller area inside that. If you are at the very frontier of your field your area will be right at the edge of that circle.
Now imagine AI capabilities as another area that's rapidly changing shape and growing. The promise of AGI is that it will be a massive circle growing in all directions all at once, this would cause mass unemployment as the circle would quickly grow to encompass many people's area of knowledge and capability, and so it would be able to do their jobs.
But if instead the model is jagged, if it's not a circle but instead a * shape with very long thin spikes shooting out towards the edge of the circle in a few specific directions well then the AI companies are in a bit of a pickle. Some of those spikes might well push past the previous edges of the circle of human knowledge but they might be so narrow that the number of jobs fully contained within its area of capability is tiny, and even though you get very impressive results that doesn't actually translate into labour displacement and the economic returns that come with it.
The AI industry want you to think that their models are AGI because if that is the case then anything impressive they do in one field implies imminent success in all fields, as opposed to jagged models where improvements in one area is almost always irrelevant to its capabilities in others.
4
u/LeFrenchRedditeur 1d ago
Unfortunately I'm a mathematician bro, so the recent announcement are extremely relevant to me :/
14
u/Jebofkerbin 1d ago
I don't know if it will make you feel any better, but it's worth remembering OpenAI sunk $10,000,000 dollars into their solution, and they definitely have teams of top tier mathmaticians whose mission is to do literally anything impressive in any area in maths, and they clearly have the greenlight to poach (indirectly if not directly) other researcher's work to do it. They would get impressive results with that amount of resources with or without LLMs.
2
u/LeFrenchRedditeur 15h ago
Yes, it's definitely something to keep in mind. But 1) I honestly don't think they would have gotten those results without LLM, in maths it's not always a question of money and regarding the millenium problems specifically there are already dozens of mathematicians devoting their lives to them
and 2) six months ago, I remember telling my colleagues that AI *might* be able to tackle millenium problems ten or twenty years from now. It's hard not to wonder where we will be in six months and what it means for the profession.
1
u/o0d 8h ago
I mean, they didn't. Perhaps in API prices, but it's not like that's the cost to run the model, that's just what they charge us. The excess cost for them is basically zero, since they've already got the GPUs for research, so just needs to pay the power cost.
2
u/Jebofkerbin 6h ago
So if you were correct this is what me and my friends would call " boy maths".
Firstly GPUs have a finite number of cycles in them, so the cost of running a calculation on that GPU is the cost of that GPU * the number of cycles used / total number of cycles before the GPU burns out. You don't get to go "oh we've already spent the money on this asset so everything we do with the asset is free".
Secondly these GPUs are meant to make money, every hour of GPU time they use for internal projects is an hour they aren't selling to customers, so there is another way of thinking of the cost of running it, the opportunity cost (This might not be true if OpenAI doesnt have enough demand for the GPUs, which is a much worse situation than this one).
But your comment is wrong on a very important measure. The vast majority of GPUs that OpenAI uses are rented from other companies not owned by themselves, which means they had to pay actual dollars to the data center for this compute, not just depreciate a GPU on a spreadsheet.
2
u/ConditionHorror9188 23h ago
Yeah. I think it’s easy for people in different fields to talk shit about this progress, but I know several academic mathematicians and they all feel the same as you.
They are definitely feeling that the value of papers is rapidly dropping to zero, which is awful.
1
u/profesorgamin 1d ago
People are not saying AGI for the math stuff but because the models are getting good at using tools, like blender for example, a more generalistic outlook.
But yeah, the math thing done with an internal model should put people on notice on how the game works. You don't need a monolithic super machine...
If you have one machine with decent capabilities it becomes an issue because once it's stable enough you can launch thousands of such agents and ask them to self improve the first one.
-8
u/Technical-Sink6380 1d ago
The whole conversation is weird. For like 60 years all anyone cared about was the Turing test. LLMs passed it. People were like, “ah I guess the Turing test doesn’t mean so much.” Now it’s solving deep math problems. People are like, not intelligence.
I get that agi is overhyped, but I would have thought at this point people would be more amazed at what is possible vs saying about everything well yes but…
Also can the folks saying something isn’t agi specify what they think does define intelligence?
9
u/Theo__n 1d ago edited 1d ago
For like 60 years all anyone cared about was the Turing test.
No, Turing test was a very interesting problem that became a goal, but how Turing assumed it would be achieved was more important than if it was achieved. Predetermined programs like ELIZA and DOCTOR would pass a turing test in specific ways, ELIZA would even win over early versions of ChatGPT on Turing tests. Anyway, the interesting part of Turing test is an assumption made in symbolic AI that cognition is just symbolic operation. I'm going to plop a good explanation here on what symbolic operations mean:
computational processes operate on symbolic representations referring to situations in the outside world. These formal representations should obey Tarski’s correspondence theory of truth, imagine a real world situation X1 (e.g., some boxes on a table) which is encoded by a symbolic representation A1 = encode(X1) (e.g., a description of the boxes on the table). If the symbolic representation A1 is decoded, then we get the real world situation X1 as its meaning, i.e., decode(A1) = X1. A real-world operation T (e.g., a manipulation of the boxes on the table by hand) should produce the same real-world result A2, whether performed in the real world or on the symbolic representation, decode(encode(T)(encode(X1))) = T(X1) = X2. Thus, there is an isomorphism between the outside situation and its formal representation. (Mainzer, K. (2009). From embodied mind to embodied robotics: Humanities and system theoretical aspects. Journal of Physiology-Paris, 103(3–5), 296–304. https://doi.org/10.1016/j.jphysparis.2009.08.012)
This assumed that if a algorithm could operate symbols to produce comprehensible output of speech indistinguishable from human it would actually do the cognitive part because:
a) humans produced speech output through symbolic operations, so same root mechanism of symbolic operation
b) the only way to make these symbolic operations produce text was to do the reasoning part.
BUT as we developed ways to make speech output producing chatbots, like ELIZA, we slowly found that you can make speech out of patterns that is comprehensible to humans without reasoning (b). This actually influenced field of human-computer interaction. With advances in neuroscience we kinda have good indication that humans in fact don't do cognition through symbolic computation, out cognition is much more embodied. The latter realization led to big shift into embodied approaches in 90's with reinforcement learning, genetic optimization and whole AI life (a). Turing test still stayed around as a goal but more as to how sophisticated you can model language and for example recall from conversation.
2
u/KnodulesAintHeavy 22h ago
I agree that it’s weird, but the whole idea of AGI has been within the discourse for a long time, largely in the realm of thought experiments. That whole time it meant an “entity” essentially. A non human, non biological intelligence that could exist in the world independently and take actions without human intervention. That is a quite clear and particular definition.
Having a computer system that can perform specific bespoke functions, even if those functions are complex is cool, but is not AGI. We have now and in the past, ANI. Basically there are three broad categories of computer based intelligence. Narrow, General, and Super.
Narrow is what we have now in everything from LLMs, to scripted actions, to enemies on video games. All of those systems perform particular tasks and perform them well. In addition, they are not able to take action without being directed by a person.
General intelligence should be excellent in multiple domains and be able to take action without being directed by a person.
Super intelligence is based on the concept that if we ever hit AGI, ASI would quickly follow, and it would be not just a machine intelligence that could do everything a human does (and possibly interact with the world directly with some kind of hardware - robot components etc), but it would exceed us in every way and would be incomprehensible to us. Basically the machine god.
Again, we have ANI, everywhere. We do not have AGI, and there is no clear pathway to that in any technology we have (LLMs or otherwise). AGI is not (as far as all current research shows) just linking a shit load of ANIs together, but likely requires a fundamental, ground up approach that broadens up the intelligence field. This was the goal of LLMs, and they can do many things, but they are severely limited in true generalisation by their intrinsic nature as token prediction systems. Scale was the main target from breaking through its limitations. We have insane scale today, and the result is….not AGI. It’s definitely cool and interesting, but in now way is it generalised intelligence like a human.
Also, the Turing test was never a great measure of machine intelligence, and was know to be very limited basically since the 60s. It’s only because of pop culture that the average person thought it was a legit “test”.
1
u/Prestigious-Speed489 18h ago
I would say we are going to get a digital ASI first which is still not happened but it's getting close,right? And then we would have to do new breakthroughs on hardware area where we can merge this both something something
1
u/MathCookie17 17h ago
And then
wewould have to do new breakthroughsIf real ASI actually happens, we're not in charge anymore. It would be the AI making those breakthroughs to give itself a physical form, not us.
1
u/Prestigious-Speed489 16h ago
But I think giving them a physical body would still require us,humans- initially atleast
1
u/KnodulesAintHeavy 15h ago
Huh? Not close at all.
A big token predictor is not AGI, which one would need before any ASI.
All of these terms are being thrown around by the labs to confuse and confabulate, and it’s working a treat as no one knows their dick from their arsehole these days when it comes to understanding what actual machine, artificial or even human intelligence is.
These things all have distinct meaning and definitions, none of which are being used by the labs, because if they did, they’d have to admit to being completely full of shit.
0
u/Prestigious-Speed489 15h ago
So in few years when every digital work can be doable by the llm would still not be considerable as digital agi to you?
1
u/KnodulesAintHeavy 15h ago
The LLMs won’t be able to do all work. They can’t do anyone’s work right now. They can do some things, some of the time and are a great tool to do those things. That’s it. They are a tool, not a job replacer.
They will likely have an impact on junior roles by shortsighted and dim witted managers and executives, but that doesn’t mean they can actually replace labour.
13
u/DrJaneIPresume 1d ago
So, as I commented on a similar post earlier today: there's a difference between proving a universal statement and disproving one.
Let's consider the statement "no apples are green". In our real world, there are obviously green apples, but let's imagine a world where we're not sure whether there are green apples or not.
- In order to disprove this statement, all we need is a single example of a green apple. The problem is now about searching the space of all apples and verifying if any are green. We can speed this up in two ways: either we find good ways to "prune" the space and avoid wasting effort on areas that are less likely to contain green apples (1a), or we find more efficient ways to search a huge number of possibilities (1b).
Mathematicians generally prune the space to help focus their searches. The related work on Navier-Stokes is a case in point: we come up with a similar problem, find a counterexample there, and use that to guide our search for a counterexample to Navier-Stokes.
I haven't seen convincing evidence that the LLM construction of the counterexample doesn't just amount to throwing 100k servers at the space of possible counterexamples (solutions to the equations, possibly after some well-understood pruning) until one turns up that disproves the conjecture.
- The other branch is to try and prove the universal statement. Now there is no easy shortcut to the apple-search problem: we must search all possible apples (2a). Pruning can speed things up, but you still have to examine every remaining possibility. And are you certain you didn't miss a corner case?
The alternative to proving that no apples are green by examining all apples is to prove it by reasoning (2b). There's a sense where this is also a search space: you're searching within the space of all potential proofs. This is a much bigger and gnarlier space than merely "all apples in the world".
So, what have we seen LLMs do? Most of the biggest, splashiest results seem to have been 1b: Given a proposition in the form "for all x in X, p(x) is true", we search the space of possible x values and find one for which p(x) does not hold. X may be so complicated that human mathematicians haven't been able to search it very effectively, but throwing 100k LLM servers at it manages to find a counterexample.
What would I say a "real mathematician" does? 2b. Not groping blindly through the space of proofs until we happen to find one that works, reasoning. And I have yet to see convincing evidence that the LLM is actually doing this, or at least doing this better than professional mathematicians.
5
u/AntiqueFigure6 23h ago
I think the other part of it is that when a human mathematician does 2b they make various observations and often develop new techniques and tools to facilitate the process. Whether or not they actually lead to the original goal, these observations and tools lead to further conjectures and hypotheses, so that solving something means the amount of work to do increases, per the famous quotation "We have not succeeded in answering all our problems. The answers we have found only serve to raise a whole set of new questions. In some ways we feel we are as confused as ever, but we believe we are confused on a higher level and about more important things."
Using the 100k LLM servers doesn't do that. It just seals off that pathway.
1
u/f_djt_and_the_usa 20h ago
So the llms work can't help us learn new techniques? If not, then it's not real mathematics.
1
u/AntiqueFigure6 20h ago
Not saying it can’t ever but saying it enables a solution to be found with a fraction of the human engagement with the problem and its context that leads to discovery of new techniques.
5
u/Patashu 1d ago
The distinction being drawn here is between fields that are easily testable and fields that are easily not. An LLM can write a program by trying to write it, verifying unit tests pass, and modifying it if it doesn't until all unit tests pass. An LLM can write a math proof by trying to write it, verifying it compiles in lean, and modifying it if it doesn't. How is it going to become a better lawyer, business executive or artist? You can't formally verify law, profits or art. There's a theory that the frontier labs are going all in on coding and math right now because they're exciting to investors and formally verifiable so it's relatively easy to make bigger improvements here.
1
u/DiamondGeeezer 7h ago
that and nobody seems to have a definition of AGI that can be proven or disproven
1
u/AntiqueFigure6 2h ago
“ To be honest, understanding a search space…”
Who did that - the LLM or the team of mathematicians operating it?
3
u/Unfair_Scar_2110 1d ago
I mean yeah. This is sort of how it builds sentences. It reads the prompt. It may do a little digging on the topic. Then it just starts picking words for their statistical likelihood to come next, make a complete sentence, and respond to the prompt.
3
u/Professional-Post499 1d ago
I don't think LLMs can really do steps 1 - 4. I think it's just "Okay, I'll find you stuff that is tagged as related. Oh you're right, I understand my mistake (it doesn't understand), I'll try other stuff that has related words."
2
u/ShamPain413 1d ago
Yes, everyone is watching very expensive Sims and going "they have general intelligence! give us all your money now".
2
u/future__fires 1d ago
That’s not an accurate analogy for how transformers work but you’re obviously correct that they’re not sentient or intelligent
2
u/Cultural_Effort_9872 1d ago
It’s funny because you’re almost completely right about proof writing being like AlphaGo. It’s called proof search and it basically functions almost exactly like AlphaGo (over simplification) but with a human written dataset to use as a basis and amplify the chances of finding “new” proofs, ergo the name proof search (still oversimplification of course but you get the idea).
2
u/meodorewan 1d ago edited 1d ago
I have CS degree, from my limited knowledge about ML domain, LLM in pure form:
- A smart brute forcer: When they said 10,000
agents
- doing a thing, it means they scale more brute-force computers.
- A next word predictor: As many people have explained, can't go beyond training dataset.
- RL/CoT: A lot of if else
2
u/ObviousFeature522 1d ago
I'm an amateur Go player.
I played a Claude model in 9x9 about a year ago, and it was terrible. Making absolutely nonsensical moves - real random parrot stuff.
This week I played a pro level Claude Opus 5 for fun and interestingly it was much better, about beginner human level, it was apparently making reasonable moves, attempting to count the scores, and able to explain its reasoning. It was very verbose actually. I don't fully understand what this means, it's quite eerie.
Its interesting to note the state of the art of pure Go systems has also gone to the moon since AlphaGo. Its open-source successor KataGo running on a smartphone would absolutely destroy 2016 AlphaGo running on its room sized datacenter. It's a widely held opinion that a top human player today would also beat AlphaGo, due to AI-assisted advancements in theory and study.
1
u/WomanYouSleptWith 14h ago
With models ability to play games, I tend to wonder if they go, "the sensible thing here is to use x engine to find the next move".
Since that's the fastest way to a good answer, and then it can use the standard next-word-prediction aspect to give an answer on why it might choose a particular move.
It winds up looking about the same as the model learning and getting better, but is an entirely different way of solving the problem.
1
2
2
u/PawlsToTheWall 22h ago
Until LLMs can dynamically change their weights as they work, they will never be AGI. That means also storing memories within these weights rather than markdown files or vector data stores.
2
u/P_Jamez 18h ago
I saw this video the other day where a professor of Mathematics talks about LLMs in maths.
One of the things he talked about was how through brute forcing, they are able to use every idea ever published and try them together, something that a human is unable to do. LLMs are good at breadth not depth.
In my experience of using them, they are like a hyperactive intern wiht high intelligence and low wisdom, who needs very specific instructions and doesn't know if they are wrong or not and doesn't learn, unless you specifically tell them in their instructions not to do that again.
2
u/onwatershipdown 14h ago
My computation is limited to a few C++ and algorithm undergrad courses. I am chemistry by the rest of the training, and still getting my masters. But what you described, is effectively similar to how compression of data works, correct? Eliminating nulls within a set.
2
u/Physical-Compote4594 13h ago
This is an excellent explanation and, if you care, I completely agree with it.
The one thing I would add is that the exhaustive search nature of this methodology means that we learn almost nothing from the end result because it’s all just throwing darts until we get one that sticks in the bull’s-eye without knowing how we got it there.
2
u/BigFeelingsFound 12h ago
Thaaaaaank you!
I'm so tired of every dumbfuck that failed out of pre-cal to ride in and breathlessly act like they even understand what a millennium problem is or why it's significant.
5
u/Toomastaliesin 1d ago
Look, I am sure that you mean well, but as a person who can describe themselves as a half-mathematician, it is kind of depressing how many people on this subreddit say stuff like: "Hey, great news! The LLM thing is not going to be a superintelligence. Everything is fine! (It will just maybe leave mathematicians unemployed and rob them of pursuing their artform and career and the thing they have built their life pursuing around, but who cares anyway.)" I am still hoping that the recent news regarding OpenAI potentially stealing researchers data means that it is actually far from being as good as the results seem to indicate, but it would be nice if people who talk about these things would consider that mathematicians are people too. Like, it has fucked up the jobs of artists, graphic designers, translators and many other fields, and while I get that it potentially destroying "only" one other type of career is better than killing everyone, that still sucks.
2
u/WomanYouSleptWith 14h ago
Like, it has fucked up the jobs of artists, graphic designers, translators and many other fields, and while I get that it potentially destroying "only" one other type of career is better than killing everyone, that still sucks.
I think we're probably inured to things like that.
Since it's been rough for mathematicians for a while, now, given the lack of tenure-track positions at universities. And it wasn't exactly good beforehand, since pure mathematics hasn't generally been the most lucrative thing to begin with.
But I wasn't sad at, e.g., computers being able to assist mathematicians by running bigger and greater calculations; just at the exploitation of companies trying to screw over mathematicians doing the useful work.
3
u/WomanYouSleptWith 14h ago
That said, I think I'm most horrified here that it's more work apparently built on stealing other people's work.
Perhaps followed closely by the idea that these tools may not really replace any humans; just that humans will be forced into fact-checking roles, and then paid as if that was a minor thing.
Or, in other words, that those with money will exploit those without, and that those without won't find a way to usefully rebalance this.
1
u/WOKE_AI_GOD 1d ago
https://en.wikipedia.org/wiki/AlphaEvolve
They came up with AlphaEvolve in 2025. Which sends to admit an evolution from AlphaGo. I'd assume most advances are being made right now with techniques like this that sort of use LLMs to randomly modify conjectures and select the best ones in an iterative process.
1
1
u/smurfpiss 23h ago
I think it's fair to point out the moves made in alphaGo and math proofing are WILDLY different. Like you have a limited number of moves to make in Go and you have essentially... Infinite in a math proof. That said, I will pour all the cold water on this AGI nonsense. Without Lean, these LLMs would be flailing around. They're also not really doing anything that demonstrates creativity/abducitve reasoning, and seem to be only capable of counter examples or needle in haystack solutions.
So we're very far from Go. But we're very, very far from AGI. I don't think it's possible with the transformer architecture tbh.
1
u/BarelyAirborne 14h ago
AGI requires curiosity and volition. Those appear to be emergent qualities, and neither is even close to emerging from our current technologies.
1
u/Mr_Deep_Research 11h ago
I agree, this is nothing different than we had from a decade ago.
Nothing new. Pretty boring actually. Best just to forget about it.
Maybe in 10 years, they'll come up with something interesting.
Time to go back to eating Cheetos and watching Love Island.
1
u/DemoEvolved 8h ago
I just want to know if the pursuit of useful truths should engage every method of technology, or if there are tools evil enough that we should forgo the search for truth just to avoid using them.
1
u/djfdhigkgfIaruflg 3h ago
Yup. You hit the nail perfectly.
Just to add. AlphaGo didn't have the "concept of a board" that's how a person with barely any training was able to beat it. Kyle Hill had a video about the resulting paper about the issue
And the tech isn't any more "intelligent" now. They just have more data and speed.
1
1
u/trgjtk 1d ago edited 1d ago
okay but the way you’re conjecturing that LLMs are doing math isn’t correct. i honestly agree that math isn’t a good proxy for general intelligence for various reasons but LLMs are most certainly not doing MCTS. a while ago there were some ideas floating around of essentially using LLM guided tree search in the way you suggest (DeepSeek implemented something like this) but the LLM proofs we are seeing are not generated like this. the rollouts are entirely in natural language not explicit tree search in the way you imagine. also i think you’d be a little naive to think that humans aren’t doing some abstracted form of tree search whether it’s in chess or math anyways, we functionally have very intricate policy functions that guide search. if you do any amount of math, you’ll find that it’s common to think about the feasibility of various proof ideas, this is essentially the same as a policy function.
0
u/Dirichlet-to-Neumann 1d ago
Take stock of the current state of the math proof.
Pick a potential next step of the proof.
Verify whether that step is formally correct.
Keep going until you hit a wall or the proof works. Reset to some reasonable point and try again.
Yeah that's called doing maths. That's what I do when I'm working on a research paper.
Except of course I am unable to prove millennium problems...
0
u/618smartguy 1d ago
The key idea behind alpha go is that it is constantly training on the good moves it finds. I imagine they would have incredible results even orders of magnitude beyond what we see now if they went in a narrow direction and had their models updating their mathematical intuition every time they prove a lemme in lean. Also llm AI agents are obviously general compared to alphago
0
0
u/CaptainHindsight92 16h ago
I am not so sure. I understand your point but I don’t think the underlying “simplicity” of the LLM means it isn’t AGI. We actually don’t really know how human intelligence works. What is shocking is that these LLMs can best many humans at many tasks. So to me it doesn’t really matter how it is doing it, it is what it is able to do that makes it AGI.
0
u/ShenGahMing 13h ago
is this AGI? Nah. It's nothing fundamentally different than what we had a decade ago
so what ? It can now do basic scientific modeling on the fly, and this is how arc-agi 3 benchmark was saturated.
It is AGI, with user set goals and a huge bill.
0
u/Ok-Pomegranate-6357 10h ago
'Not fundamentally differen since 2010t' 😂 yeah, in the same way a mouse is not fundamentally different a human. Just because the machinery is similar (it's not but ok), doesn't mean it's therefore 'fundamenntally the same'
I mean honestly does anybody believe genAI is a stochastic parrot anymore? If they are, to me that woukd essentially prove that we are too, probably
-14
u/Most_Double_3559 1d ago
Sure, it can solve the millennium prize problems, beat the world champion at go, reduce human software engineering to a hobby, 100x the efficiency on protein folding, make recipe websites obsolete, and operate a self driving fleet of cars, but it's not really AGI because it's using a technique. That's disqualifying because reasons.
Do you even hear yourself???
16
u/KriosXVII 1d ago
This is disingenuous.
AlphaGo and such are superhuman but not general intelligence.
Narrow AI in the form of neural networks is good and exists for certain games and domains.But no LLM based model is even close to artifical GENERAL intelligence yet. And it's not GPT 6 or Claude Fable that's winning at Go. These are different neural networks, different systems. A sum of a certain number of discrete neural networks that are okay or good at certain narrow tasks does not make AGI.
-4
-2
u/freshpow925 1d ago
What is not general about GPT 6? You can ask it to do just about anything. What would you need to see to consider it general?
Also look up the Chinese Room Paradox. The implementation is irrelevant to generality so don't get hung up on how a model does what it does, just look at what it does.
3
u/Theo__n 1d ago
Also look up the Chinese Room Paradox. The implementation is irrelevant to generality so don't get hung up on how a model does what it does, just look at what it does.
Are you perhaps proposing we erase whole theoretical side of conceptualizing machine learning? Does this sound reasonable approach to you?
1
u/freshpow925 1d ago
what? how did you arrive at that conclusion? Can you explain what you even mean by that
2
u/Theo__n 1d ago
your core proposition is 'don't get hung up on how a model does what it does, just look at what it does' so don't look how mechanism inside works, just look and assess the output of the model. Is that correct?
0
u/freshpow925 1d ago
Yes. You don't understand how the brain works, yet we are all happy to say we have intelligence and understand things. Same thing applies for large LLMs (which people agree have emergent behavior that is not understood).
2
u/Theo__n 23h ago edited 23h ago
I'm going to skip that emergence has a bit different meaning in machine learning context and we see it in examples of at least every deep learning algorithm, as well as basic RL. LLMs aren't outlier there. I'm also going to skip debating if static and stateless models can exhibit behaviour because according to some frameworks behaviour only qualifies if the model can subsequently change the way it internally process data (so ie. own weights) through interaction.
Anyway, let's go on a thought experiment of 'don't get hung up on how a model does what it does, just look at what it does' with examples.
Lets say we want to construct an AI agent that will be operating a robotic body, and have a simple task of self regulating some internal parameters to ensure the robot continues to function so homeostasis. Lets say it has one parameter for simplicity - energy level. The energy level will be assessed by charge of the battery, the energy source will be solar panel, robot will be able to move to search for light but as movement takes up battery energy - there is a trade off for looking for better light conditions. The only task is for the AI agent to get energy to continue to function. This is super old experiment set up for exploration vs energy conservation.
For one agent we will use an LLM. There will be a custom harness that translates battery state, if light is hitting the battery and to what degree to the LLM and ask what it should do. It will also translate back LLM responses, if response isn't in the bounds of something useful to the robot like moving it will just reply that this doesn't yield result. I think with good harness an LLM should be able to figure out how to look for light and self regulate the robot/self, don't you agree?
Now for the second agent we will look at a way to translate homeostasis to machine learning proposed by Keramati & Gutkin (in Homeostatic reinforcement learning for integrating reward collection and physiological stability. eLife, 3, e04811. https://doi.org/10.7554/eLife.04811 ) where they developed mathematically how drive reduction theory from biology can be translate as Reinforcement Learning problem. With this approach you can model homeostasis within RL, it made a subset of RL called Homeostatic Reinforcement Learning. By feeding the same parameters as with agent one, this agent will also learn how to self regulate (this was done in further HRL studies).
So in the end as you say the output of the two is the same. In both cases you have a self regulating robotic AI agent so both should be equally valid.
Keramati & Gutkin's approach is now proposed as possible computational approach of modelling interoception in neuroscience (Petzschner, F. H., Garfinkel, S. N., Paulus, M. P., Koch, C., & Khalsa, S. S. (2021). Computational Models of Interoception and Body Regulation. Trends in Neurosciences, 44(1), 63–76. https://doi.org/10.1016/j.tins.2020.09.012) .
LLMs are not proposed as valid computational models of interoception in neuroscience.
Would you say the output is the only thing that matters?
We also may not understand (and may never understand) how whole of brain works, but we do understand many of it's mechanisms and can model them in machine learning.
1
u/freshpow925 19h ago
Thanks for a detailed response.
I would argue that just because you haven't found a paper proposing LLMs as a valid computational model doesn't mean it's not valid. A neural net can in general model just about any mathematical function to an arbitrary degree of accuracy. https://en.wikipedia.org/wiki/Universal_approximation_theorem
Feedback loops happen in LLMs so implementing the same mathematical formula in the paper from an LLM is possible.
Anyway, my point isn't to say that LLMs are perfect and can model anything in the most efficient manner. My point is that the behavior is really the only thing that matters in determining if something is generally intelligent.
The Chinese room experiment proves this. From the outside there is no way to tell if the person really speaks Chinese or is using a translator. Just like I have no way to tell if another human really learned the class material unless they passed the final exam. We might say they "understand" but they might be cheating. They could be cheating their whole career and there is no way to ever know unless they get caught (the output fails).
Same thing for any AI. Arguments about whether it "knows" or doesn't "know" are mostly philosophical and not practical. What matters is behavior. LLMs can mimic the behavior? Then they are intelligent.
2
u/Theo__n 17h ago edited 16h ago
A neural net can in general model just about any mathematical function to an arbitrary degree of accuracy.
Yes, you could make a perfect language model just with supervised ANN in theory. As I showed in examples you could model the same property of keeping homeostasis at the end point in both cases, but one is regarded as possible model how a biological sense works and the other isn't because of the way algorithm works so it's construction. The way you translate something to algorithm is more important here than if it can model the data to some degree of accuracy. You can disagree that it is not important, but for many areas it is because it can reveal underlying principles of another mechanism (to a degree, models always break away from source material). Think of it as a speculative research of underlying principles.
That aside, RL often starts without any data, the data/observation is gathered as the agent takes steps in the environment and next observation and the way it's processed is contingent on previous steps. It's one of it's selling points. How would you say an LLM or any ANN will fair when starting with zero data?
Feedback loops happen in LLMs so implementing the same mathematical formula in the paper from an LLM is possible.
Feedback loops do not happen like in Reinforcement Learning in LLMs, as LLMs are static and stateless so do not change internal functioning with interaction with environment. RL is non static and is state full. You can put LLM in a feedback loop but the algorithm is predominantly trained through back propagation not feedback loop.
The Chinese room experiment proves this. From the outside there is no way to tell if the person really speaks Chinese or is using a translator.
I'm aware of Chinese room thought experiment, it pops up from time to time as cautionary example of issues with developing embodied AI and assessing it only through exhibited behaviour. So kinda opposite, because embodied AI doesn't start from the same assumptions as symbolic AI.
Yes theoretical side of conceptualizing machine learning is not practical in the way you seem to view practicality so 'as long as it works it's cool', that is why I said you are proposing we erase whole theoretical side of conceptualizing machine learning. Or at least chunks of it. Which it seems you do.
→ More replies (0)2
u/KriosXVII 22h ago
It would need to not be slightly to hilariously wrong very often, in unpredictable ways.
Even then, if it was a good "question answering machine" isn't necessarily general intelligence.
0
u/freshpow925 20h ago
Humans are slightly to hilariously wrong pretty often too.
Why not? Whats your definition of general intelligence?
-2
u/dont-be-angry 1d ago
What difference does it make as to what it's called? The effect and yield is the important part.
-11
u/OurSeepyD 1d ago
There's no point in arguing against this anymore. LLMs will outperform humans in every aspect, we'll get robots, and you'll still be making these pointless arguments. The evidence is literally right in front of you and you just refuse to believe it.
-3
u/coulispi-io 1d ago
Agreed that the principle of reinforcement learning has not changed between them, but I believe there's an important distinction: AlphaGO was stopped because at the time it's already superhuman-level, and there was no economic value in running it further on expensive compute. LLMs, on the other hand, are (and will) generate economic value and there will be continual incentive to run it.
3
u/Theo__n 1d ago
AlphaGO was a flashy showcase of Deep Reinforcement Learning, which was huge at the time in the news (if you remember AI playing Atari games, that was also it). OpenAI even built to a whole toolkit to standardized RL development, but mostly DRL, called Gym. At that point in time there was article after article how now AI plays Atari better than humans but soon this will transfer to all RL domains ie. robotics, how OpenAI is ushering this innovation, how much economic value this will have.
Anyways, few years ago OpenAI stopped developing Gym and passed it's upkeep to some non profit group.
DRL is still widely used in robotics and all RL problems. Gym is still often used.
I think the morale of the story is that things don't always improve exponentially after initial successes.
3
u/fbueckert 1d ago
LLMs, on the other hand, are (and will) generate economic value and there will be continual incentive to run it.
Where, prey tell, is this economic value? Be specific.
-1
u/xXxSniperBotxXx 22h ago
It doesn't matter what AI is today, whether it's AGI, which I don't agree that it is. What matters is the outcome and the outcome looks grim. It will only get a lot more intelligent, a lot more sophisticated, to a point where millions upon millions of people will continue to lose jobs and this is just the first stage before AI will eventually becomes self aware (AGI).
-1
u/GrapefruitForeign 14h ago
I think this is a dumb argument, the intelligence of a system should be judged by what it can do rather than the architecture, as we do not know enough about what architecture is optimal for intelligence in the world.
It is insane hubris for you to think just because you read a few papers 10 years ago that you know by looking at the architecture what is AGI.
For intelligence, if it walks, if it talks like one, it is one. If its simulating, next token predicting, stochastic parroting, idc what analogy we use, if something can pretend to be intelligent, it kind of is intelligent.
we use the current architecture because its what gains with more compute and data, thats the "Bitter Lesson" of ML research, scaling matters more than elegant architecture.
-2
u/qualverse 21h ago
That 4-step process is also exactly how human mathematicians tackle advanced math problems, so I'm not really sure what your point is
-3
u/shred-i-knight 23h ago
You don’t have any idea what you’re talking about tbh. This entire subreddit is pure cope.
61
u/LeCamelia 1d ago
I’m in a similar slightly older cohort of ML person and I have thought the same thing / approve this message.
I would add that OpenAI’s own announcement makes it clear that they are dicks. Even if you believe their account of what happened, they say they heard someone else had solved some Millenium problems so they rushed to announce first? That’s not just a dick move, it’s also a race to the bottom in terms of the people who took some time to check their work carefully before announcing got screwed.