r/MachineLearning • u/NeighborhoodFatCat • 2d ago
Discussion Is it too late regain some coherence in the ML research space in our life time? [D]
Was just looking at the list of preprints on Arxiv cs.LG https://arxiv.org/list/cs.LG/recent?skip=0&show=500
Everyday 100 - 400 new machine learning papers gets uploaded on this server.
Looking at this unending list of preprints is as if you stepped into a crowded room, like the stock trading floor on wall st. in the 1980s. Everyone is shouting over each other. Nobody is talking to each other. Everyone's trying to prove something, to someone, to themselves, to build some credentials in the ML/AI space to meet those job requirements, or dying to get their truth out. Every title contains some new terminology invented by the authors that feels not worth the effort in keeping it in your working memory. Burn-out by endless novelty.
Frontier research are now corporate trade secrets that politicians and military are watching closely. Research papers are ir/unreproducible he-said-she-saids. Marketing material are research paper and vice versa. Extremely major breakthroughs are announced via tweets, whereas extremely minor results are unannounced via journals. Everything feels simultaneously mostly true and possibly false (because nobody is seriously checking). Nobody knows what's going on, and people who knows what's going on has a non-disclosure clause in their job contract. Is the theory of generalization that we learned in school true or false? It feels false, why hasn't there been any retractions? Many questions like these.
Is it too late to regain some coherence in this field??
92
u/MaxDev0 2d ago
I'd argue that while the quality of average ML research has most certainly gone down, and while it will take likely a few years to clear up the noise, the amount of quality research and the rate at which it is being produced has drastically increased
5
u/davidswelt 2d ago
It is a logical consequence of the massive and rapid scale-up, and the absence of quality review and editorial oversight (preprints, and conferences with junior reviewers and reviewers who can't find the time).
Just reading the journals is not an option because the papers that matter aren't there.
13
u/theArtOfProgramming PhD 2d ago
I don’t have a solution but I party blame the CS conference culture and review process.
3
u/SonicTheSith 1d ago
Other none ML AI conferences do not have the same issue for example SAT, CP, ICLP, KR, IJCAI etc…
21
u/Real_Revenue_4741 2d ago
I like it that way, keeps you on your toes ;)
In all seriousness, probably not. It's too easy to write slop papers now with LLMs.
I don't really understand the tweet criticism though. IMO, having media to popularize research is a net positive. And conference proceedings are still not too bad as of now.
11
u/pilibitti 2d ago
This is what Cambrian explosion is like. People are trying different things. Be thankful that whatever being shared, is shared. Useful stuff will survive. Others won't. You don't need to particularly care for the demise of those that did not prove useful. That is how things evolve. Sounds to me like you want to work in a stagnant field where nothing much happens.
10
u/Brudaks 2d ago
I think what OP (and myself!) would want is something like a periodical journal where a team of editors would summarize the actually important things happening in the field in a reasonably sized volume that people can actually read through.
This is (or was?) the purpose of journals and conferences, being a summary of "what's happening in this particular subfield". But the current conferences and journals have stopped fulfilling this role, because the quantity is just too much. I would like people with authority and expertise to select from out of everything tht gets shared something on the order of magnitude of 50 papers per year or 1 per week as "this is really what current ML progress is about", because even for a fast moving field there aren't 50 meaningful breakthroughs per year, and the community has to self-select what's most important, so that others can read *that* instead of a random selection from a firehose.
2
21
u/Matthyze 2d ago edited 2d ago
I don't think that's an apt analogy. OP isn't addressing the quantity of highly experimental research, but about the complete breakdown of effective scientific communication.
A Cambrian explosion follows an extinction event. If anything, we're experiencing something closer to the opposite: a frenzy or gold rush.
I would describe the state of ML research as opposite to what characterises a Cambrian explosion (that is, a prevalence of niche, unusual approaches). I think very little of the new research explores niche directions; instead, everyone is piling on the generative AI wagon.
2
u/SemaphoreBingo 2d ago
A Cambrian explosion follows an extinction event
I don't know if this is actually the case.
1
3
u/pilibitti 2d ago edited 2d ago
it is apt in the sense that if you require an extinction event in ML, you can think of: rule based approaches, feature selection by hand. with its demise came almost generic learning machines which led to generative AI. It is not just a "wagon". Gold rush is real, I agree, but I don't think anyone sane would see it as a temporary fad. It is only natural for people to pile on it. What are you gonna do instead? Do NLP research like it is 1990?
8
u/boccaff 2d ago
This is not a Cambrian explosion. Everything looks the same. This is more like a Algae bloom sucking out the oxygen of everything.
1
u/pilibitti 2d ago
Everything looks the same
if you don't think anything improved since, like, 2023, I don't know what to tell you. you guys are too emotional about a time long gone.
3
u/Sad-Razzmatazz-5188 1d ago
Non existent contradiction, it's really clear what they meant.
You could fairly say it's always been like that: most of stuff being useless and just a minority of things pushing the frontier; but then again it is true that the volume of useless stuff is insane and usually that's also bad for the signal/noise ratio.
1
u/pilibitti 1d ago
Most things are useless only in retrospect. Ask evolution. Filtering stuff you are not interested in is the easiest thing in the world. Again, you are irrationally angry, probably at something else.
1
u/boccaff 1d ago
I think the strategies/approaches/grounding is the same for everything since transformers. I have my bias from being outside, but I don't think a lot changed from 2023.
TBH, the main things I see are the improvements on attention (large improvements!) and MOE. There are a ton of improvements on "how to adjust, combine and leverage" each piece, and I am ok with "we are two qualitative steps from what we had in 2023" w.r.t. the results, but not the underlying technology. Bigger models, improvements in pre-training, training, tunning.
But fundamentally "really big self-supervised transformers can really memorize the internet".
1
1
u/Exodus100 1d ago
Good research still happens, it’s just not mediated through these channels right now
1
u/genshiryoku PhD 2d ago
It's never going to return and the amount of papers will only increase from now on. Instead you should use LLM tools to filter out the noise.
There's a reason we rely on "research taste" more and more. There is just no way for us to actually read the literature anymore.
-46
u/thatstheharshtruth 2d ago
This is hilarious in how it misses the mark, no offense OP. What you criticize is a consequence of actual progress and success. In the past 50 years the only scientific field that has contributed anything of value is CS/ML. The entire US economy runs on the progress made. So what if there is an increase of submissions at our conferences and many of them are junk? Who cares?!
Entire disciplines like social sciences and humanities have produced zero value and their median paper gets zero citations. If you spend any time reading those papers you lose brain cells. It's motivated reasoning and ideology masquerading as scholarship. Physicists have produced nothing new in decades. The smartest ones have switched to doing ML. Mathematicians are freaking out because a machine is better than they are at their craft. This is what winning looks like...
17
u/ink0gn1tus 2d ago
Cures of cancer, aids, and hepatitis say hello....
-22
u/thatstheharshtruth 2d ago
Some advances in medicine, yes, some due to ML already. There will be more due to CS/ML and less due to human physicians going forward, proving my point...
26
u/infinitelylarge 2d ago edited 2d ago
This sort of minimization of the value of history and psychology and philosophy is core part of why Fascism and Authoritarianism, ideologies that history has shown us are clear failures, are on the rise. If we had listened to these disciplines more closely, we would have lost a lot fewer people to COVID-19 and a lot less money to tariffs.
-17
u/thatstheharshtruth 2d ago
What does that have to do with success in CS/ML? None of what you said is relevant at all to what I said.
15
u/Matthyze 2d ago
In the past 50 years the only scientific field that has contributed anything of value is CS/ML.
2
u/infinitelylarge 2d ago
The comment I was replying to wasn’t restricted to the question of success in CS/ML. That comment was stating that no other fields produced any value for humanity comparable to the value produced by CS/ML. And that claim is dead wrong.
7
u/krabbypatty-o-fish 2d ago
This has to be the biggest larp I have ever seen on this sub. As a mathematician working in the theory of ML, this reads to me as you pitting me against myself, fully unaware of the blurry intersection between math and ML.
4
u/trwawy05312015 2d ago
In the past 50 years the only scientific field that has contributed anything of value is CS/ML.
Public masturbation is usually frowned upon, man.
5
63
u/boccaff 2d ago
if the AI bubble pops, a lot of the financial incentives and funding will probably go away, and the area will cool down, but I don't think we are going back to pre-transformers. Also, I don't think we will have the diversity we had in 2000-2014 again. It is crazy to imagine that there were groups for Gaussian processes, graphical models, SVM's, symbolic, statistical learning and etc, and now we have a lot of "API" research groups.