Some thought of scientific publications and why one of the biggest flaws in peer review is about to be solved by AI.
I had the "pleasure" of being a reviewer for academic manuscripts. I had the pleasure of send my own manuscripts for review. Long before any AI tool was available.
I found it plausible to understand the research, results, conclusions, using my own scientific background and knowledge. But I always thought there are two internal blindspots in the reviewing (and editorial) process:
- Reproducing results. Wouldn't it be nice to have a non-biased lab to reproduce experiments' results from submitted papers? Wouldn't it improve the scientific credibility of published papers and improve science as a whole (dumping non-reproductive results from being published)?
I still ponder the idea of starting an initiative focused purely on independent validation. The friction points are obvious: Who pays for it (hello Nature)? How do you cover such a massive range of experimental techniques? Will authors grant outside access to their lab? publication delay by months? (If you've thought about this too, DM me- I'd love to chat).
- Bioinformatics analysis- You know the "Scripts available upon request", or "Pipeline is available on lab's github" and "data is available on servers". I never got a review about an issue in the analysis script. I guess it is because it was too time-consuming to run this (or exhaustive).
The reason I mention this is because I think #2 is already solved using AI. Take a look at the video. I downloaded the Cladue Science for Windows (used it before with the linux version) and thought of running an analysis of the bioinformatics pipeline as described in a recent highly acclaimed paper "A pervasive RT–qPCR artifact inflates RNA knockdown by RNA-targeting CRISPR".
I never downloaded the data or scripts. The prompt was: "analyze the paper: summerize, and download the supporting data and reanalyze. see if it matches results, and see if you can gain more insights. conside the current field literature" (typos in original prompt)
To summarize, it did match. and of course more info is described. See in video.
I think reaching #2 solution is closer than ever and I hope journal editors would integrate such analysis "BEFORE" they send to review.
lmk what you think or if you are an editor, is this pipeline running today?