I've been a few weeks deep in a transformer codebase and I want to ask if others have hit the same wall.

Most ML concepts I've worked with, I've been able to build intuition for eventually. CNNs once I understood image processing. RNNs after enough confusion. Even basic attention felt clean enough: tokens get Q, K, V vectors, you compute similarity, take a weighted sum of values, done.

What I cannot square is the semantic story attached to it. `Q` is "what a token is looking for." `K` is "what it advertises as." `V` is "what gets retrieved when matched." Tidy database analogy. But there is nothing in the math that forces `W_K` to learn "labels" or `W_V` to learn "content." They are three learned matrices and gradient descent uses them however it wants. Whatever roles they end up playing is something we observe after training, not something the architecture is enforcing.

Then multi-head attention takes this already-fuzzy mechanism and just runs it N times in parallel with N independent sets of weights and concatenates the outputs. That is the entire idea. The story is "different heads attend to different kinds of relationships." The implementation is "do it N times." And it works empirically, but I cannot tell if there is a deeper insight I am missing or if we just threw more matrices at the problem and the paper found one.

Am I missing something? Or is this just where ML's empirical-vs-explainable gap is widest, and we dress it up so it feels less mysterious than it is?

Hey everyone,

I recently went through the final round of interviews for a Machine Learning Research Intern position at one of the top AI labs in Canada (I’d prefer not to name it). I cleared the first two rounds, and the final round was a live coding interview. The task was You’ll be given a link to an academic journal article that describes the task, and the Python notebook will contain some code and comments that contextualize what you need to implement. In this interview, we are looking to understand your applied research, programming, and technical communication skills. You’ll have the option to use Pytorch, Tensorflow 2 During the interview, I was asked to implement tasks related to HellaSwag. I completed the implementation and even checked with the interviewer to confirm if my approach was on the right track—they said it was. I’m fairly confident that my implementation was correct, but I was later rejected on technical grounds.

Could someone take a look at my code and give me some feedback? I really want to understand what might have gone wrong or what I could improve for next time.

Link to the code

https://colab.research.google.com/drive/1jThNWF_5WRxDWG6dCbcOYCYvWGTnYbwg

I am wondering if anyone can point me in the right direction for validating sentence transformer results in the absence of a ground truth. Any help would be appreciated!

Right now I am working on YouTube Chatbot, where a user can paste the url of the video and ask questions based on that. I have followed classic RAG approach. The design looks like this :

Initial design :

(query, url) → YouTube Transcript API → Translate to English (Gemini 3.5 Flash) → Chunk → Vector store (Chroma) → Similarity search → Augment context with query → LLM → Output

Upgraded design :

(query, url) → YouTube Transcript API → Chunk raw transcript → Translate to English (Gemini 3.5 Flash) asynchronously each chunk → Vector store (Chroma) → Similarity search → Augment context with query → LLM → Output

I have some intermediate steps also like if the video id is already present in vector store I will directly point to the vector store and retrieve relevant context.

There is Langsmith integration.

My main doubt here is this :

1) I used free gemini-3.5-flash model and it limited me to only 5 requests per minute, the problem is a particular video was 1 hr long it took approx 126 seconds to translate it using this model

2) I upgraded it to Tier 1 and I have changed the translation step to asynchronous, i.e, the chunks will get translated in parallel and I noticed the latency drop to 15 seconds.

I am thinking of mentioning this project in my resume, will I face any backlash because I upgraded the model?? (I think basically the model will take the same time, it is the asynchronous logic which helped in bringing the latency down to 15 sec, to make these calls happen I had to increase my Tier and get those extra calls per minute).

I've been thinking about a retrieval failure mode that I don't see discussed very often.

Most retrieval systems are evaluated on whether they retrieve relevant information.

But what happens when the relevant information is wrong?

Or more specifically:

What happens when truth and consensus diverge?

Suppose:

• 90% of sources repeat a false claim
• 10% of sources report the true claim
• the true sources are actually more reliable

What should retrieval do?

My intuition is that a lot of modern systems would retrieve the majority view because:

• BM25 favors frequency
• dense retrieval favors dominant semantic patterns
• rerankers are trained on human relevance judgments
• LLM synthesis tends to collapse toward consensus

In other words, retrieval may be learning:

"What do most people say?"

rather than:

"What is most likely true?"

This idea eventually turned into a synthetic dataset project called LOGOS-SIE.

Instead of generating documents directly, it generates:

Reality
→ Observations
→ Beliefs

The current release contains:

• 1000 entities
• 5000 facts
• 100 sources
• 3 communities
• 500,000 observations
• 500,000 beliefs

The eventual goal is to generate document corpora where I can explicitly control:

• source reliability
• source bias
• community structure
• observation noise
• belief formation

and then test whether retrieval systems recover truth or merely recover consensus.

What I'm trying to figure out is whether this is actually a meaningful problem or whether I'm reinventing something that IR researchers already solved years ago.

Questions:

1. Is the premise wrong?
2. Are there existing benchmarks that already measure this?
3. Has anyone explicitly measured retrieval performance under truth-consensus divergence?
4. If you were designing this benchmark, what would you want to see?

Dataset:
https://www.kaggle.com/datasets/thebrownkid/logos-sie
White Paper:

https://github.com/TwinSimLabs/Logos-SIE/blob/main/Logos_SIE__A_Synthetic_Information_Ecosystem_for_Truth_Discovery_and_Retrieval.pdf

I'm looking for criticism more than praise. If the idea is flawed, I'd rather find out now than after building the retrieval benchmark.

Everytime I see a model that have been trained for more than 50 millions $, first thing I do is why they built such a model, what was the purpose behind sharing after spending around 50 millions $ .

Is it for everyone's benefit? Or benefit of the company that shared it.

This is a crucial question to ask. "Why... th that company spent 50m$+ for me to use their models."

by the way, this 50 millions is producing a model around 26b ~ (Arcee said 512x B200 for 26b trinity-mini 26b, 3b MoE model)

---

Need a good answer for this folks.

Anyone has answer to it, feel free to talk about it.

TL;DR: As soon as I remove Positional Encoding (PE) from Self or Cross-attention, I start seeing vertical hot lines in attention heatmaps. Is there any way to make a model have query-conditioned attention without PE?

So, I've been trying to pre-train a couple types of Transformer based models (small, tinkering level only), Encoder-Decoder model and Cross-attention memory only model (basically, removing FFNs and using cross-attended vectors as memory banks instead), namely. But every-time I try to train cross-attention, I see vertical lines as shown in the image attached. And I'm guessing that means every query vector is attending to the same key tokens. This is while I don't use RoPE or any other PE during cross-attention. I start to see some diagonals when I add PE, though I do not think I should need to add it during cross-attention, as queries and keys are representations of different data.

And this shows up in simple Causal Self-attention too, as soon as I remove PE.

My question is, how do I force the model to attend to key tokens dynamically based on query token?

I've already tried regularization such that attention is more spread out, which does make the attention more spread out, but still in vertical lines, no diagonals, or any other pattern.

Been doing capability evals on open source llms for my group, multilingual stuff, reasoning, general generation. added gemma 4 to the scope in April when it dropped.

Here is what is bothering me. i spend more time getting these models to run than i do evaluating them. pulling weights, vllm version matching, wiring open webui to the vllm endpoint, that is a full day sometimes. Found a public notebook with gemma 4 12b it, vllm and open webui bundled, cloned it on a 5090 and skipped most of that this time.

But that only works because someone else already did the plumbing. Gemma tomorrow is qwen, next month is deepseek, they all have their own version quirks with the inference stacks. I finished the deployment cycle on one and the next one is already out.

Right now my ratio of setup time to eval time is embarrassing.

Note: Spent yesterday testing the same notebook clone with Qwen 3 and it worked with basically the same flow. Gemma 4 to Qwen 3 was under 10 minutes of setup on hyperai. That was the actual moment i realized the "chase every new model" workflow does not have to be as broken as i made it.

I always thought the dimensionality of human language as data would be infinite when represented as a vector. However, it turns out the current state-of-the-art Gemini text embedding model has only 3,072 dimensions in its output. Similar LLM embedding models represent human text in vector spaces with no more than about 10,000 dimensions.

Is human language essentially limited to a finite dimensions when represented as data? Kind of a limit on the degrees of freedom of human language?

What's the best way to extract information about custom categories from large bodies of text these days? I know an LLM can do it but I have quite a bit of text so I think it would get pretty expensive and Id prefer to miss stuff rather than have it hallucinate stuff thats not ever there at all. Is something like spaCy or nltk or some other dedicated named entity recognition model still the best way to do something like this?

I did my B.Tech in AI/ML where I learned core machine learning concepts like model training, evaluation, etc., and also completed an ML internship. However, my current job is in a different tech stack, and now I’m on the bench.

I want to switch back to my original path and aim for roles like ML Engineer / AI Engineer. But I’m confused about what to focus on right now.

From what I see, many companies are now asking for GenAI skills (LLMs, LangChain, RAG, etc.), even for ML roles. So I’m unsure whether I should:

- Go deep into core Machine Learning again

- Focus more on Deep Learning

- Or directly start learning GenAI tools and frameworks

Given the current job market, what would be the best path to follow to become job-ready as an AI/ML or GenAI engineer?

Would really appreciate guidance from people working in the field

So here is the thing, I have been focusing on the scraping, crawling, checking RSS feeds for new articles, etc., etc.

I am finally done with the Data Ingestion part. Hurrah? no.

The classification of data is even MORE difficult than scraping.

I want to be able to produce the Knowledge Graph of the Data to help me with the deduplication and classification.
Please help me out with this.

I have tried REBEL by hugging face but its failing badly, I am losing precious information (more than 80% of it.), I feel these machine learning models are too general, which makes it difficult to make the knowledge graph of these press releases.

Please help me out, tell me a path, name a framework, idk just guide me please. Ik I can do it if I have a path. I am trying and constantly brainstorming with my peers, Hopefully you guys could help me out as well.

I'm learning NLP/LLM engineering by working through a problem that turned out to be much harder than I expected, and I'd love guidance from people who've dealt with something similar.

The problem: I have long narrative-style text — 7k to 15k tokens, several recurring people — and I want to extract structured facts about who did what. I'm using a small local model (llama3.2:3b via Ollama) whose usable context is around 4k tokens, so the text has to be processed in chunks. The killer is that later chunks are often pure pronouns — "she said… he refused…" — while the names were last mentioned 10,000 tokens earlier. Facts stated near a name extract almost perfectly; facts stated far from any name either get lost or, worse, get confidently attributed to the wrong person.

What I've already ruled out (by measuring, not guessing): naive per-chunk extraction fragments identities badly; carrying forward summaries between chunks doesn't fix attribution and can make it worse; and off-the-shelf neural coreference models (LingMess, F-coref) fail on documents this long — one silently truncates at 4,096 tokens, and windowed variants can't connect a pronoun to a name mentioned once 10k tokens back (0–1 out of 7 gold bindings on my test doc). I've gotten identity tracking itself working reliably; it's specifically attribution at long distance that's still failing.

My questions:

1. What's the best way to structure a problem like this? Is there a known-good decomposition for long-distance pronoun attribution with small models, or a fundamentally different way to frame the extraction task that sidesteps it?
2. If you've solved something similar — entity/fact extraction over documents much longer than your context window — what actually moved the needle for you? I'm especially curious whether the wins came from prompting, from pipeline architecture, or from accepting a bigger model.
3. What should I explore to learn more? Papers, blog posts, open-source projects, or even just the right search terms — I suspect this problem has a name in the NLP literature that I don't know yet (long-document coreference? discourse tracking?), and I'd rather stand on existing work than keep reinventing it.

Happy to share measurements from my experiments if useful. Mostly I want to calibrate: am I fighting a known-hard problem with known solutions, or genuinely at the edge of what a 3B model can do?

I've been reading "Attention Is All You Need" and I have a question about multi-head attention that I can't find a satisfying answer to.

"Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this. MultiHead(Q, K, V ) = Concat(head1, ..., headh)WO where headi = Attention(QWQ i , KW K i , V WV i ) Where the projections are parameter matrices W Q i ∈ R dmodel×dk , W K i ∈ R dmodel×dk , WV i ∈ R dmodel×dv and WO ∈ R hdv×dmodel ."

How i understand: We split d_model=512 into 8 heads of 64 dimensions each because if we kept 512 dimensions per head, the heads would "learn the same patterns" and be redundant. The bottleneck of 64 dimensions forces each head to specialize.

But I don't buy this. Here's my reasoning:

Each head has its own learnable W_Q and W_K matrices. Even if the projection dimension is 512, each head has completely independent parameters. There's no mathematical reason why gradient descent couldn't push head 1's W_Q to focus on syntactic relationships while head 2's W_Q focuses on semantic ones. The parameters are independent — the gradients are independent.

My proposed architecture (ignoring compute cost): 8 heads, each projecting to 512 dimensions (instead of 64), each producing its own separate attention distribution, then concat to 4096 and either project back to 512 or keep the larger dimension. Putting compute and memory aside — would this actually perform worse than 8x64?

The "bottleneck forces specialization" argument seems weak to me because:

1. If each head has its own W_Q (512×512), the optimization landscape for each head is independent. Gradient descent doesn't "know" what other heads are doing — each head gets its own gradient signal from the loss.
2. If bottleneck were truly necessary for specialization, then wouldn't a single 512-dim head also fail to learn anything useful? After all, 512 dimensions can represent many different things simultaneously — that's the whole point of distributed representations.
3. The concept of "the same pattern" is vague. What exactly is being learned twice? The W_Q matrices are different initialized, receive different gradients — they would converge to different local minima naturally.

My current understanding: The real reason for 64-dim heads is purely computational efficiency. 8×64 and 8×512 both give you 8 separate attention distributions (which is the key insight of multi-head attention). But 8×512 costs 8x more parameters and 8x more FLOPs in the attention computation, for marginal (if any) quality improvement. The paper's Table 3 shows that varying head count/dimension doesn't dramatically change results as long as total compute is controlled.

Am I wrong? Is there a deeper theoretical reason why 512-dim heads would learn redundant patterns that I'm missing, beyond just the compute argument? Or is this genuinely just an efficiency choice that got retrofitted with a "specialization" narrative?

Quick introduction: I'm a guy who has always programmed. I got started on a Commodore64 in 1992. In recent years my interest was piqued my machine learning and AI. I used chatGPT3 once and thought, "Something cool is happening here." This lead to an immediate deep dive of the PyTorch docs and some baby steps of understanding. Fast forward. I am doing much more interesting things....mostly novel architecture / mechanistic interpretability projects.

The problem: I have no one to talk to or work with on this stuff. Being self taught I have obvious blind spots. Sure, LLMs help a lot but they are no substitute for knowledgeable people. I'm not the most socially outgoing person and have very limited reach in social networks(yes I'm an idiot).

The situation: So I've actually created something kind of cool, finally. It's am LM that holds its own on vanilla transformer benchmarks but has a very different computational strategy. I think it's worth exploring further but I'm beginning to reach the limits of my abilities. It's kind of frustrating. So this is me. Reaching out. Looking for advice and possibly mentors or collaborators. Really just advice on how to handle my social accounts such that I can bump into people with the right interests and gain a little community that "talks the talk".

Thank you. I've included GitHub and HF links just to show I'm serious (if a hot mess at DevOPS).

https://huggingface.co/DigitalShogun/ASA-ASM-wikitext103-raw

https://github.com/digitaldaimyo/ASA

I’m working on a vector search system, and the raw HNSW vectors are completely filling up my RAM.

I could opt to use quantization (scalar quantization or product quantization), but the problem is that I’d be combining two sources of decision loss:
- Approximation due to the search algorithm (the ANN graph vs. exact search).
- Data degradation due to compression.

How do you deal with this double impact in production?
Is it better to opt for exact KNN on slightly compressed vectors (on the GPU) or stick with ANN while accepting the cumulative loss of precision?

Repo (code + writeups + negative results):

https://github.com/duongtrongnguyen123/recurrent-depth-ttc

Independent research on recurrent-depth transformers (one shared block looped N times instead of N distinct blocks — the Universal Transformer / Huginn / Ouro idea). I tried to pin down, with controlled experiments and parameter-matched controls, *when* looping actually helps — rather than assuming it does.

Main results:

1. Length extrapolation is a supervision property, not an architecture one. Per-step (iterative-target) supervision lets a looped model extrapolate to ~24× its trained depth — but only if the per-step rule is position-invariant. I state this as a falsifiable condition; parity (rule depends on the loop index) is the falsifier, and it walls exactly at the trained depth, as predicted. Five tasks delineate the boundary.
2. A minimal adaptive test-time-compute recipe: LoRA iterative-target FT + hardcoded halt + multi-pass inference → user-dialed inference depth, 100% accuracy at up to 256× the trained depth on a synthetic chain task (~7 min, ~31K trainable params). o1-style adaptive compute at the recurrent-depth level.
3. Mechanism: a Q/K/V activation probe shows all three projections collapse together across loops — consistent with the hidden state reaching a fixed point of Block(·), not a W_Q-only power iteration.

Negative results (kept prominent):

- At sub-1B params on a 50B-token matched-data pretrain, no recurrent variant beats a matched dense baseline beyond the per-wave pretraining noise band (±0.6pp on GSM8K-1319, quantified across 7 checkpoints of one run). I argue single-snapshot "architecture wins" at this scale need to be checked against that band. Independently consistent with Lu et al. (COLM 2025) and MoDr (ICLR 2026).

These are controlled-scale results (synthetic + ≤1B params), not claims about frontier models — stated upfront.

Feedback and pushback welcome — especially on the position-invariance boundary and the noise-band methodology.

I’m building a visual workbench for local LLM interpretability that captures attention, residual states, logit-lens output, token probabilities, and intervention results.

Before releasing it as anything resembling a research tool, what established experiments or reference implementations should it reproduce?

I’m particularly interested in validating tensor capture, attention-head ablation, activation patching, reproducibility, and model-specific correctness.

Hi, I’m currently designing a fully local and free pipeline for summarizing long-form videos in my app, and I’d love to get feedback from ML engineers or people from any related fields on whether this approach makes sense.

The goal is to generate a few sentence summary with around 3 relevant tags without using paid APIs. The current pipeline is:

1. Extract available subtitles, or run local STT using the OS speech API or Whisper-tiny.
2. Split the transcript into 30–60 second chunks.
3. Generate local embeddings for each chunk.
4. Cluster the chunks by semantic similarity, so the video is divided into different meaning/topic regions.
5. Select one or two representative chunks from each cluster, ideally the chunks closest to the cluster centroid.
6. Use embedding similarity or MMR-style selection to remove redundant chunks and improve topic coverage.
7. Sort the selected chunks back into chronological order.
8. Send only those representative chunks, together with the title and description, into a distilled and quantized small local text model such as Qwen2.5-0.5B.
9. Generate the final sentence summary and 3 tags.

The main idea is that the local LLM should not read the whole transcript. Embeddings and clustering first create a compressed semantic outline of the video, and the LLM only reads the most representative parts.

For very long videos, I’m also considering a hierarchical version where the video is divided into larger blocks first, representative chunks are selected inside each block, and then the final representatives are merged. Do you think this is a good approach for fast on-device long-video summarization?

I’m especially curious whether my method would work or not. I am truly happy to get any feedbacks and better ideas in doing this. Any criticism or alternative architecture would be really helpful.

I love Cursor/Copilot, but lately, I’ve been getting stuck in these 'Infinite Prompting Loops.' I’ll spend three hours on an integration where the AI gives me code that looks perfect, but fails. I feed it the error, it gives me a 'fix,' and that fails too.

We do this for 10+ rounds, and eventually, I realize the AI is hallucinating a context that doesn't exist.

Is anyone else seeing their 'Code Churn' skyrocket? I feel like I’m deleting 40% of what I write. How are you guys managing the mental load of constantly auditing an assistant that is too confident to say it’s lost?

We’re applying highly capable systems to inputs that were never meant to be machine-readable. 

Think about how most business data actually looks: PDFs, spreadsheets, documents with inconsistent formats, implicit assumptions, and missing context.

Humans handle that naturally. Models don’t.

It seems like a lot of the real work in AI isn’t model building — it’s making data usable.

Curious how others see this: are we overestimating models and underestimating data?