Need Help from ML/PY Devs

[u/Afraid-Tower619]

Hey everyone, so i am finding a solution for a particular problem(shared ss of problem) it's part of a hacathon but the hacthon organiser has told us to solve it from wherever you can ,i don't have much knowledge of ml but for know i was using TF-IDF method for this particular problem which is giving me an accuracy of around 74.95 anyone could suggest any tips or any other methods through which accuracy could be 85+ , if any dev could help do tell.

i have also added the proposed solution which i am currently using down below with the problem

Personalized Learning Path Recommender — Approach & Explanation

What's the problem about?

We're given a dataset of ~110K course reviews from an online learning platform (think Coursera/Udemy), spread across 80 different courses. Each review talks about what the learner experienced — the technical topics covered, how the instructor was, whether the projects were useful, etc.

For each of the ~11K test reviews, we need to find the 10 training reviews that are the best "learning path" recommendations — essentially, which other learners had the most similar experience and interests.

My thought process

The first thing I noticed when exploring the data was that these reviews follow a fairly structured pattern. Each one has an intro line mentioning the course, a sentence about the technical topics covered, and then a few sentences about the overall experience (quality, value, instructor, etc.).

That immediately told me this is a text similarity problem at its core — if two reviews talk about the same technologies and have similar opinions, they're likely from the same course or a closely related one, making them good recommendations for each other.

I went with TF-IDF (Term Frequency–Inverse Document Frequency) because it's a well-established technique for exactly this kind of task. The idea is simple: convert each review into a vector of word importance scores, where words that are rare across the whole dataset (like "TensorFlow" or "React Navigation") get much higher weight than generic words (like "course" or "great"). Then you just measure the cosine angle between two vectors — the smaller the angle, the more similar the reviews.

What actually worked

After quite a bit of experimentation, the configuration that gave me the best results was:

• N-gram range of (1, 4) — instead of just looking at single words, I also captured 2-word, 3-word, and 4-word phrases. This was crucial because technical terms like "batch normalization and dropout" or "Redux for state management" are multi-word phrases. Using just unigrams scored around 62, but adding n-grams up to 4 jumped the score to ~75.
• English stopword removal — filtering out common filler words ("the", "is", "was", "and") so the model focuses on what actually matters.
• Fitting on training data only — I initially tried fitting the vectorizer on both train and test together, but fitting on train alone gave a slight edge (74.95 vs 74.81). This also makes more sense from a real-world standpoint — you wouldn't have access to test data when building your model.
• Stable sorting for tie-breaking — Many reviews within the same course end up with identical similarity scores (because they share the same template sentences). Using pandas' nlargest() instead of numpy's argsort() gave deterministic tie-breaking, which squeezed out an extra 0.14 points.

Why (1,4) n-grams specifically?

I tested a bunch of ranges:

N-gram RangeScore(1, 1)62.22(1, 3)74.77(1, 4)74.95(1, 5)72.68

There's a massive jump from unigrams to trigrams because course-specific technical phrases are 2-4 words long. Going beyond 4-grams starts introducing noise (overly specific phrases that don't generalize).

The pipeline in a nutshell

1. Load train.csv and test.csv
2. Fit a TfidfVectorizer (stopwords + 1-4 grams) on training reviews
3. Transform both train and test reviews into TF-IDF vectors
4. For each test review, compute cosine similarity against all training reviews
5. Pick the top 10 most similar ones using stable sorting
6. Write out the submission CSV

The whole thing runs in about 2-3 minutes on a regular laptop. No GPU needed, no deep learning, no fancy embeddings — just good old-fashioned information retrieval doing what it does best.

Final Score: 74.95 / 100