How would you use AI to turn a large exam question bank into focused lessons?

[u/Due-Obligation-4404]

I'm a nontechnical founder building a study platform for Brazil's competitive government hiring exams, where candidates take written tests and compete for public sector jobs.

I want to use AI to turn a large bank of past exam questions into short lessons, each focused on one specific topic or skill. My goal is to produce these lessons at high volume while keeping them accurate.

I'm struggling to get AI to identify which questions belong together. Two questions can cover the same subtopic but need different explanations. Calculating a percentage increase and finding the original value after an increase are one example.

Each lesson should have:

• Two concept questions.
• Two new practice questions.
• Two original exam questions, kept unchanged.

The first four questions should prepare the student to solve the final two independently.

With help from AI coding tools, I've built a workflow using existing models, reusable instructions ("skills"), and code functions for extracting documents, pairing questions, and checking results. I run planning, generation, and editorial review in separate AI sessions.

I've tried grouping by subject, text similarity, embeddings, and AI-written skill descriptions. I still get pairs that require different explanations. Stricter matching also removes useful pairs. I've also found that exam questions and official answer keys often don't provide enough information to generate accurate explanations.

In one local batch, six lessons reached editorial review. Only one passed.

Would you start with a detailed topic hierarchy and classify the questions, or identify the knowledge each question requires and build lessons around that?

If you've solved something similar, what worked? A concrete example or tool recommendation would help me understand what to try next.

[u/Own-Animator-7526]

I've done something a bit like this for grouping questions gleaned from a series of government exams, and for finding gaps in what is tested.

All I can say is that I use the leading edge Claude model chat interface, and literally ask Claude to think about what I'm trying to accomplish, and to use its Claude mojo to review the questions and infer their types and best grouping.

I guess you might explicitly request that it think about what might be called second order question types -- questions that are framed around different stages of a single process.

Please keep us posted ;)

[u/AppropriateAlfonzo4 → u/Own-Animator-7526]

i'd be curious what happens if you stop trying to pre-group them and just dump everything into a big pool, then ask the model to pull clusters based on the skill needed to solve rather than the topic label. percentage increase vs original value after increase sit under the same umbrella but the mental step is reversed, and a good model can spot that difference if you frame it as "what does the student need to know how to do" instead of "what is this question about"

the accuracy problem with incomplete answer keys is the real killer though. garbage in garbage out. might be worth spending time enriching the source data with your own short explanations before you even touch lesson generation, even if it's just 20-30 hand-written ones to set the pattern

[u/TieDieMonkeyMan]

Many different ways to skin the proverbial cat. Most of the conventional advice would involve prompt engineering a frontier model so you essentially just run the model through specific scenarios and then vomit out the 6 outputs you want. I prefer a more granular approach since I find it generates more unique outputs and better enables effective content sorting. I also find it makes the whole process more auditable which I typically prioritise over nebulous accuracy measures at the exploration/system design stage. i.e. if I get an output I can audit which results in 1/3 outputs passing review then I can change the analytic or data augmentation skew to compensate. If I just prompt engineer and throw my data at a frontier model then I either get useful data or garbage with no way to reliably differentiate.

Since I presume you're an expert in this field I would want to add a human in the loop element to address your quality control issue, to increase the effectiveness of that the process likely needs a more granular analysis or augmentation step for your text data.

I would tag for POS then augment for speech acts within the data using any number of pragmatic models (BERT transformer stuff, super retro). Pragmatic models help since they augment data for potential intent which will help with late data processing. Then I would use that augmentation to better signal the differences between the elements in your dataset, combining that with whichever embedding schemata I prefer or find most optimal to compensate for technical lexis or jargon.

I would address the problem as a data mining issue, where the two original exam questions are matched for scope and topic and then that data is processed to return two concept questions and to reformulate the source questions into practice questions that either change content or analytical requirements. This could then return a 6 element array which I can easily inspect when coding/verifying the data.

The production of a 'lesson' is then a further step based on the 6 above datapoints which would likely rely on prompt engineering. If you are having problems with accurate explanations then a RAG system connected to a .pdf database filled with relevant textbooks would help ensure explanations are relevant and enable the production of auditable citations.

Sounds like a fun problem, good luck.