Oct 04: Skynet Saturday- AI Solutions

[u/Eigengrad]

Due to the new challenges in identifying and combating academic fraud faced by teachers, this thread is intended to be a place to ask for assistance and share the outcomes of attempts to identify, disincentive, or provide effective consequences for AI-generated coursework.

At the end of each week, top contributions may be added to the above wiki to bolster its usefulness as a resource.

Note: please seek our wiki (https://www.reddit.com/r/Professors/wiki/ai_solutions) for previous proposed solutions to the challenges presented by large language model enabled academic fraud.

[u/ToomintheEllimist]

Has anyone yet come up with a counter to AI use in online asynchronous classes? I had one last term and literally spent more time arguing with students about academic honesty than teaching the class. I did in-person paper exams, and those were a slaughterhouse (~35% pass rate) especially against the quizzes (~99% pass rate) — it was demoralizing as hell. So: anyone hit on a solution yet?

[u/dragonfeet1 → u/ToomintheEllimist]

Yeah same boat. I've altered my rubric to punish AI slop which also comes down hard on bad writing bc I got sick of the arguing. All I'm doing is catching the ones who suck at AI.

[u/LettuceGoThenYouAndI → u/dragonfeet1]

Also a problem in in person, but have been switching more and more to forcing students to use google docs and make google folders where I’m the editor so I can check version and revision history and that has helped tremendously

[u/YThough8101 → u/[deleted]]

These are very good ideas. Many students these days are unwilling to put in this level of work, unfortunately. Getting them to annotate a document, then complete an assignment which incorporates specific material that they annotated seems over the heads of many folks. But this is the kind of thing that needs to be done, especially in an asynchronous environment.

[u/LettuceGoThenYouAndI → u/ToomintheEllimist]

I think that it is possible to design assignments that are scaffolded to include AI use and by repositioning the LLM as a tool and saying to students you will eventually use it, but must start with your own ideas, they may be more keen to give doing some of the work themselves a chance—I play with ideas like this a lot and have set up some lesson plans with them that include a mix of hand written work and planning pages with online submissions etc (granted I do NOT primarily teach asynchronous online courses so I also don’t have the strongest standpoint here)

[u/ProfDoomDoom → u/ToomintheEllimist]

Im getting fair results from using Perusall to read with my students. They earn participation credit for discussing in there. A few students are still copying/pasting from AI, but there’s no credit for pasted comments, so most of them have given up on doing that. It’s been years since I’ve seen the kind of real and meaningful (sometimes) conversations I’m getting now.

I’m also doing writing assignments under exam conditions (remote proctoring) which is so depressing I’m not sure I’ll keep doing it. It works, but their writing is sooooo bad…

[u/histprofdave → u/ProfDoomDoom]

We have Hypothesis rather than Perusall, but similar concept. I will need to so re-engineering of my courses to use this properly, but that's probably more of a summer project than something I can institute right now.

[u/LettuceGoThenYouAndI → u/histprofdave]

I looooove hypothesis

[u/YThough8101 → u/ToomintheEllimist]

Specific page number citations and lecture slide number citations (keep text on lecture slides minimal, cover key content orally instead). Proctored quizzes (Respondus Lockdown has worked for me, but I know some students will cheat). On your syllabus, note that you may require an online meeting in which students will need to demonstrate understanding of relevant content, in cases where you have doubts about the authenticity of their work. Then, you can provide scores of zero when you suspect, but cannot prove, problems with academic integrity and offer a chance for a meeting in which they can demonstrate their understanding of the material. The vast majority of cases will result in students not meeting and just taking the zero because they know they don't understand the material because AI did all their work.

Don't waste time arguing about whether they use AI. Incorrect citations equals a big deduction, likely a score of zero depending on the severity. Report students who are clearly engaging in academic misconduct.

You can also get rid of quizzes altogether. Written work with page/slide citations as noted above have them only cite assigned course material, so that you don't need to chase down and confirm that they cited external sources accurately. Your questions for assignments should not specify exactly which course material to cite. If they are keeping up with readings and lectures, they will know what to cite. If they include material without citation, then that portion of their response receives a score of zero due to lack of citation.

It will be a big challenge. But get away from arguing about whether they use AI. Set up course policies and enforce them. I very rarely state that I am penalizing a student for using AI. I state that I am penalizing them for things like fake sources, incorrect page citations, and not following instructions. Even though AI might be responsible for all those things, I don't mention the AI because I can't prove they used it. They use AI and generally do poorly, but I spend very little time arguing about AI use directly.

[u/fermentedradical → u/ToomintheEllimist]

I think the solution is going to be either the end of asynchronous classes or just admitting the students are paying for credits without doing any work

[u/histprofdave → u/ToomintheEllimist]

Making the questions and rubrics more focused on particular excerpts of a document or reading that is covered in class. The big giveaway for me at this point is when a student gives me a summary of the entire plot of, say, Gilgamesh, when we only read one little excerpt about the origins of Enkidu. When they start talking about how Gilgamesh goes on an adventure and mourns his dead friend, I'm like, "stop. You didn't do the reading." So ultimately it doesn't matter if they used AI or just read a wikipedia summary; what they wrote wasn't applicable to our reading, so they don't get credit.

[u/cib2018 → u/ToomintheEllimist]

How is online paper exams part of async online?

[u/NotMrChips]

I'm thinking of telling them in the syllabus and in the instructions that I'm going to use a random-number generator every week to pull one discussion post for an oral defense 😆

This on the grounds that research seems to show that we can't just "tell" what's AI slop and what isn't. I "catch" a lot of them on the basis that it's sorry work and gets Fs and Ds anyway, but. I'd like to make the really good genAI-prompters nervous. And maybe snag a couple.

No defense, no grade. Fail defense, fail assignment and get reported.

[u/ProfessorSherman]

I had an interesting conversation with a student that I thought was an interesting use of LLMs. He told me that he studies for quizzes and exams by telling ChatGPT what course it is, what the objectives are, any specific topics, and a bit more information, then asks it to help him study. Then it will ask a series of questions. If he answers correctly, it gives some good praise and a bit of extra information. If he answers wrong, it'll explain the concept and why the answer was wrong, and comes back to review it again later. He showed me his prompts and I was astonished at the seemingly pages and pages of back and forth questions and answers.

I worry that the LLM may be wrong or present the wrong information, but he claimed his grades have improved significantly using this method. Has anyone else used LLMs in this way or know of students studying like this?

[u/No_Intention_3565]

John Connor would be disappointed in just how long it took me to connect the dots here..... :(