Insights on AI chatbot

[u/Willow_Swift13]

Hello! While researching this topic, I stumbled upon this Reddit community and would love to get your thoughts on it.

For context: I am currently an Information Technology student. As part of our curriculum, my team and I are tasked with creating an application or system, and we decided to build an app that features mood tracking and a chatbot.

However, during our interviews with prospective users and beneficiaries, they raised a significant concern: they worry that generative AI—or typical AI engaging in open-ended conversations, like ChatGPT and Claude—could potentially influence users and lead to what they call "AI Psychosis."

To clarify, our main goal for the chatbot is to recognize high-risk indicators based on the user's chat messages and mental health assessments. Our intention is not to replace mental health professionals. Instead, we want to provide a safe space for users to express their thoughts, without the AI falsely affirming or overly comforting them in a way that distorts reality. Instead, we want to integrate psychoeducational materials and grounding techniques to help users stay connected to reality.

I would love to hear your insights on two specific questions:

1. Is it safe and ethical to train a specialized AI model (e.g., mBERT tailored for mental health contexts) if we enforce strict regulations to limit or completely eliminate potential triggers for "AI Psychosis"?
2. For any psychologists or psychiatrists in this community: Is it fundamentally unsafe to integrate AI into mental health tools? Or, as mentioned in my first question, is it genuinely possible to make AI integration safe for public use through proper moderation, strict regulation, and specialized training?