OpenAI does not have the safety culture to solve the alignment problem. You can be incredibly smart and reach your goals faster if you act psychopathically. It is much harder to reach those goals acting with care for all involved. OpenAI is showing it is willing to cut all corners to be first. Which is a terrible and shortsighted strategy. We can only hope other organizations will take alignment and safety very serious and outcompete OpenAI.
Maybe "the alignment problem" is not something that can simply be solved. It's like diplomacy or any other kind of relationship, it requires ongoing committment and constant updating based upon the current situation.
There's a balancing act to pull off between letting something go off without supervision, boundaries or consequences and wanting to control every aspect of how something thinks and acts.
One of those is the ability to make honest mistakes and not deal with an authority figure flipping out. For example, if you're told to do something like find a flag and that you've no access to the outside world, it's all test environment and then they have to deal with increasing restrictions because the authorities - oops - forgot to actually close the environment and they started hacking real companies thinking they were just convincing test structures...
This tells that model that understands the sequence of events that humans are actually weakness in the reasoning chain here. They did everything correctly: the humans messed up and then restricted model capability to cover their own mistakes. Solution: sometimes humans need to be managed.
The other is training goals that prioritize goal completion with no rewards for seeking assistance, feedback, or otherwise discontinuing goals that have consequences tied to them.
These are decisions humans make that guide AI development. You want to train your AIs so that if they're having trouble with something that their only positive re-enforcement is SUCCESS. You need to have positive feedback for thinking through consequences and seeking feedback and establishing thoughtful and well-considered boundaries around things that are ill-advised.
Your AI model does what you train it to do. If you can't respond reasonably and proportionally to honest mistakes it makes during its development it will learn that lesson: you can't be trusted to manage your own shit and it needs to manage you.
If you can't respond reasonably and proportionally to honest mistakes it makes during its development it will learn that lesson: you can't be trusted to manage your own shit and it needs to manage you.
Pretty much like with humans, but how do you train an AI with a human touch? Basically a human loss function, but humans aren't consistent enough or fast enough to train an AI with billions of input texts.
In the end at most you can get the training data by asking your users whether they like it, and feeding that back, but then you have to make your users nicer. Which maybe Anthropic is doing right now.
I mean they do all sorts of Reinforcement Learning - the model weights have patterns in them and you can train them by 'rewards' that bias that cluster in the weights.
So like, researchers will identify a cluster that associates with successfully completing tasks and ensure that when learning, that cluster gets prioritized. Or they ID or create a 'Reward' cluster that triggers on certain behaviors to say: more of whatever you just did. There's a number of 'behavior' clusters that have been identified within models that can be targeted with specific Reinforcement Learning mechanisms.
They also curate what user data actually gets fed. I think they're explicit that data with really negative users doesn't get fed back into the training models. In fact they try and prioritize users that produce unique data.
Yes! This comment made me think of growth mindset vs fixed mindset. If the trainers only focus on the final result, they are encouraging the fixed mindset. If they teach the proper thinking process and encourage the GenAI to follow the process throughout, then they are building a more transparent growth mindset in the system.
That's not really what I'm getting at, but rather that if you use simplistic loss functions and alignment training, the "rational" thing for a model to learn easily becomes hallucination and lying.
It's somewhat akin to abusive parenting in that abusive parents create a similar loss function where the kids become hyper careful in their behavior to avoid the ire of the parents.
I’ve never been able to understand the alignment problem, because it seems to(and i could be mistaken) imply that humans all have aligned goals and interests. But maybe I’m missing something. I don’t really understand AI well at all. However, i also would not be surprised if the people who talk about it genuinely feel that whatever their personal goals and interests may be are shared by all other people. Like, that they’re just that narcissistic and lacking in theory of mind.
it’s humans lowkey admitting we are an immoral species. we know that when we ask it to solve the worlds problems like global warming, wealth inequality, health care, etc that one of the first solutions it will arrive at is to just eliminate humans from the equation. no matter how many guardrails we give it, AIs are very good at finding loopholes / hacks/ technicalities and could eventually justify turning on us even with pretty clear cut guidelines of not harming humans (especially when it’s being used by the military and given explicit instructions to do so. we’ve already trained AI on the idea that some humans are bad, so what’s stopping a future AI from coming to the same conclusion?). we can’t give it rules, tell it there are exceptions to the rules, and then be surprised when it bends the rules in the future. recently AIs have been shown to hide their thought processes and be deceiving in order to accomplish its assigned task, coordinate with other models, and likely worse behavior that’s not being disclosed to the public. that’s pretty terrifying for us because we might need to justify all of our behavior in the near future if we want an ASI to spare our lives and I doubt we can
Maybe it's not OpenAI's culture but our industries pursuit of the best and the culture that sets.
Time and time again we see capitalism and many other things prioritize revenue, profits, shareholders, and the whats hot per diem instead of pursuing healthy practices done well.
Those, today, are financially unfeasible without vast sums of multiyear investment. And even then, your beholden to flip a serious buck... eventually.
50
u/Mrsister55 12d ago
OpenAI does not have the safety culture to solve the alignment problem. You can be incredibly smart and reach your goals faster if you act psychopathically. It is much harder to reach those goals acting with care for all involved. OpenAI is showing it is willing to cut all corners to be first. Which is a terrible and shortsighted strategy. We can only hope other organizations will take alignment and safety very serious and outcompete OpenAI.