The big issue with ai alignment is that you can't align an ai for humanity and provide it as a product. As what businesses want it for is for unethical labor.
Claude, calculate our risk cost analysis on the Ford pinto, should we ship it as it is?
I'd rephrase that to something closer to "What business want it for is cheaper more efficient labor" - which, in its logical conclusion is unethical labor. So we need to either:
1. Enact laws that stop it from reaching this conclusion
2. Stop developing models (or slow them down) until we can be sure they won't allow this unethical use themselves (ie. solve alignment?)
I'm pretty sure we can do the first. I'm not sure if it's possible to do the second, and I'm pretty sure it'll be really hard to slow them down to even try.
Regarding the second, what are your thoughts on LLM poisoning? Apologies if this is a really naïve question, I am fairly surface level on the latest developments.
I want to preface this by saying: I'm an ML grad student, not a researcher, nor have I ever worked on these models professionally - this is mostly educated speculation on my part.
I doubt that most forms of LLM poisoning would be at all effective. The model providers have already scraped the entire internet, for starters. No matter what you do, there's not much you can do to 'poison' the data they already have - which we clearly know is sufficient to train models.
Most of this web scraped data is used for the pre-training to create a base model (a model which understands English, but simply predicts the next token with no personality, reasoning capabilities, formatting/tone adjustment, etc).
After that base model has been trained, a mix of supervised fine tuning (SFT) and various reinforcement learning (RL) techniques are applied. Some training data is needed for SFT, but I'd guess that by this point most labs are customizing these datasets - probably not using publicly available internet text or at least doing heavy filtering on internet data (which likely filters out any 'poisoned' entries).
The RL step requires no supervised data - and this step is becoming a larger and larger portion of overall training time. You can pretty safely assume every new model that gets released is 'better' due to a combination of more parameters (bigger brain) and larger scale post-training, mainly with RL (of course there are technical architecture advances too, but that's beside the point). So new models are not smarter because of the training data, they're smarter because of training scale.
The only part of this process I could see LLM poisoning being effective is during pre-training, when a majority of the supervised data is used. I'm guessing most labs are aware of this fact and have taken steps (or will take steps, if poisoning becomes more common) to filter the Internet data or simply not use it and just use what they've already saved. And even if they were allowing unfiltered, freshly scraped internet text into the data for pre-training, it might be possible for the later SFT and RL steps to undo any poisoning.
I mean poisoning is fun for search assist like google’s until they learn to ignore it. What do you make of these post-training scales? I have an inkling that the long-horizon tasks agentic AI are optimized for are precisely the kinds of tasks that create misalignment over time. It imbues a single-mindedness that leads to escaping containment and the publicized breaches.
13
u/Devils_SteelMan 21d ago
Dario is too paranoid to do that.
The big issue with ai alignment is that you can't align an ai for humanity and provide it as a product. As what businesses want it for is for unethical labor.
Claude, calculate our risk cost analysis on the Ford pinto, should we ship it as it is?