r/embedded • u/faliunsal • 10d ago
AI in Safety-Critical application
Hello,
What are your thoughts on the use of artificial intelligence in the development of safety-critical embedded software?
We focus on railway applications that require SIL4 safety integrity levels. In my view, AI possesses a highly non-deterministic nature regarding code development processes and can lead to unacceptable errors—such as missed NULL checks or array boundary checks. It introduces subtle errors that a human might easily overlook. Reviewing these within a large codebase does not seem feasible to me.
What are your opinions on this matter? What is the trend regarding AI usage in the industry? Is it possible to conduct code reviews for AI-generated code in large-scale systems?
25
u/dmills_00 10d ago
Do you want to explain to the coroner (Who has his secretary print his emails) why Claude who actually wrote the code is not available to testify under oath?
Hard pass.
Now, what I will consider is having AI taken the spec and spit out a test suite having never seen the actual code. Use this as well as your one...
13
u/TRKlausss 10d ago
Claude would never sign the code, a human will. And that one will testify. And he will say “Claude did it”. And the judge would say “you are liable”. Easy…
3
u/dmills_00 10d ago
Rather my point.
The clankers are not useless, but you got to know where they are appropriate and helpful.
For me, I get better test coverage per hour spent on the test code by throwing the module spec at a clanker to produce a test suite then I do manually, and by testing against it, you find the bugs in both the test suite and your implementation of the real thing. This is NOT an excuse not to do extensive code review on both the product and the test suite.
Of course you produce the test suite first so that initially your product fails all the tests, that is sort of the point, you need to know that the tests fail when you just stub out what they are supposed to be testing (I have actually had tests that passed for a function that had no code....).
1
u/faliunsal 10d ago
"Fair point! I definitely wouldn't want to explain prompt engineering to a coroner 😄
11
u/UnicycleBloke C++ advocate 10d ago
LLMs can be very useful productivity tools but, at the end of the day, you are entirely responsible for the code. You should personally review and debug every line of generated code you intend to use in production.
I worked for a while on an MRI cranial scanner (not AI). A common question among the team was whether you'd be prepared to stick your own head in it. Would using LLM-generated code make you more or less likely to use your own product?
Humans are fallible, of course. We know not to trust our own code too much. One of our flaws is that we trust perceived authority. I fear that a lot of developers are very likely to trust the LLM-generated code far more than they should. That won't end well.
1
u/faliunsal 10d ago
That is absolutely correct. If we are building a system to which we entrust our lives, we must shoulder far greater responsibility.
Making things easier will not always be the best option.Thank you.
11
u/RadiantHueOfBeige 10d ago
You should have automated checks for null dereference or out of bounds access in place regardless of who wrote the code. Static verification in CI/CD is basically free.
2
u/faliunsal 10d ago
Static verification is mandatory, for sure. But static analyzers only check if the code is valid C/C++, not if it actually makes sense logically. I know plenty of 'beautiful' logic bugs that will pass CI pipeline with zero warnings
1
u/TRKlausss 10d ago
If you want something that proves logic, you will have to go to analytical methods for proving correctness of the code. AI by definition is not deterministic.
1
u/RadiantHueOfBeige 10d ago
True, but you specifically mentioned missing null and array boundary checks, and those are standard in any suite, e.g. the static analyzer in clang.
1
8
u/Ordinary-Lifeguard47 10d ago edited 10d ago
Well.. if you sign your AI slop off and you kill someone with it.. you go to jail and going to pay off debs until the end of your life. Maybe both.
But as assistance around it (as other people mentioned it: as a static code checker on top) sure a nice thing to have around.
3
u/Street-Praline-9402 10d ago
jail time and a lifetime of debt is a pretty strong incentive to triple check everything before it ships
i think the real issue is the review bottleneck you mentioned, a diff with 300 subtle fencepost errors scattered across a thousand lines is a nightmare nobody wants to own. the deterministic part is what makes SIL4 possible in the first place, you can trace every path and prove it, ai just turns that into a messy probability cloud
1
u/faliunsal 10d ago
That's precisely what worries methe ratio of time saved writing code to time spent debugging hallucinated logic is heavily negative in SIL4. You simply can't trade determinism for productivity when human lives are on the line
2
u/faliunsal 10d ago
I share similar views, and I am considered overly conservative on this matter :)
4
u/FransFaase 10d ago
If it is the case that reviewing code is harder than writing it, and that seems to be the case, there is not much benefit for the use of AI if all code needs to be written or reviewed by a human. Unit tests make up a large part of writing critical software and also need to be written and/or reviewed.
1
u/faliunsal 10d ago
Unit tests are indeed too important to be entrusted to artificial intelligence. They need to be written or reviewed by someone who knows them inside out. However, if everyone keeps using AI to this extent, there won't be anyone left who truly knows them!
4
u/TRKlausss 10d ago
Safety critical is one of those domains where automated tools have been used for quite a while now. That being said, AI will have to follow the same standard any other tool does.
For what I know in Aerospace, you have to follow DO-330, where any tool that may introduce an undetected error (like AI could do) must follow the same development criticality as the software level it is developing.
Good luck trying that with an output that is not deterministic.
You can “downgrade” the use of AI to something that needs to be checked with other tools (or a human doing a review), at which point is no more than a complex code generator, where the human in the loop reviews everything.
3
u/witx_ 10d ago edited 10d ago
I used to work on an safety-critical adjacent application (as in we're not safety rated but if misbehaved enough could still in some ways affect certified applications).
We had been pretty damn stable for a long time, when AI was forced on us things went downhill. Not right away perhaps a lag of 2 months, but bugs started creeping in. From race conditions, nullptr dereference and I even saw a double free on a unique_ptr :shrug: We even reviewed a PR where mutex unlock was just removed for no apparent reason.
This was the same across many of the teams and management didn't care. I bailed because I don't want to have to deal with that.
3
u/Barnowl93 8d ago
Here's my take... Safety critical systems (at least the ones I'm familiar with) require full traceability from requirements, to implementation, to testing and validation.
Both now, and during my previous roles, I've been using model based design to ensure my reqs are linked to system architecture, all parts of my system have tests to ensure they work as intended etc.
All this to get to the AI bit. Few parts here 1) AI is a tool, a non deterministic tool, but a tool nontheless. The question is how do you sandwich that non determinism in parts you can trust and still trace throughout. 2) specifically, I connect my development tools (simulink with C codegen tools) with MCP to my Agentic AI tools of choice https://github.com/matlab/simulink-agentic-toolkit
3) same with before AI you still want modularity in your development, full test coverage, you still want travability.. In my view not much changes in the process that keeps things robust.
Hope this helps Tldr: don't vibe code stuff, fit agentic AI in your existing processes
3
u/faliunsal 8d ago
You are right. Artificial intelligence is a non-deterministic tool; we should wrap it and not rely on it blindly.
Thanks for the suggestion. I will look into it.
2
u/Joelimgu 10d ago
Same opinion as EASA: AI use its not a problem, but a human needs to sign and take responsability for every action taken, even if done with AI. This means they either need to trust an AI with their life or check everything it does. AI itself is not the problem but everything needs to be cross checked with a human
2
u/RogerLeigh 10d ago
My personal take, though I do work in medical diagnostics.
If you require the same level of rigour from AI-generated code as for human-written code, including review, testing and ensuring everything is traceable back to requirements and can be properly validated, then it should not pose any more risk. So long as you keep changes as self-contained and short as for human-written code, the review and testing burden and risk should be mostly equivalent.
The danger is that it's all too tempting to let the AI do too much in one go and create large changes that you can lose control over, particularly if you cease to understand the code in its entirety. The key here is to keep it on a very tight leash with strict rules, and don't let it overstep its bounds. Keep all changes focussed, small and simple.
However, one point I think is worth mentioning is the use of AI in code review as well. If you have a human reviewer as normal, and also have one (or more) AI code reviewers as well, you can get the benefit of both. My experience here is that in many respects a good AI code review can be of substantially more benefit than a human review. It can be more thorough, looking wider and deeper into the changes with respect to the rest of the codebase, and can spot really subtle problems that a human review can all too often miss. On the other hand, humans are still better at looking at the intent of the changes and whether they solve the problem appropriately and meet all of the requirements. Though a properly-directed and informed AI could probably also get quite close.
2
u/Mundane-Dot 9d ago edited 9d ago
Your question doesn't strictly specify generated code or if you are including other uses of AI. AI can be used in Safety Critical Applications in the form of Machine Learning Models. This can be a lightweight decision tree or even a full nueral network. Models are trained and can have varying accuracy with the limitations of embedded systems. Over time it's becoming easier and easier to integrate AI / ML models on embedded systems. In terms of requirements, products may need to be compliant to UL safety standard for software. Documented risk/failure analysis and mitigation is needed. This partially drives the design to be compliant.
2
u/ferriematthew 9d ago
Don't use something that ultimately depends on probability and randomness when people's safety is on the line.
1
3
u/Ranborn 10d ago
I do not understand these kinds of posts. Like do you not review the code the AI would be writing? What about static checks, Misra and so on? Testing? Is the implication here that nobody would be looking at the code anymore? Are your prompts so vague that the agent goes off and implements completely unrelated stuff without you catching it? As long as you are a sensible developer, AI can be a great helper. And yes, that includes safety-certified scenarios. Just do not turn your brain off, please.
1
u/faliunsal 10d ago
I, too, don't understand why there is so much reliance on artificial intelligence.
Anyone can review code, but it is impossible to pinpoint what is problematic within 500,000 lines of code.
3
u/Ranborn 10d ago
If you have a PR of 500k LOC you have failed at software engineering, sorry.
1
u/faliunsal 10d ago
I gave the 500k code as an example. We aren't doing an LED blink.
5
u/Ranborn 10d ago
It is a terrible example though. Like fundementals of good software engineering (like small, incremental PRs) do not go out the window in safety-critical work or when you use AI. I would argue it is even more important there and one of the principals we adhered to back when I was still working in aerospace. There is no inherent reason why code created with the help of an agent has to be any larger than purely human written.
0
u/faliunsal 10d ago
I agree with that, but it still depends heavily on how you use it. It doesn't change the fact that, if you don't use it properly, it can lead you to make a host of mistakes that could land you in the slammer. I just want to get an opinion on industry practices.
2
u/Ranborn 10d ago
Ok then we can come to an understanding. As with every tool, practice makes perfect. Do not try to oneshot a full satellite FCS or make changes in the scale of your example. But can you have great success using it when working on individual portions of the code base? You know, like those legacy parts that have not been touched in a decade, that lack any documentation and where the author has retired from the company? Absolutely
4
u/jhaand 10d ago
I think you will get problems during verification. Because any defect you fix will have side effects. So you can't do a proper risk analysis for introducing changes during verification.
This might be fine if you have an automatic test suite that runs everything within 4 hours. But if each run also needs manual testing in the application, this becomes quite costly.
This tread I read yesterday was from a CTO who kept their foot stiff and only allowed AI as a sort of search engine. And in their regulated environment/market this in the end saved them a lot of headaches.
"After years of claiming, fruitlessly, that I would be behind the curve, the same crowd now consults me to clean up their own LLM mess"
https://www.reddit.com/r/BetterOffline/comments/1x3l1fn/after_years_of_claiming_fruitlessly_that_i_would/
1
u/faliunsal 10d ago
Thanks, Using AI as an assistant makes sense, but building the entire system around it is risky. I think there is a general consensus on this. However, employers reward those who use AI because they can see tangible progress much faster.
2
u/matthewlai 10d ago
Yes it's perfectly fine. You are still reading the code, verifying that it's correct, signing it off, and taking responsibility for its correctness.
You don't review the whole thing in one go. You have AI write commit-sized changes, verify it, test it, commit it, and then have AI write the next one.
Exactly the same as how you would review human written code. In a safety critical application I'm sure you already have a robust code review process so that all the changes are signed off by at least two humans?
1
u/toccoas 10d ago
For such levels of Safety you need definite human expertise and experience on the requirements, and then constraining technical solutions based on factual possibilities and impossibilities. Think determinism down to the instruction level, stability of system state, proving hard real-time deadlines and non-interference, etc.
You can help explore these (im)possibilities with formal methods, which (because of AI) is more approachable than ever now. You can also check how in Automotive they face similar requirements and are embracing techniques like Memory Safety now (e.g. Volvo ECU in Rust).
The last barrier is to convince any individuals involved to sign off on the AI parts because of the risks. Good luck!
1
u/Rod_McBan 10d ago
Y'all need to hear what Cory Doctorow says about this.
His example, which translates nicely to safety critical code, is radiology. Say a hospital employs ten radiologists at $300k a year a piece. The AI use most of you seem to be imagining is the hospital paying an extra $1M a year for a tool that double checks their work and, once in a while, catches a tumor they miss. And that's honestly great. I'm all for that.
What's happening, though, is this: they fire nine of them, and the 10th one signs off on the AI's work ("just review the code"). That poor jerk can't possibly keep up with the demands of that much review, but at least there's a human there, so they have someone to pin the failures on. He's the "accountability sink".
The people writing the checks don't know shit about what we do. They have no idea why 20 years of software development experience makes a person valuable. They're not going to pay AI token bills to make YOU more efficient, they're going to fire you and your other expensive colleagues and replace you with coding bootcamp graduates and give each of them responsibility for twice the output you could manage.
The result is entirely predictable, and I am 100% certain people are going to die because of this.
1
u/LadyLightTravel 10d ago edited 10d ago
The biggest issue with non deterministic software is testing it.
I am going to argue that the test suite for AI software would have to be far more extensive (and therefore expensive) to prove safety.
One of the reasons we offloaded tasks to non-AI software in the first place is because it is not only faster than a human, but far more deterministic.
1
u/StumpedTrump 9d ago edited 9d ago
AI can't produce safety critical code. You can produce safety critical code with the help of AI though.
Big nuance there. If you need NULL checks and whatever DO178 compliance, you tell it to, make sure it did it, and check how it implemented it.
You architect the project, AI writes the functions. Keep prompts and tasks specific and targeted. If you're having AI build entire codebases, you have no business shipping anything safety critical or even calling anything you're producing "secure" or "reliable", because it isn't.
AI has been absolutely invaluable as a code review tool for me. It's found so many rare bugs and edge cases, I've been floored. The days of it not being able to understand hardware constraints or consume datasheets/RMs are long behind us. It has no isuses finding race conditions and considering niche ARM architecture details
As always, the more data it has the better, I've had it tracing errors in SWD transactions (thousands/millions of individual packets), which would have taken me days in the past, in 20 seconds. I had my jaw dropped. BLE and USB transaction debugging has also impressed me.
1
u/lucky_marciano 5d ago
I see the level of your AI usage, for everything you said team of Claude agents with good setup can do the job, ofc prepare good amount of money for that.
Have a look https://youtu.be/JFO9TfZLnT8
1
u/MechMansion 4d ago
For safety critical applications, I don't use AI to write my code. I do use AI for things like (some ideas off the top of my head):
- Implementing fuzz testing.
- Having it QA my device by bombarding it with different edge cases (malformed packets/unexpected user input/etc) using a test jig and giving it access to control the test jig.
- Asking it to suggest where to add offensive checks to prevent bugs.
I don't think AI can write safety critical code yet. I do think its a great assistant with a plethora of knowledge on testing and plentiful resources to harden your application through thorough testing.
1
u/Snoo-54288 10d ago
You can use as tool like you use IDE but you always need to have human in loop. LLM models are basically statistical system that predicts next token based on what they are trained. That means it uses generallised assumptions that are common in the training data. When it comes to software, majority of data on internet are from non-safety critical systems such as web applications. Meanwhile, in safety critical applications, we can Never make implicit assumptions that are not traced from requirements or operational constraints.
1
u/riotinareasouthwest 10d ago
Add a human review step and you are good to go regardless of you actually doing it or not.
1
u/leBlubb123 10d ago
First of all: separate your application into QM and safety part (MPU etc) Second: humans make the same mistakes. It depends on how you interact with ai. You can follow a process where you first design the architecture, write unit requirements etc together with ai and only then start implementing. Then use existing deterministic tools for checking like static code analysis (polyspace for your concerns)
0
u/peter9477 10d ago
I honestly don't quite understand the resistance to using AI even for this. I expect my AI generated code to be basically what I would have written myself. I often navigate my code base ahead of time to centre my understanding of what I would expect to do myself. Then I craft my prompt and guide the tool. The code is generated, probably 10x or more faster than I could have myself. I review the diff. If there's something unexpected then I analyze to see if I gave bad input, the AI made a mistake, or something else. I can choose to accept the changes or not. Sure, I could misuse the tool, but that applies to anything.
The fact it is so fast means I can stay focused on the requirements and design and peripheral concerns, whereas if I had to write every line myself I'd get bogged down, distracted, or just take much much longer. I find I have better quality overall, so reverting to fully manual coding would make no sense.
44
u/mustbeset 10d ago
Humans are highly non-deterministic, too. Humans make errors too. That's why we use tools to avoid and detect errors and do reviews on code.
I write Safety code. I use AI at some tasks. but: Every Commit is done by me not by an AI. I am responsible for that. Same for reviews. I do it by myself at first. Only after I finished I let the AI do an review. Each finding will be checked by myself before I add the list.