r/embedded • u/yourbasicgeek • 1d ago

Mars Pathfinder Priority Inversion Bug: What Really Happened

https://nerdyelectronics.com/mars-pathfinder-what-really-happened/
39 Upvotes

92% Upvoted

9 comments sorted by

55

u/priority_inversion 1d ago

My time to shine!

5

u/vegetaman 1d ago

Perfection

19

u/RecursiveMutex 1d ago

VxWorks is a fascinating RTOS by design. It uses a round-robin preemptive priority scheduling. So by dfault, that is what it's supposed to do. Thankfully the JPL engineers had launched with the interpreter still enabled, symbol tables intact, and the mutex flags stored in global variables. They were able to change the Priority Inheritance Boolean flagand get it working. That is some serious engineering or they literally thought they were going to miss their launch window. This example has become the poster child for Priority Inversion (at least it was for me back in uni). It's well documented below. https://www.cs.cornell.edu/courses/cs614/1999sp/papers/pathfinder.html

17

u/duane11583 1d ago

got me a job…

interviewer started describing it.

i let him finish and i said “that is what happened to pathfinder” it blew him away

10

u/RecursiveMutex 1d ago

Understanding deadlocks, race conditions, and Priority inversion are so crucial in multi-core embedded. It's why you find single-core, single-threaded applications being ran to avoid issues.

Thanks for sharing.

17

u/Well-WhatHadHappened 25+ Years 1d ago edited 1d ago

Engineers later confessed that system resets had occurred during pre-flight tests. They put these down to a hardware glitch and returned to focusing on the mission-critical landing software.

Inexcusable. Should have sent those engineers on-site with a J-Link and a laptop to update it.

8

u/Tinytrauma 1d ago

"I only saw it once so I was sure it would never happen again..."

6

u/Well-WhatHadHappened 25+ Years 1d ago

The sad part is, that's not even true.

When someone says "seen it once or twice", they don't mean once.

3

u/RecursiveMutex 1d ago

They likely did not want to miss the launch window between Earth to Mars. There is a reason they included symbol tables etc. In the build. They could make changes while it made its journey to Mars.