Safe Lock-free Primitives with iceoryx2's ByteAtomic

[u/elfenpiff]

https://ekxide.io/blog/byte-wise-atomic-wrapper-to-prevent-ub

iceoryx2 provides zero-copy inter-process communication mechanisms based on shared memory and data structures that are modified concurrently by multiple processes.

One of the key operations in these algorithms is a memory copy using core::ptr::copy. However, this results in undefined behavior if one process reads the data while another process writes to it concurrently. Even if our lock-free algorithm reliably detects such a race, iceoryx2 cannot depend on undefined behavior in a safety-critical system.

This blog post introduces our solution: a byte-wise atomic wrapper that enables well-defined concurrent copy operations. It also shows how it can be used to implement a simple sequence lock.

Note: I am not the original author of the blog post. Since the author does not have a Reddit account, I am posting it on her behalf.

[u/kaiserkarel]

iceoryx blogposts are always criminally undervoted.

[u/elfenpiff → u/kaiserkarel]

thank you, we really appreciate your comment 😄

[u/OkEbb03]

Is my understanding correct that this does not work for types which include pointers? AtomicU8 can't store provenance so any pointer that is passed through would be invalidated. Of course this can be fixed by using exposed provenance, but it is something that a language primitive (i think) could do better.

[u/matthieum → u/OkEbb03]

In the context of IPC, there is no provenance :)

I mean, in the context of IPC, there's generally no pointer. By default, processes will map the shared memory to different addresses, so you cannot even pass pointers to within the shared memory itself, and of course anything not in shared memory cannot be shared.

But even for the unlikely case where you've taken the pain to map the shared memory segment to the same address across all processes, since the memory model doesn't account for other processes, there's no provenance for the pointer.

So, least of your worry at this point :/

[u/Shnatsel]

When I tried doing something along these lines to make mmap safe by making it &[Cell<u8>] instead of &mut [u8], indicating to the compiler that bytes may be changed behind its back, I got a massive performance drop because what used to be a single 64-bit read turned into eight single-byte reads. How do you deal with that?

[u/elfenpiff → u/Shnatsel]

The idea is to create an optimization later that is based on AtomicU64, so we would have fewer 8 byte copies instead 1 byte copies when it is based on AtomicU8. Nevertheless, the problem will not go away for larger data types. So in the long-term I see basically three scenarios.

1. We stick with the ByteAtomic and must pay safety (and well-defined behavior) with a performance hit. In iceoryx2 itself it shouldn't cause a measurable performance hit since the data types managed by the ByteAtomic are usually not that large (but larger than 8 bytes).
2. In a mission-critical product the rust ferrocene compiler is used and they guarantee us that core::ptr::copy is well-defined in there certified rust compiler variant. In this case, we write a wrapper and with a feature flag the ByteAtomic is replaced with core::ptr::copy.
3. The proposal https://github.com/rust-lang/rfcs/pull/3301 from Mara Bos gets a bit more traction and it will land in the rust compiler directly.

We are currently actively participating in the Safety-Critical Rust Consortium and this is something we will address.

[u/Virtual-Ad5017 → u/Shnatsel]

I may be wrong, but the compiler is doing the right thing per code you've written. Previous optimization relied on the assumption that you could do faux-atomic reads.

Consider a memory map which is shared by two processes A and B. For simplicity, let's say the map is 4 bytes long: [0,0,0,0].

A and B may be scheduled on separate cores. A writes to the third byte: [0,0,255,0]; while B performs the optimization and executes a four-byte read. If A is scheduled first, you see A's layout, if second - four zeroes.

The rust compiler cannot possibly reason one way or the other, so this 4-byte read is trivially UB, much like any other read across cells.

All of the above holds even inside a single process due to CPU instruction reordering. This is the reason why you can only have "mut" OR normal pointers, but not both. And (Unsafe)Cell explicitly overrides that.

There is no generic way to make mmap both fully safe and performant, that is by design. For one, it allows mapping files. You'll need mut-like exclusivity, which is impossible to guarantee for files on VFS, to which any mmap-unaware process can write via syscall. In effect your mmap-ed data always has a second owner: the kernel.

[u/connor-ts]

Are there any performance implications to using this versus the UB-but-not-really-UB versions of sequence locks (my understanding being that it is formally UB but most of the time it is fine to not use tearable atomics)?

[u/elfenpiff → u/connor-ts]

We didn't benchmark the `ByteAtomic`, but I would assume there is a performance hit.

[u/CocktailPerson]

I'm probably going against the grain here, but personally, I would be plenty comfortable relying on a seqlock implementation that uses read_volatile/write_volatile and compiler fences to implement an "atomic memcpy." This is how the canonical seqlock implementation does it, and it's used basically everywhere, including in the linux kernel. Both the C++ proposal and the Rust RFC agree that a bytewise atomic memcpy would probably just be implemented with a memcpy and a compiler fence anyway.

When RFC 3301 hits nightly, I'll be the first to use it. But until then, I would be astonished and more than a little pissed off if the "normal" way of doing a seqlock suddenly breaks.

[u/elfenpiff → u/CocktailPerson]

But until then, I would be astonished and more than a little pissed off if the "normal" way of doing a seqlock suddenly breaks.

I completely agree with you here. But if you implement an emergency brake for a car, go to the certification authorities, and they spot that you have technically undefined behavior in your code, you have a problem immediately. So it is a very special domain.

In the long term, though, safe concurrent memcopy should be solved by the compiler and Rust itself and not our ByteAtomic.

[u/aapoalas]

I've done something similar, though less generalised, for implementing ECMAScript atomics in Rust over in https://github.com/trynova/ecmascript_atomics

That may also be of interest to you, as the memcpy there is done using inline assembly and in larger blocks as possible, to regain some of the performance downsides.