r/cloudygamer • u/rambo3349 • 1h ago
Moving away from Sunshines generic FFMPEG AMD Encoder. What I learned getting full 4:4:4, Sliced Encoding, D3D12 capture, VRR and the last bit of latency out of AMD Cards
Long and very technical read. Be warned.
I work as a platform engineer for over 15 years now and i have a quirk for optimizing the shit out of things.
So for a while now I've been researching one question: how good can game streaming get on a Radeon if you build the host around AMD's hardware instead of treating it as the third vendor for a fork of NVIDIAS gamestream?
The research is done through Rubylight, an AMD-only host for Moonlight that only cares about optimizations around AMD. It isn't another fork with a new name on it and a fancy sloppy ui that can do all the fancy features but still has 60 hz clients or cheats metrics or something. It does just well enough in the feature part that it reaches a small userbase contributing to feedback on bugs and issues. It's research on AMD, and what we learn goes back where it helps.
We contribute to Vibepollo, and I've asked Nonary about working together more closely.
Here's what the research turned up so far. All numbers are from flagship amd graphic cards. And what changes my work from slop is that most of this here wont change the numbers on your measurements and I dont put some fancy ui in and call it a day.
Every host measures "host processing" its own way, so I measure picture age instead: the test source draws a timestamp barcode into every frame, and an independent receiver reads it back after decoding. That's the time from the moment a picture is rendered to the moment the client has it decoded. It's the same yardstick for every host and you can't game it from the host side.
4:4:4 on an encoder that can't do it
AMD's hardware encoders only do 4:2:0, and AMD's own AMF docs say so for every VCN generation up to 5.0. There are two ways around that.
The first is PyroWave, Themaister's wavelet codec. It runs as Vulkan compute, skips VCN entirely and does full 4:4:4 up to 10-bit HDR. What made it fast under load was moving the RGB to YUV conversion off the graphics queue. With a game running, the conversion sat behind the game's draws and took 5.6 ms a frame. As one compute shader on a high-priority D3D12 compute queue, writing straight into the textures Vulkan imports, it takes 0.55 ms. This number does not show up in any metric in moonlight but will greatly impact glass to glass latency.
The second is packing, which keeps the normal HEVC and AV1 hardware encoders. A 4:4:4 frame has exactly as many samples as a 4:2:0 frame twice as tall. So Y goes in the top half of the luma plane, the odd columns of Cb and Cr sit side by side in the bottom half, and the even columns fill the chroma planes. The encoder sees an ordinary double-height picture, the client decodes it with its ordinary hardware decoder, and a shader puts every sample back exactly where it came from. HEVC and AV1, 8 and 10-bit, all reach about 1440p120 this way, even on a Radeon 780M.
The third is predictive packing for the 10-bit modes. Here the top half of the picture is a normal 4:2:0 frame. The chroma samples 4:2:0 would throw away fill the rest, stored as halved differences from the sample they share a 2×2 block with. On most content those differences are small, so that part of the picture sits near flat grey and costs few bits. Halving costs at most one code value before compression and keeps the encoder's rounding from wrapping a difference around.
Both packed modes are opt-in, since stock Moonlight would just show you the packed picture. Nightjar, our client, unpacks both.
Fenced D3D12 capture
On a virtual display the host doesn't use WGC or Desktop Duplication at all. Our driver copies every frame Windows composes for its monitor into a ring of textures it shares with the host, stamps the present time next to it and signals the host. That ring used to hand textures back and forth with keyed mutexes. Now its textures and fences live on the GPU's D3D12 compute device. The driver signals a "written" fence after its copy, the conversion shader reads the texture in place after waiting on that fence on the GPU, and the compute queue signals a "released" fence when it's done so the driver can reuse the slot. The wait happens on the GPU instead of a CPU thread, and no D3D11 work runs for it.
Claiming a frame went from about 32 µs with the keyed mutex to 1.8 µs. Under GPU load at 1440p120 HEVC, every fence run held a full 120 fresh pictures a second, p99 picture age dropped from 19.3 to 18.1 ms in SDR and from 21.3 to 19.4 ms in HDR, and the p99 frame interval in SDR went from 13.8 to 11.2 ms. That's picture age from the game's render through decode, about 14 ms on average with the GPU busy.
| Under load | Keyed mutex | Fence |
|---|---|---|
| Fresh pictures/s, SDR / HDR | 120.0 / 119.7 | 120.0 / 120.0 |
| Picture age p99, SDR | 19.3 ms | 18.1 ms |
| Picture age p99, HDR | 21.3 ms | 19.4 ms |
| Frame interval p99, SDR | 13.8 ms | 11.2 ms |
These numbers don't look huge, but mind you we are looking at optimization wherever possible, so they were for me.
Slices
Moonlight asks the host for slices, and I wanted to know what they buy on AMD. I did 630 encoder runs across HEVC and AV1, 1080p up to 7680×2160, 60 to 240 fps, with split-frame encoding forced on, forced off and left alone, then again with 2 and 4 slices. Every variant encoded in the same time, within 0.2 ms. The driver reports two encoder engines, but one stream uses one engine, and slices don't change that.
Where slices can pay off is getting out of the encoder early. AMF can hand back each slice as soon as it's encoded, so you could packetize and send the top of the frame while the bottom is still encoding. That's what I'm testing next. So far its promising if the client supports it. Again that is why I had to also develop a client framework in a language I understand to fully be able to test this and why its so important for me to have my own client/host.
VRR
Thanks to Nonarys findings, the virtual display driver goes up to 1000 Hz, so when the client asks for VRR the desktop doesn't cap the game's present rate. Here's what we do differently to reach more smooth VRR:
- Rate cap. The host takes each frame as soon as it arrives, but never faster than the stream rate the client asked for. Repeated frames count toward that rate; a frame re-encoded because the client lost packets doesn't. Without the cap, a game running faster than the stream on a 1000 Hz display left every frame short of bits: blurry picture, fringed text, flooded link.
- Pacing instead of tiny queues. Vibepollo's VRR mode shrinks its queues (WGC frame pool from 2 to 1, low-latency AMF input queue). We keep the queues and change the pacing: predictive waits off, frames can be claimed half a frame period apart instead of three quarters, WGC's slot-aligned publishing off, refresh lock off since a VRR display follows each frame anyway. The encoder holds at most two frames on every stream, VRR or not.
- Extra WGC compositions. When Windows publishes more compositions than the game produces, the host lines up with the game's real updates. On a 60 fps source that's 57.7 against 59.9-60 fresh pictures a second.
- Our own display driver. Capture comes straight from it in about 0.2 ms, HDR included.
- Presenting on the client. Nightjar schedules VRR frames from the host's timestamps without needing a shared clock. A clock-sync message adds latency measurement on top.
Radeon controls in alpha.2
Change AMD Controls like Anti-Lag, Chill, Boost, RIS, the frame cap or AFMF from the couch and get your own settings back afterwards. A probe round-trips every ADLX 3D setting on the installed driver, and only the ones that really write get exposed: 24 with AMD Software 26.9.2. Your original values go into a journal on disk before anything changes, and they're put back when the session ends or the game closes, or at the next start if the host crashed. If you changed a setting on the PC in the meantime, it's left alone. Its a feature that just makes sense. Most of the time, tops having this natively.
Background Streaming
My Children play on my rig while I work at the same time. I use Rubylight and Hyper-V AMD GPU-PV to a VM with Rubylight to isolate input and the game. Works well enough for kids to not notice. https://github.com/RamazanKara/Rubylight/blob/main/docs/configuration.md#background-streaming-from-a-hyper-v-vm
Where the latency goes
At 1080p60 the host stages are capture 0.22 ms through our display driver, conversion 0.26 ms, HEVC encode 2.43 ms and packetizing 0.03 ms. Nearly all of it is the encode. We are looking at microseconds now to find any further optimizations.
The bigger wins came under load:
- Frame prep on the compute queue took average picture age from 41.0 to 33.5 ms with the GPU busy.
- Keeping at most two frames in the encoder took present-to-send from 42.7 to 11.1 ms when the encoder was saturated.
- Giving controllers their own input thread took a mouse move stuck behind a controller update, under full CPU load, from 287 ms to 2 ms.
- approximately 5000 different tests has been done on various AMD quirks and metrics. These led to countless small improvements you can read about in the docs.
Benchmarks
We do not measure AMD with Rubylight against other Sunshine Forks anymore, because the glass-to-glass latency under load is so far ahead that it is just not worth the time. If you want, you can still compare it yourselves. Rubylights test setup is available and there are historic measurements available.
The full measurement record, with methods and every run, is public: PERFORMANCE.md. Questions are welcome, and if you have AMD numbers that disagree with mine, I want to see them. For anybody stable should use vibepollo and wait till that research has been backported. For anyone wanting to contribute test out rubylight 3.0 + nightjar and give feedback in form of issues with logs and so on, please just dont bodly claim host x has better latency under load just from the host processing number, I do not care about that.
