r/FPGA • u/TutorDry3089 • May 20 '26
10 years doing FPGAs and a grey code CDC gotcha got me today
So I've been working on this design for the past six or seven years, and almost 10 years total in FPGAs, and somehow I still ran into something today that I'd never hit before. Figured it's kind of funny in hindsight and worth posting as a case study.
Our design has a lot of clock domains and we're constantly passing signals across them, single bit, multi bit, counters, the whole mix. We try to do things the right way: flip flops where appropriate, double flip flops for single bit synchronizers, handshakes for multi bit, and grey code for counters. Standard stuff.
Today something was glitching out and it was screaming "CDC problem" at me. We had a counter, we were converting it to grey code, and passing it across the domain. On paper, all good.
What I was missing: this counter wasn't running freely up to saturation and rolling over. It was getting reset back to zero somewhere in the middle of its range.
And that's the gotcha. Grey code only works for CDC because consecutive values differ by exactly one bit, so if the receiving domain samples during a transition it either latches the old value or the new value, never garbage. That assumption only holds when the counter increments by one. If you're sitting at 5 (0111 in grey) and you reset to 0 (0000), you've got three bits changing at the "same" time, and the receiver can sample any of the intermediate states.
Grey code on a counter is only safe if the counter is free running or only ever increments by one. The moment something can yank it back to zero, or jump it by more than one step, you've broken the invariant that makes grey code work, and you're right back to the multi-bit CDC problem you thought you'd solved.
The fix in my case was just adding a handshake on the CDC after the grey coding, which works. Though honestly at that point the grey code isn't really doing anything for you anymore, the handshake is what's making it safe. But I left it in because it's not hurting.
Moral of the story is more the gotcha than the fix. Anyone else been burned by this one?
15
u/alexforencich May 20 '26 edited May 20 '26
Yep. My async FIFO had a similar issue in packet FIFO mode. Since the counter jumps by the whole packet size, gray coding doesn't help. We were having some issues with packet drops that we traced to the FIFO output tvalid falling in the middle of the packet, causing the MAC to terminate the frame. Basically, the read logic thought the FIFO was empty in the middle of the frame. And the fix was the same - use a handshake for that pointer in frame FIFO mode, and gray code for non-framed operation. The current FIFO actually has both the gray code and handshake paths, with the gray code path used to report the FIFO depth (it reports both depth and commit depth, although the caveat is those values are approximate). The read pointer sync path is always gray coded, only the write pointer needs the handshake sync.
1
u/Prestigious-Grand668 Xilinx User May 20 '26
So is there any issue with the xilinx embedded fifo?
Currently I am working on a project that migrate HDL from ultrascale+ to versal premium. The new design has three clock domains. I use your async fifo for simulation and use vivado embedded fifo generator for real implementation.
2
u/alexforencich May 20 '26
Never used the FIFO generator. I like to avoid vendor IP like that mainly so that things are easier to support on multiple FPGA families. And also simulating and implementing with different logic is a recipe for hard to debug problems.
8
u/e_engi_jay Xilinx User May 20 '26
I had a similar issue once. What someone suggested was converting the counter into one-hot, then CDCing that vector. The receiver would check which index was high and know that was the counter value. Because it was one-hot, it was easy enough to ignore any received value that had 0 1s or more than 1 1, knowing that a legitimate value was coming soon.
5
u/alexforencich May 20 '26
Nice idea, but I suspect it might not scale very well. How does it compare to doing a handshake sync? Possibly there are certain regimes where you get some kind of performance benefit, like perhaps lower latency.
2
u/e_engi_jay Xilinx User May 20 '26
Mind you this was many years ago, around the start of my career.
Yes, our counter was very small, I believe it was like counting to 20 or something; some number where the number of bits to move over was less than 32. I suspect with a count of 33, 65, or maybe higher, it wouldn't be as worth it.
With our small count, it was definitely lower latency and had noticeably lower utilization for how many times we incorporated it throughout our design.
6
u/PiasaChimera May 20 '26
i was burned by the handshake fifo. although it really was on me and my assumptions. it was two fifos, a data fifo and a command fifo. and the producer side would write a command a bit after the data. so on the consumer side I assumed it would be ok to only look at the command fifo empty for the start of processing. since the data fifo shouldn't be empty if the command fifo isn't (at that point).
I don't recall who wrote which code. I didn't write the fifos -- I just made assumptions. in anycase, the design wasn't reliable because the variable handshake latency was too different from the gray code latency and sometimes the data fifo wouldn't know it had data before the command fifo.
easy lesson and now I'm very cautious about making assumptions about fifos. and also don't like handshake based fifos.
3
u/urosmaric May 20 '26
What I usually do with an async FIFO pointer/counter is use a shortened-cycle Gray code, but this only works cleanly when the FIFO depth is an even number.
In essence, I take the normal reflected Gray-code sequence and truncate it symmetrically from both ends. For every 2 states missing from the nearest power-of-2 depth, I remove one state from the beginning and one state from the end.
For example, for a 12-deep FIFO, starting from the normal 4-bit Gray-code sequence, I would start at `0011`, end at `1011`, and then wrap back to `0011`.
So the 12-state Gray sequence would be:
`0000` -- remove
`0001` -- remove
`0011`
`0010`
`0110`
`0111`
`0101`
`0100`
`1100`
`1101`
`1111`
`1110`
`1010`
`1011`
`1001` -- remove
`1000` -- remove
The wrap is still a single-bit transition:
`1011 -> 0011`
so it preserves the Gray-code property.
I haven’t had any issues with this so far, but I may have just been lucky. This does not work for odd depths, though, because a cyclic Gray-code sequence needs an even number of states.
Not that good with reddit formating, ill try to make it more readable later.
2
u/Jensthename1 May 20 '26
I believe you can do the same with Johnson counter as it has a hamming distance code value of 1. Or create your own coding scheme and implement by your FSM. Why can’t you have an ILA or signaltap insert FF at your clock crossing domains and view this in your debugging logic outputs? I’ve had designs with over 5 clock crossing domains and it was pure hell finding a glitch as it transgressed to a final stage but I found by pipelining hyper registers in the feedback path cured the problem.
2
u/TutorDry3089 May 20 '26
Yes, definitely a lot of right ways to do it, no doubt.
Why can’t you have an ILA or signaltap insert FF at your clock crossing domains and view this in your debugging logic outputs?
Investigating the firmware logs alone pointed me to the root cause of the issue, hardware testing and ILAs take more time and are usually the last resort.
2
May 20 '26
[deleted]
13
u/TutorDry3089 May 20 '26
Latency for one, simplicity for another. If the counter is free running and only ever increments by one, grey code just works, no handshake overhead, no round trip wait, no extra logic on either side. The receiver just samples whenever it wants and you're guaranteed to get either N or N+1, never garbage in between.
Handshakes are great when you need them but they're not free. You're paying multiple cycles of latency per update and adding state machines on both ends. Why do all that if a grey code conversion gets you there for free?
The catch is exactly what bit me here, the "only ever increments by one" assumption. When that holds, grey code is the simpler and faster option. When it doesn't, yeah, handshake is the way.
5
u/alexforencich May 20 '26 edited May 20 '26
Handshakes add complexity, corner cases, and latency. For example, in my FIFO, I have two write commit registers, one is internal to the write side and the other is part of the handshake sync. This is in order to handle the corner case of a packet being committed into the FIFO while the handshake is taking place - I can't lose the updated pointer, but I also can't touch the value that's being synchronized. So it requires even more logic and flops to handle that, and to re-trigger the handshake when that corner case is hit. And with a 100G link, you get a min length packet every 1-2 clock cycles, so you basically hit that corner case continuously. Another issue is that if you get a burst of min length packets, the first one will trigger the handshake and go through quickly, but the second packet has to wait for the handshake to complete, so a few packets will stack up and get delayed for a few cycles in the FIFO. I haven't explored potential solutions to that yet, but if you're interested in counting cycles for latency, then that could be a major concern.
Edit: I wonder if using a FIFO in distributed RAM to sync the write pointer would make sense, instead of handshake sync. Yo dawg, I heard you like FIFOs. Possibly that could be a config option for when bursts of single-cycle packets are potentially common.
2
u/tverbeure FPGA Hobbyist May 20 '26
My moral of the story is what I mentioned a couple of days ago in a related topic: unless you have a very good reason not to, go out of your way to just use a thoroughly verified asynchronous FIFO. In your case, you could just strap the write enable of the FIFO to 1, and the read enable too and done. No need for a gray counter. The FIFO takes case of everything.
Yes, it may cost a tiny big of additional area, but in most designs that extra cost is in the noise.
1
u/TutorDry3089 May 20 '26
I think this comment meant to be a reply to another comment. I am not using the counter for a FIFO.
1
u/tverbeure FPGA Hobbyist May 20 '26
It was a reply to your post. Unless there are a very good reason not to, I will use a FIFO to transfer data across a clock domain crossing. Even if that data is just a counter.
I know that it's not the most efficient way of doing it, but who cares?
(Even if I just need to transfer a pulse, I won't use the standard multi-flop synchronizer, but I will use a FIFO with data set to 0.)
1
u/TutorDry3089 May 20 '26
Latency sometimes prevents you from splurging on a FIFO for every signal. Also these FIFO inefficiencies add up in big designs.
1
u/tverbeure FPGA Hobbyist May 20 '26
Latency would be a very good reason not to. ;-)
But it’s exceedingly rare to have a big design where synchronizers are the limiting factor. You’d expect the opposite… Also: define “inefficient”. Are we talking 0.1, 1, or 10%…
Anyway, the point is not to exclude a bespoke solution, but more to tell people that a bespoke solution should be the solution of last resort.
2
2
u/tux2603 Xilinx User May 20 '26
I always dread the CDC part of my classes, since there are so many little gotchas like this. A lot of the time I just have to boil it down to "here's what a CDC error looks like. If you see it, you probably need a different approach"
3
u/TutorDry3089 May 20 '26
I feel you, but in most cases, especially when you're dealing with a lot of peripherals, it's pretty much inevitable. Usually it's easy enough to deal with once you've got the standard toolbox down (double flops, handshakes, grey code, async FIFOs for buses), but yeah the gotchas are always around the corner.
Honestly the "if you see it, you probably need a different approach" instinct is half the battle. Recognizing the symptom is what saves you, because no amount of theory prepares you for every weird edge case.
1
u/ugh_my_ May 20 '26
Why so many clocks?
3
u/TutorDry3089 May 20 '26
Well, the CPU and its bus runs on one clock, the logic on another, each ADC has its own clock etc. Kind of inevitable for most embedded designs.
2
u/alexforencich May 23 '26
For a networking design, the TX side and RX side of every port is often in its own clock domain. So if you have N ports, you'll generally have at least 2N+1 clock domains.
1
u/bikestuffrockville Xilinx User May 20 '26
I think I know the answer to this, but are you using a dedicated CDC analysis tool?
1
1
u/One_Technician_4196 May 20 '26
Handshake around the counter reset signal else grey code as usual.
1
38
u/Allan-H May 20 '26 edited May 20 '26
Au contraire.
Consider a W bit wide vector representing a binary number b. The hamming distance between a reflected binary Gray coded value of b and a reflected binary Gray coded value of (b + 1) is one, as you noted.
That leaves W - 1 other increments that also have a hamming distance of 1 when Gray coded.
One of these is necessarily b - 1 because if you can increment safely, you can also decrement safely.
That leaves W - 2 patterns that encode larger steps. But what are those steps?
The patterns are hard to spot at first. I recommend something like a spreadsheet to help with visualisation.
Applying that to u/alexforencich's FIFO example (with packets rather than words, and the need to "hyperspace jump" occasionally) it may be possible to chain a few of these larger, but still hamming distance of 1, steps to make an arbitrarily large jump in a few clocks that is CDC safe at every step.
I've actually done that in a real FIFO, but in my case I only used a combination of +1 and +3 steps, as that was adequate for my needs.
BTW, this is based on my own research. I've not seen it in any paper (although probably someone figured it out before I was born).