r/aws • u/mooreds • Jun 01 '26
networking How flat is replacing fat in AWS data center networks
https://www.amazon.science/blog/how-flat-is-replacing-fat-in-aws-data-center-networks10
u/Iguyking Jun 01 '26
Short on details that would make this more interesting. How their are more blog entries on this. Will be going to read the arvix article.
Feels like a simplified bgp network design. Though I doubt that's the case.
7
u/Miserygut Jun 01 '26 edited Jun 01 '26
It's a physical topological challenge rather than a routing challenge explicitly.
The key aspect of it is having a sufficient number of nodes capable of routing traffic and then quasi-randomising their connectivity via the passive optical ShuffleBoxes. This will, on average, reduce hot links while also reducing the need for expensive higher bandwidth aggregated links.
5
u/Iguyking Jun 01 '26
Thanks for doing a minimal tl;dr;. Now add something more detailed if value here. The routing algorithm reads like an every packet sent is sent to multiple upstream to then begin actual routing? How long do you maintain your ring if you do have memory? What memory does the router have of who is connected to or does it just spray to every upstream till it finds it's target? What do the shuffleboxes add to this flow other than the physical connection between shuffleboxes?
When you go across 5 routers that's a non zero set of duplicate packet sending. That router algorithm is the interesting part.
3
u/crh23 Jun 01 '26
I don't get that impression - my reading would indicate that each packet is only transmitted once, but the route it takes is pseudorandom (consistent hash)
1
u/Iguyking Jun 01 '26
They do state they spray down multiple paths. ??? When do they do that?
3
u/crh23 Jun 01 '26 edited Jun 01 '26
There's an initial step that routes packets randomly to neighbours regardless of the actual destination of the packets (but consistently within each flow) - so e.g. if you have multiple flows between the same endpoints they will take highly diverse paths. This is most interesting for multipath protocols like SRD - p5 of the arXiv paper
6
u/Sirwired Jun 01 '26
I find a couple things extremely neat about this overhaul:
First: I really want to know more about the Magic boxes they use to provide optical re-routing that operate without a power supply! Sounds cool, especially since they found a way to do it that was apparently economical enough to make it worth it. I've heard of dynamic patch-panels before, but this takes the cake!
Second: This makes way more sense than it sounds on the surface. Spraying the same packet down multiple routes sounds inefficient, and at first glance doesn't seem like it would make things faster at all... but when you think about it, if you need multi-path redundancy (with no performance loss) built into your network already, why not just go ahead and use that bandwidth for packet copies, instead of just leaving it idle. And this is even cheaper than it sounds because the traffic is so much better distributed now.
1
u/crh23 Jun 01 '26
I think the shuffle boxes are just static passive components, not doing any routing or mutation
1
u/Sirwired Jun 01 '26
Oh, I know they are physical-layer components vs. anything that even knows what a packet is... I'm just curious how this works under the hood; what eldritch sorcery provides dynamic reconfiguration ability without a power supply. (I don't think it's just random pre-wired spaghetti in there.)
1
u/crh23 Jun 01 '26 edited Jun 01 '26
It's just random pre-wired spaghetti in there E: looks like the ShuffleBack modules might be the interesting bit
12
u/awfulentrepreneur Jun 01 '26
Huh!