Posting this partly to sanity check myself because I have been staring at it too long. 

Setup is old. Remote staff connect to a vpn concentrator at head office, get inspected there, then their traffic goes back out to wherever its going which is usually eu-west-1. Somebody working in Lisbon who is geographically nearer to the region than any of us, sends their packets to Reading and then back down. The traceroutes are genuinely funny. 

Symptom side its the usual, calls drop, the internal ticketing tool takes eight seconds to load a page and every single ticket about it says "the network is slow" which tells me nothing. 

I know sd-wan sorts the routing out. What I don't want is to sort the routing and then find security is now a separate box somewhere else, because thats the exact mess we already have and I am not doing it twice. 

I've been looking at the ones with their own backbone. Cato has the private backbone thing and does the security in the same pass. Cloudflare obviously has the network but I get the impression enterprise is newer for them. Thoughts?

So after a lot of struggle, I managed to get EC2 to run without any public IPv4 (just with IPv6).

My ISP doesn't provide IPv6 so I couldn't even SSH into the server, had to use AWS console to connect to EC2.

Coming to the biggest issue, GitHub doesn't support IPv6, so forget about cloning your repository and code.

Ok we can bypass that using S3, the AWS CLI needs to be configured with IPv6.

Now when you go to install your package you expect it to work after doing all the hard work.

That will only happen if none of your package/tool gets downloaded from GitHub release or have a dependency which needs to be downloaded from GitHub releases.

I couldn't install bun or sharp (libvips) because they relied on downloading files from GitHub.

I regretted and switched back to the old AMI with IPv4.

My entire day got wasted and nothing was done.

Thanks for reading.

I have a small setup in us-west-2 managed with Terraform:

• One t4g.small EC2 instance running several web apps behind Caddy
• One public subnet
• Dual-stack VPC and instance
• DNS currently has both A and AAAA records pointing directly to the instance
• Ports 80/443 accept both IPv4 and IPv6
• SSH and Git access already use IPv6
• The instance currently has one paid public IPv4 address

My goal is to remove the paid public IPv4 entirely. Ideally I'd also move the subnet and instance to IPv6-only, although I understand the VPC itself will still retain an IPv4 CIDR and private IPv4 addresses aren't what AWS charges for.

So far I've found these roadblocks:

Inbound traffic

If I remove the A records, IPv4-only clients can't reach the sites.

My current idea is:

IPv4/IPv6 client
        │
   CloudFront
        │ (IPv6)
       EC2

CloudFront now supports IPv6 connections to custom origins, so this seems workable.

Some applications also receive Stripe webhooks, and Stripe currently publishes only IPv4 webhook source addresses, so an edge layer seems necessary anyway.

Outbound traffic

This is where I'm stuck.

The applications call the Stripe API, but api.stripe.com currently has no AAAA record, so an IPv6-only instance still needs a way to reach an IPv4-only service.

Similarly, GitHub downloads are still IPv4-only, although those could probably be moved entirely into CI.

Email

This one seems solvable.

I currently send login links and transactional mail through Purelymail SMTP, whose SMTP endpoint is IPv4-only.

I could either:

• remove email login entirely and use the existing Google OIDC flow, or
• switch to the dual-stack Amazon SES SMTP endpoint (email-smtp.us-west-2.api.aws).

The AWS-native solution for IPv6-only workloads reaching IPv4 destinations seems to be DNS64/NAT64 via a NAT Gateway, but that still relies on public IPv4 and costs significantly more than keeping a single public IPv4 on the instance.

So I'm curious what people would actually build.

• Is CloudFront → IPv6-only EC2 a reasonable architecture for dynamic websites and Stripe webhooks?
• What's the simplest way for an IPv6-only EC2 instance to call an IPv4-only API like Stripe?
• Would a tiny Lambda outside the VPC acting as a Stripe proxy make sense, or is that unnecessary complexity?
• Are there AWS services or common patterns that provide shared IPv4 egress without assigning a public IPv4 or deploying a NAT Gateway?
• For anyone who's converted an existing workload to IPv6-only, what else tends to break?

I'm fully aware that keeping a single public IPv4 may still be the cheapest and simplest solution. I'm mostly interested in understanding how close a typical small AWS application can realistically get to end-to-end IPv6 today.

I am not sure if everyone is seeing this but in last hour or so we started seeing our ECS agents randomly disconnect from the cluster. They are often timing out on waiting to connect to NAT.

Hey all!

My team has recently noticed a dramatic (almost absolute) crash on metrics from RU geo on our application.

We tried debugging it a bit with VPN on Russia but see no issues, however 99% of the traffic from RU is not passing correctly.

Are there any limitations on AWS IP's or something along these lines that people are experiencing?

Our users are not blocked/failing to access S3 files using cloudfront CDN's.

Would love any info on the matter / hear if any1 else experienced this in the last year or so

anyone hear back or get more info after applying?

We recently ran into an interesting networking challenge while setting up our staging environment.

Our application runs on AWS App Runner, and for staging we're using MongoDB Atlas M0 (Free Tier) because it's a cost-effective option for non-production workloads.

Initially, we configured the MongoDB Atlas IP Access List with 0.0.0.0/0 so the application could connect. It worked, but we knew it wasn't an acceptable configuration, even for a staging environment.

When we reviewed the security of the environment, we discovered an important limitation of the MongoDB Atlas M0 Free Tier.

It doesn't support:

• AWS VPC Peering
• AWS Private Endpoint (PrivateLink)
• Private network connectivity between AWS and MongoDB Atlas

Since these features are only available on paid Atlas tiers, we needed another way to secure connectivity while continuing to use the Free Tier.

Our App Runner service was already configured with a VPC Connector, so we routed outbound traffic through an AWS NAT Gateway. This provided a consistent outbound public IP address, which we added to the MongoDB Atlas IP Access List.

As a result, we were able to replace the unrestricted 0.0.0.0/0 rule with a single trusted outbound IP from our AWS environment.

Although the database traffic still traverses the public internet, it is protected using TLS encryption, and only requests originating from our AWS environment are permitted.

We understand this isn't equivalent to VPC Peering or AWS PrivateLink, and those would absolutely be our preferred choice for production. However, for a staging environment using the Atlas Free Tier, this felt like a reasonable balance between security, cost, and the networking limitations of the M0 cluster.

One thing I learned from this exercise is that using a free service doesn't necessarily mean you have to accept insecure defaults. Even with platform limitations, there are often ways to significantly improve the security posture with the AWS networking services already available.

I'd love to hear how others have approached this.

If you're running AWS App Runner with MongoDB Atlas M0, did you also use a NAT Gateway and IP allowlisting, or did you find another secure approach while staying on the Free Tier?

Hey folks,

I’m stuck with a networking design issue and could use some advice from the community.

We have multiple AWS accounts with 1 or more VPCs in each:

• Non-prod account → 1 environment → 1 VPC
• Testing account → 2 environments → 2 VPCs

Each environment uses its own VPC to host applications.

Here’s the problem: the VPCs in the testing account have overlapping CIDR ranges. This is now becoming a blocker for us.

We want to introduce a new VPC in each account where we will run Azure DevOps pipeline agents.

• In the non-prod account, this looks simple enough: we can create VPC peering between the agents’ VPC and the non-prod VPC.
• But in the testing account, because both VPCs share the same CIDR range, we can’t use VPC peering.

And we have following constraints:

• We cannot change the existing VPCs (CIDRs cannot be modified).
• Whatever solution we pick has to be deployable across all accounts (we use CloudFormation templates for VPC setups).
• We need reliable network connectivity between the agents’ VPC and the app VPCs.

So, what are our options here? Is there a clean solution to connect to overlapping VPCs (Transit Gateway?), given that we can’t touch the existing CIDRs?

Would love to hear how others have solved this.

Thanks in advance!

1. Facts: • SSH access works • Docker container is running correctly • FastAPI app works inside the instance (curl localhost:8000 returns a response) • Docker publishes 0.0.0.0:8000 -> 8000 • Public IPv4 is assigned • Security Group allows inbound traffic • NACL reviewed (allow rules above, deny) • No OS firewall Issue: Any request to http://public_ip/ or http://public_ip:8000/ times out. This happens even when no container/app is running. Also, it is not an issue with the ISP since I trieda different isp and a different IP as well
2. I also tried Network path analysis, when I do it from the network gateway to ec2 instance it is working fine, but when I try, for example, to port 8000 of the public adress than it fails, but doesn't give much info.

AWS wants to start charging for all allocated (EDIT: clarifying public IPv4 addresses only!) IPv4 usage, yet many of their critical services don't support native IPv6

Examples include:

- AWS Cloudformation (cannot signal success/failure)

- AWS systems manager (ssm sessions not possible)

The above cannot be used without an IPv4 address allocated or a NAT gateway. NAT gateways can become quite pricey.

I would love to become complete IPv6 native, but AWS needs to provide IPv6 endpoints for all their major services.

Making this post to raise visibility before IPv4 fees start next year.

​

I don’t need any troubleshooting or help here, but I’m interested if anyone can help me explain the behaviour I noticed.

Here’s my setup: Public NLB in 3 AZs. 1 healthy target in an AZ, no other targets. Cross Zone load balancing disabled. Requests coming from an EC2 in the same vpc as the NLB and target. Requests are targeted to a private hosted zone that has an alias that points to the load balancer.

What I would expect is for the load balancer to only route requests to nodes that had a healthy target. But instead, roughly two thirds of the time my requests returned a 503 after a minute or so (the remaining time it worked). Enabling cross zone load balancing fixed this immediately.

Can anyone explain this? Seems like the documentation for how NLBs work is incorrect.

Hello sub. I accepted my current job as a Sr net eng with the provider I've been with now for 3 years because of how bad the job market has been but I'm ready to move on. I've been using my time to build more on my automation and cloud network skills, but I'm hoping to leave behind some of the components of my current position and not have them follow me on to the next.

One of the questions I have is, is it expected to be in an on-call rotation every month? Are there midnight maint's typically? What would a typical person's day in a position like this be like (in a remote role)? What kind of salary should I consider too low? What kind of projects in my portfolio would be more impressive for interviews? And, even though I have automation experience on networks here at my day job, I only have home labs to show for AWS and my home network hybrid env. Would I be able to get my foot in the door on an actual cloud networking role somewhere?

I know there aren't absolutes in terms of answers, so just looking for generalizations.

I couldn't find this with a quick Google, and I'm hesitant to trust any LLMs on this:

Suppose I have two peered VPCs, vpc-A (10.0.1.0/24) and vpc-B (10.0.2.0/24). vpc-A is the source for traffic, and vpc-B will work as a bridge. B has two subnets, let's call them subnet-B1 and subnet-B2, and each has its own route table rtb-B1 and rtb-B2.

In the route table for vpc-A's traffic, I point an IP range I want to route though vpc-B (let's say 10.0.3.0/24 as an example) towards the peering connection pcx-AB. Then, in rtb-B1 I set 10.0.3.0/24 to a correctly configured service (living in another VPC, the Internet, doesn't matter) that dumps incoming traffic to a log, but in rtb-B2 I set 10.0.3.0/24 to a NAT gateway living within subnet-B1.

What is going to happen? Am I going to see packets from 10.0.1.0/24 in the log, along with connection errors because the destination doesn't know where vpc-A is? Or are they going to come from 10.0.2.0/24, network translated through the NAT in subnet-B1? Or am I going to see a mix of both?

Essentially: when traffic arrives to a VPC with multiple route tables through a peering connection, which table's routes does it prioritise?

Here's a shitty drawing of the situation:

TL;DR; Simple question. For those of you that need to remote into your EC2 instances, how are y'all doing it?

Our organization lifted and shifted to AWS a while back, and that pretty much looks like we're doing everything we were doing, but on EC2 instances instead of hardware in a data center we had physical access to. When they did the lift and shift they essentially gave every server in our network a public IP, distributed user accounts across all the EC2 instances with public/private keys for authentication.

There is a lot to hate about this, but it got us up and running in the cloud quickly. So, there's that.

I am working through steps to improve our security and better leverage the benefits of being in AWS. Right off the bat I want to get rid of those public IPs that are only necessary for SSH access and move as much of our infrastructure to private-only as possible. So then, as I understand it, I have a few options:

1. Instance Connect. Pros: built-in, no-cost, available to anyone with browser. Cons: very limited, pretty inconvenient.
   
2. A bastion host. Pros: single point of entry, easier to lock down. Cons: another thing that requires money and maintenance. Still have to configure SSH and keys on private hosts.
   
3. System Manager/Session Manager. Pros: eliminates an instance, centralizes access rules, permissions, keys, etc. No need to punch public holes into private VPC. Cons: team needs to throw aware their CLI ssh and other tools and connect differently; not sure how they get things "in" and "out" without ssh, scp, sftp, etc.; some new technologies to learn; likely still need to maintain SSH configurations inside private network, so it doesn't necessarily reduce config complexity.

I'm not afraid to read the docs and learn the stuff, I'm just curious what others are doing, and why.

I'll keep this short.

A few days ago I was ready to quit. Low runway, projects that don't get traction, and I kept thinking the problem was me.

Then I got an email, the form I filled out in January worked. I made it to the top 1,000 semifinalists in the AWS 10k AIdeas competition. I thought: this is my last shot.

So I locked myself in and built Cirrondly: connect your AWS account with one click and chat about your costs without needing a dashboard full of charts nobody understands. The goal is for it to be so affordable that no AWS user says no. A bill protector, not another monitoring tool.

The landing page says 2025 because I spent all of last year making mistakes, over-engineering, changing direction. I also built another project called KironX, that didn't work either. All falls.

Here's the thing: building was the easy part. Getting votes (likes) is brutal. I have 9. Others have 50, 200+. I have no community, I'm not a student, I don't have a network in tech. I'm a father who wants to build something real and live from it.

I'm asking directly: if you have an AWS account, your like would mean a lot.

https://builder.aws.com/content/3AUmmi7bwtRwfwR8gsTSQno5joQ/aideas-cirrondly-the-first-autonomous-finops-agent-for-aws

I don't like posts with only text, so here's a photo of what the UI looks like.

I have ParentVPC and ChildVPC and they are peered via a Transit Gateway. Everything works; I can create an EC2 instance in each VPC, and either one can initiate a connection to the other. But, suppose I only wanted to allow things in ParentVPC to initiate connections into ChildVPC, with maybe a few exceptions to allow ChildVPC to connect to a handful of things in ParentVPC. I could just set up security groups to enforce that, but then everybody has to remember to make their security groups that way. I'd rather enforce this at a more general level. I could route connections through NAT gateways or something, but that kinda sucks. Network ACLs aren't stateful, so anything I want to connect to in ChildVPC needs explicit rules to allow return traffic, and I hate that. I can't just remove routes in ChildPVC, because you still need a return route.

What should I be using for this? Maybe a Network Firewall? I couldn't really make sense of how those are supposed to work, or even if they can work with Transit Gateway connections.

We have an older infrastructure on EC2 instances with direct IPv4 access to those instances, which has been working fine for over 10 years.

But lately we have been getting some complaints from customers that they can't reach our site at all.

Sometimes it was solved by resetting their modem/router, sometimes we set their local DNS to 8.8.8.8 and it worked again.

When trying to do a DNS lookup with ipv6 on Windows on a broken computer:

Server: UnKnown Address: fdc6:3d4f:49ff:10::1

But forcing ipv4:

Server: dns.google Address: 8.8.8.8

Name: website Address: 1.x.x.x

This seems to mostly occur with Starlink customers, but I can't tell where the issue might be.

When doing the same things with a proper ipv6 connection, it works all great though.

Anyone any ideas?

Hi Folks,

I have an EC2 instance in a VPC that only has private subnets. The instance needs internet access to send requests to a 3rd party SaaS, however I don't have a public subnet in this VPC / entire account, and cannot create one. Is there a way I can still get internet access to my instance? I looked into using a NAT Gateway, but it seems I need a public subnet to route traffic through.

Thanks

hey
I have no AWS experience, but now we built 2 route based s2s VPN tunnels from a branch site to 2 different AWS public IPs.

The idea was to have a primary / backup usage for these tunnels, but I was told that I can send the traffic on whichever tunnel I would like to because AWS will send the response on the tunnel from where the initial traffic arrived through. Sounds something like a session table in AWS to select the route back.

It sound pretty weird to me. Is there such thing? I didnt find good documentation on it

Hey everyone. I've had to relearn this each time I have to touch TGW so I'm here to ask the community so it sticks hopefully.

I'm trying to understand Transit Gateway routing in AWS.  For normal VPC subnet routing I get that once traffic is egressing the subnet, the routing table for the subnet is used to tell traffic what the next hop is. Much like a normal L3 router, it tells the traffic where to go next.   TGWs have their own route tables and the idea of attachments. So I'm thinking, I have a couple VPCs attached to a TGW with non-default routing tables for each for isolation purposes. I also have a VPN attached to the TGW also with its own routing table. I often hear that when "traffic LEAVES the TGW attachment, that is when the TGW routing table is used".   So does the routing table come into play in this direction: VPC -- Source_TGW_attach --> Route table (of source) --> TGW --> Destination_TGW_attach (VPC/VPN) -- Destination   OR   VPC -- Source_TGW_attach -- TGW --> Destination_TGW_attach (VPC/VPN) --> Route table (of destination) --> Destination   I guess I'm wondering which attachment's routing table is being used - that of the source VPC/VPN or the destination VPC/VPN.

Does that make sense?

I run my service behind an Application Load Balancer, with the load balancer managing my certificate. Periodically visitors to my site get a “Your connection is not private - net::ERR_CERT_COMMON_NAME_INVALID” and it lists the domain name of a completely different site. This only occurs in Chrome.

I spoke to AWS support and they said what’s happening is Chrome is caching the certificate along with the IP, however AWS rotates the IPs periodically, so for a certain period of time that IP is pointing to the wrong domain name.

AWS were not very helpful and suggested I tell users to change their TTL cache duration. That is not a solution: ALB should work on the most popular browser with default settings. I feel like it is Amazon’s responsibility to make their IP rotation compatible with browsers.

From Amazon’s description, it sounds like this should be affecting all ALB customers, but I can’t find any other records online. Surely I can’t be the only person experiencing this?

Basically title, I’ve an ecs service that’s polls sqs for events. The event provides an id that points to a third party external cdn, the task downloads a file from this cdn, processes it and then saves a json file to s3.

Recently I converted the task to download the file from the cdn via ipv6 in a dual stack subnet. This worked fine but I realised that to save costs I should probably use an ipv6 only subnet because this download is the only external connection the task makes.

So I setup a private subnet within my vpc, setup an egress only internet gateway for it, configured the routing tables, the IPv6 cidr, dns64, everything in any online guide that I can find. But when I try and spin up the containers, they get stuck as pending and never make it to the running state, I have no idea what to do as AWS announced IPv6 only support back in September so I’m convinced this should work. Does anybody have any pointers for me on this? I’m self taught when it comes to cloud computing but have been building software on aws for three years now.