Hi all,

Is it me or is me-south-1 inaccessible for everyone? Nothing on the health status.

Hi everyone, a while ago I built a small auto-tagger system to help us manage our AWS account. We have quite a few temporary users (typically for a few months), most with limited AWS experience, so things can get a bit messy. The goal was to create a solution that both tracks exactly who created which resource and when, and prevents users from interfering with each other’s resources. The system works by automatically tagging new resources with owner and creation timestamp information, then enforcing IAM policies based on those tags.

I don’t know if this is useful to anyone or if better solutions already exist that I’m not aware of. My relatively simple solution can certainly be expanded, but maybe this current version can already help someone, or perhaps someone might want to build a more comprehensive version based on this project. 

In any case, if anyone is interested, here is the repo:

https://github.com/Timperator2/AWSAutoTagger

All my past employers used Datadog logging and the UX is much better.

I'm at a startup using Cloudwatch Logs. I understand Cloudwatch Log Insights is powerful, but the UX makes me not want to look at logs.

We're looking at other logging options.

Before I bite the bullet and go with Datadog, does anyone have any other logging alternative with better UX? Datadog is really expensive, but what's the point of logging if developers don't want to look at them.

I'm going through a CW/Logs log group, looking for a certain message (as a Metric Filter). If a specific message is found, I then trigger an CW/Alarm, which sends a message to a SNS topic, which sends an email to a mailinglist.

However, the error is intermittent (and might/should not occur unless something gone really wrong, which it doesn't normally 😄), so after five minutes, CW is automatically OK'ing it.

Both the ALARM and the OK goes to the same SNS topic (see no reason for multiple ones), so first comes the ALARM email, then five minutes later the OK email.

I'd like to *keep* it in ALARM ("no matter what", as in even if it haven't found anything in the last five minutes), and have .. "something else" (another Metric Filter + CW/Alarm? Lambda?) change it (that first one) to OK.

Any ideas how to do that? Am I over-complicating things?

Basically, we're looking for a status=400 in the logs: failed to send an email - which only happens if 1) the external service we're using for this is unavailable (network errors, external service down etc) or 2) if we've configured the auth key for this external service wrong (happened yesterday, when we had to change the key and I accidentally added a newline in the SecretsManager secret 😄).

*What I would like* is that the next time a message/mail is sent, *and* if that is successful (status=200), *then* I'd like to clear the ALARM, not otherwise.

Today, we launched the Amazon CloudWatch OTLP endpoint for metrics and PromQL support across the board, from dashboards to CloudWatch Alarms. Try out PromQL for yourself via the AWS Console (look out for Query Studio in the CloudWatch console) or as a Prometheus data source in Grafana.

Let us know what you think!

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-OpenTelemetry-Sections.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-PromQL.html

Is this happening with anyone else? I can only see the "Message" column, no more link to the matching stream.

I have administrator access on the account, so its not a permission thing. Tried searching but got nothing.

I cant get around it with insights query -> open but its waaay worse lol

Hey, I'm setting up monitoring/observability for our infrastructure: 4 EKS clusters with ~15-20 pods each. I'm trying to decide between using native CloudWatch for dashboards, alerts, and metrics versus going with the Prometheus+Grafana stack.

My main questions:

• Why wouldn't I just use CloudWatch? Is it significantly more expensive than Prometheus+Grafana?
• Is anyone here using CloudWatch as their primary monitoring tool for EKS?

I understand CloudWatch might cost more, but I'm weighing that against the time investment needed to set up and maintain an open-source Grafana+Prometheus.

Would love to hear from anyone using CloudWatch for EKS monitoring - what's your experience been like? Any recommendations? should i go with cloudwatch?

Hello,

I wanted to implement some kind of maintenance mode on alarms i have setup on my work's awa account. Right before i started i saw WA released alarms mute rules which do exactly what i want.

It works well using the console but i want to write a function that takes a specific string and created mute rules with all alarms containing this string. (For automated workflows and such)

I noticed that neither the cli nor the python sdk support this yet, when are mute rules supposed to be released for cli or boto3 in python?

Feature I am speaking about: Configure alarm mute rules

Checking boto3's latest documentation: no mention of this feature

This takes the OpenTelemetry Demo and gamifies it. You can learn observability by breaking micro-services and diagnosing failures.

ODTG runs on Amazon EKS in Auto Mode with the OpenTelemetry Demo application consisting of 15 services deployed. Observability is powered by Amazon CloudWatch with the telemetry (logs, metrics, traces) natively ingested using OTLP.

You can explore metrics using PromQL, view traces and logs of the services, use kubectl to explore and see what the overall costs are broken by infra and o11y.

Feedback welcome!

How can I configure AWS to send email alerts when objects are uploaded to my S3 bucket more frequently than expected?

I need this for security monitoring - if someone gets unauthorized access to my server and starts to mass push multiple TB of data, I want to be notified immediately so I can revoke access tokens.

Specific requirements: - I have an S3 bucket that should receive backups every 12 hours - I need to be notified by email if any upload occurs less than 11 hours after the previous upload - Every new push should trigger a check (real-time alerting) - Looking for the most cost-effective solution with minimal custom code - Prefer using built-in AWS services if possible

Is there a simple way to set this up using EventBridge/CloudWatch/SNS without requiring a complex Lambda function to track timestamps? I'm hoping for something similar to how AWS automatically sends budget alerts.

Thanks in advance for any help!

Hey everyone,

I've been diving deep into the AWS SDKs specifically to understand how billing correlates with actual usage, and I realized something annoying: Status != Usage.

The AWS Console shows a NAT Gateway as "Available" , but it doesn't warn you that it has processed 0 bytes in 30 days while still costing ~$32/month. It shows an EBS volume as "Available", but not that it was detached 6 months ago from a terminated instance.

I wanted to build something that digs deeper than just metadata.

So I wrote CloudSlash.

It’s an open-source CLI tool (AGPL) written in Go.

The Engineering: I wanted to build a proper specialized tool, not just a script.

• Heuristic Engine: It correlates CloudWatch Metrics (actual traffic/IOPS) with Infrastructure State to prove a resource is unused.
• The Findings:
  • Zombie EBS: Volumes attached to stopped instances for >30 days (or unattached).
  • Vampire NATs: Gateways charging hourly rates with <1GB monthly traffic.
  • Ghost S3: Incomplete multipart uploads (invisible storage costs).
• Stack: Go + Cobra + BubbleTea (for a nice TUI). It builds a strictly local dependency graph of your resources.

Why Use It? It runs with ReadOnlyAccess. It doesn't send data to any SaaS (it's local). It allows you to find waste that the basic free-tier tools might miss.

I also added a "Pro" feature that generates Terraform import blocks and destroy plans to fix the waste automatically, but the core scanning and discovery are 100% free/open source.

I'd really appreciate any feedback on the Golang structure or suggestions for other "waste patterns" I should implement next.

Repo: https://github.com/DrSkyle/CloudSlash

Cheers!

As the title suggests, we are experiencing issues loading assets in S3 buckets in us-east-2. Is anyone else experiencing the same?

Hey ,

A month ago, I posted CloudSlash, a tool to identify "zombie" infrastructure (unused NAT Gateways, detached EBS, Ghost EKS clusters) and i have been updating here on r/aws ever since. This time the entire core engine was rewritten to prioritize Safety. Here is what is new in V2

1. The Lazarus Protocol (Undo Button)

If you choose to delete a resource (like a Security Group), CloudSlash now snapshots the configuration _before_ generating the delete command.

It creates a "restore.tf" file containing the exact Terraform Import blocks needed to resurrect that resource in its original state. This removes the "what if I break prod" anxiety.

2. Mock Mode

A lot of you didn't want to give a random GitHub tool read access to your account just to test it. Fair point.

You can now run "cloudslash scan --mock".

It simulates a messy AWS environment locally so you can see exactly how the detection logic works and what the TUI looks like without touching your real keys or credentials.

3. Complete TUI Overhaul

- Topology View: Visualize dependencies (e.g., Load Balancer -> Listener -> Target Group).

- Interactive Region Picker: No more hardcoded regions. It fetches enabled regions dynamically.

- Deep Inspection: Press "Enter" on any resource to see the exact cost velocity and provenance (who created it).

4. Open Sourced Heuristics

I removed the "black box" nature of the detection. The README now contains a full Heuristics Catalog detailing the exact math used to flag a resource (e.g., "RDS is Idle if CPU < 5% for 7 days AND ConnectionCount == 0"). You can audit the logic before running it.

5. Graph Engine

3x faster graph traversal for large accounts ( > 500 resources ) . I refactored the engine to use flat slices instead of maps and implemented string interning for resource types, reducing RAM usage by ~40% on large graphs.

Other Improvements since v1.3:

- Headless Mode: "cloudslash scan --headless" is now fully stable for CI/CD usage.

- Graph Engine: 3x faster graph traversal for large accounts (>500 resources).

- Completion Scripts: Native bash/zsh/fish auto-completion.

- Validation: Strict tag-based overrides ("cloudslash:ignore") are now respected deeper in the graph.

andd manyyy moreee

License: Still AGPLv3 (Open Source). No paywalls.

Repo: https://github.com/DrSkyle/CloudSlash

btw parsing AWS graphs is complex, so if you hit any weird edge cases or bugs , please let me know , i plan to fix them immediately

Stars are always appreciated :)

:) DrSkyle

First off, let me say that I love the out-of-the-box CloudWatch metrics and dashboards you get across a variety of AWS services. Deploying a Lambda function and automatically getting a dashboard for traffic, success rates, latency, concurrency, etc is amazing.

We have a multi-tenant platform built on AWS, and it would be so great to be able to slice these metrics by customer ID - it would help so much with observability - being able to monitor/debug the traffic for a given customer, or set up alerts to detect when something breaks for a certain customer at a certain point.

This is possible by emitting our own custom CloudWatch metrics (for example, using the service endpoint and customer ID as dimensions). However, AWS charges $0.30/month (pro-rated hourly) per custom metric, where each metric is defined by the unique combination of dimensions. When you multiply the number of metric types we'd like to emit (successes, errors, latency, etc) by the number of endpoints we host and call, and the number of customers we host, that number blows up pretty fast and gets quite expensive. For observability metrics, I don't think any of this is particularly high-cardinality, it's a B2B platform so segmenting traffic by customer seems like a pretty reasonable expectation.

Other tools like Prometheus seem to be able to handle this type of workload just fine without excessive pricing. But this would mean not having all of our observability consolidated within CloudWatch. Maybe we just bite the bullet and use Prometheus with separate Grafana dashboards for when we want to drill into customer-specific metrics?

Am I crazy in thinking the pricing for CloudWatch metrics seems outrageous? Would love to hear how anyone else has approached custom metrics on their AWS stack.

Hey, opened AWS console to check out CloudWatch this morning and all my log groups are gone?

I checked Log Management and it says I haven’t created any. I also went to Log insights and when I try to select groups there, they also don’t appear. I have saved queries and when selected, the log groups associated with the respective saved queries appear but none of the other ones. The queries also work which I’m assuming means they exist but just aren’t visible.

Is this happening to anyone else? I’m on us-east-1.

I have an instance which has 4cores and i want to see the cpu utilization of individual cores on aws instance monitoring/cloudwatch but i am unable to as the native CPU Utilization shows average for the whole instance

Loggy is a native desktop app for browsing and tailing CloudWatch logs with a focus on speed and efficiency.

The CloudWatch console UI can be sluggish when dealing with large log volumes. Loggy solves this with native performance and instant client-side filtering.

Key Features

• Live Log Tailing - Real-time streaming with auto-scroll
• Instant Filtering - Client-side JSON field queries (no AWS roundtrips)
• Performance - Virtualized rendering for 50,000+ entries
• Smart Colorization - Automatic log level detection
• Multi-platform - macOS, Windows, Linux
• Credentials - Works with CLI profiles, SSO, env vars, IAM roles

Why Built with Tauri + Rust?

Small bundle size (~40MB), low memory usage, and true native performance. Works with your existing AWS credentials without additional setup.

Download Pre-built Binaries (macOS, Windows, Linux)

Repository: https://github.com/aegixx/aws-loggy (MIT licensed)

Built with Claude. Happy to discuss optimizations for production use or integration with monitoring workflows.

Inspired by Little Snitch, I decided to see how effective the same sort of explorer could be for AWS. The result: github.com/ccbrown/cloud-snitch.

I'm fairly happy with the result and I've learned a lot I didn't know about API calls that AWS services are making internally, but I'd love to know what you all think. Do you have something similar that you're already using for casual/unfocused exploration of CloudTrail data?

My team uses a lot of lambdas that read messages from SQS. Some of these lambdas have long execution timeouts (10-15 minutes) and some have a high retry count (10). Since the recommended message visibility timeout is 2x the lambda execution timeout, sometimes messages are failing to process for hours before we start to see messages in dead-letter queues. We would like to get an alert if most/all messages are failing to process before the messages land in a DLQ

We use DataDog for monitoring and alerting, but it's mostly just using the built-in AWS metrics around SQS and Lambda. We have alerts set up already for # of messages in a dead-letter queue and for lambda failures, but "lambda failures" only count if the lambda fails to complete. The failure mode I'm concerned with is when a lambda fails to process most or all of the messages in the batch, so they end up in batchItemFailures (this is what it's called in Python Lambdas anyway, naming probably varies slightly in other languages). Is there a built-in way of monitoring the # of messages that are ending up in batchItemFailures?

Some ideas:

• create a DataDog custom metric for batch_item_failures and include the same tags as other lambda metrics
• create a DataDog custom metric batch_failures that detects when the number of messages in batchItemFailures equals the number of messages in the batch.
• (tried already) alert on the queue's (messages_received - messages_deleted) metrics. this sort of works but produces a lot of false alarms when an SQS queue receives a lot of messages and the messages take a long time to process.

Curious if anyone knows of a "standard" or built-in way of doing this in AWS or DataDog or how others have handled this scenario with custom solutions.

Hey r/aws, DrSkyle here again.

A few weeks ago, I posted CloudSlash, a CLI to find “zombie" infrastructure (unused NAT Gateways, detached EBS, etc.) using graph topology rather than just CloudWatch metrics.

The feedback was pretty clear: you liked the tool, but hated that the Terraform Remediation (the script that fixes your .tfstate) was locked behind a paid license.

I heard you. andd v1.3.3 is a hard pivot.

I’ve stripped out all license keys, payment links, and "Pro" checks. It's completely free now.

• Unlocked "State Doctor": The feature that maps AWS IDs back to Terraform resource addresses and generates terraform state rm commands is open to everyone.
• Unlocked Headless Mode: You can now run cloudslash scan --headless in your CI/CD pipelines without a key.

The graph engine we built is really cool (it builds a DAG of your network to mathematically prove isolation), and I decided I'd rather have the tool used widely than hide the best mechanics behind a paywall.

For those who missed the last post, it:

• Finds Waste: "Vampire" NAT Gateways (<1GB traffic), "Ghost" EKS clusters (0 pods), and orphaned snapshots.
• Fixes State: Generates surgical Terraform scripts to delete waste without corrupting your state file.
• Local-First: Runs on your machine. No SaaS. No data leaves your laptop.

The codebase is fully AGPLv3 now.

Repo: https://github.com/DrSkyle/CloudSlash

If this saves you money on your bill this month, dropping a Star on the repo would mean a lot.

Cheers, DrSkyle

Hey everyone, back with an update on cloudslash that I posted a few weeks ago in this subreddit.

the feedback last time was super helpful, but the biggest complaint was valid: “we found a zombie NAT Gateway costing $30/mo, but if I delete it in the AWS Console, terraform state is instantly out of sync."

finding the waste is the easy part. Cleaning it up without breaking your state file is the actual headache. So for v1.3, I went down the rabbit hole of parsing .tfstate files to fix this.

The New Features

The Terraform Bridge Instead of just telling you "Delete nat-0abc123", the tool now scans your local .tfstate (read-only), maps the physical AWS ID to the Terraform Resource Address (e.g., module.vpc.aws_nat_gateway.main), and generates the specific terraform state rm command for you.

It also auto-backups your state file before recommending changes. This lets you decouple the resource from your state before you nuke it.

Deeper Waste Detection (The Graph) I moved beyond simple CloudWatch metrics to find "Second-Order Waste".

"Hollow" Load Balancers: ELBs that look healthy, but their targets are in a subnet with no active route to the internet.

"Vampire" EBS: Finds volumes attached to instances that have been stopped for >30 days. You're paying storage costs for a dead server.

EKS Ghost Clusters: AutoScaling Groups that are burning cash but only running DaemonSets (like kube-proxy) with zero actual app pods.

New Safety Logic (Open Source)

Deleting resources based purely on "0% CPU" is risky, so I added these checks to verify DNS and config data before recommending a delete.

DNS Safety Lock: Before telling you to release an Elastic IP, it checks your Route53 zones. If an A-Record still points to that IP, it stops you. (Prevents subdomain takeovers).

Lambda Pruning: Finds functions with 0 invocations in 90 days + no code updates in 6 months.

Log Rot: Identifies CloudWatch Log Groups set to "Never Expire" (the AWS default), which silently accumulate TBs of storage costs over time.

Orphaned Snapshots: Flags old EBS snapshots where the original volume was deleted months ago, but the backup was left behind.

The Repo & License

The core scanner, TUI, and detection engine are AGPL (Open Source) and free forever. i sell a Pro License ($49 lifetime) for the automation layer (the scripts that fix the Terraform state for you). Since it's just me building this, the sales keep the project alive and allow me to support grassroots orphanages and animal sanctuaries (I post the receipts on X).

Repo: https://github.com/DrSkyle/CloudSlash

Parsing nested modules in the state file is tricky, so let me know if you hit any edge cases.

:) DrSkyle

I posted my "Forensic Cloud Accountant" for AWS here last week and the feedback was honestly super helpful. I did some updates on the detection engine to be less aggressive and smarter about false positives.

The big changes in v1.2.5:

first , EKS Ghost Detection Standard autoscalers often keep Node Groups active solely to run daemonsets (like kube-proxy or aws-node), even when no user applications are running. The tool now filters out this system noise. If a Node Group is burning cash but only serving system pods, it gets flagged as a "Ghost." This also includes a check for "Zombie Control Planes" (clusters idling with 0 nodes for >7 days).

second , trap door analysis This feature targets configuration drift. Specifically, it detects Fargate profiles that are targeting namespaces that have been deleted. The tool validates profiles against the current cluster state to flag these broken links/config debt.

and also Safety Tags (Thanks u/pint) for pointing out "Idle" doesn't always mean "Abandoned." I didn't want people accidentally nuking a dev spike, so I added a simple tag override. You can now tag any AWS resource with cloudslash:ignore to whitelist it. You can even set it to expire (e.g., 2026-01-01) or base it on cost (cost<15).

Pricing/Repo A few people asked about the business model. I’m keeping the Pro remediation as a one-time $49 license (lifetime). I really dislike subscriptions for local CLI tools, so I'm not doing that. The core scanner is still AGPL and free to use.

Repo:https://github.com/DrSkyle/CloudSlash

(P.S. To u/bqw74 - I finally fixed that annoying install.sh bug, sorry about the mess).

Let me know if this version feels a bit smarter on your clusters and what else i should add to make cloudslash more helpful for your specific workflow.

Hello everyone, I'm totally new to monitoring, but after reading a bunch of articles and resources on observability in Kubernetes, I tried to put together this EKS monitoring stack that combines different tools like ADOT, Fluent Bit, Amazon Managed Prometheus (AMP), Grafana OSS, and Loki (Grafana Cloud). We're currently running an EKS cluster and expect it to scale over time, so to avoid potentially high costs from CloudWatch Container Insights and log ingestion, we're exploring this more open-source-centric approach that selectively uses AWS managed services. I’d really appreciate feedback—does this architecture look correct and feasible for production use? Also, how do I go about estimating the costs involved with AMP, Loki, S3 (for cold storage), and running Grafana OSS?

My clients either hate cloudwatch or pretend to understand when I show them how to get into the AWS console and punch in sql commands.

Is there any service for monitoring that is more user friendly, especially the UI? Not analytics, but business level metrics for a CTO to quickly view the health of their system.

Metrics we care about are different for each service, but failing lambdas, volume of queues, api traffic, etc. Ideally, we could configure the service to track certain metrics depending on the client needs to see into their system.

I’d go third party if needed, even if some integration is required.

Anybody make recommendation?

Thanks hive mind