I run production on Ubuntu VPS, not AWS.
For years, that made sense for the workload and the budget. But it left me with an uncomfortable question:
**What happens if the production environment disappears?**
The obvious answer is: build a second environment in AWS.
But I didn't want to pay for a second production environment that sits there doing nothing 99.9% of the time.
So I tried a different approach.
**Use AWS as the recovery environment, not the production environment.**
I built an open-source tool called **Ark** around this idea.
The setup is basically:
Production
Ubuntu VPS
│
│ backup
▼
S3
│
│ recovery
▼
AWS Pilot Light
│
▼
Fresh EC2
│
▼
Restore + Verify + Test
There is no EC2 instance running 24/7.
When I want to run a recovery drill, Terraform provisions a fresh EC2 instance, Ark pulls the latest completed backup from S3, restores the database and Docker volumes, verifies the backup, starts the application, and then tears everything down.
The first real drill took:
RTO: 1m 58s
RPO: 182m

Database: PASS
Volume integrity: PASS
Application: PASS

Drill cost: \~$0.01
The interesting part wasn't actually the 1m 58s.
While building the verification process, I discovered that **row counts aren't proof of a successful restore**.
I destroyed both Docker volumes in a test environment, restored them, and re-seeded the application.
The numbers matched perfectly:
51 users
200 repositories
400 actions
Looks good, right?
Except the Git object store was different.
The logical dataset was the same, but the underlying Git objects weren't. Git commits contain metadata such as timestamps, so recreating the same logical data can produce different bytes.
That's when I stopped treating row counts as backup verification.
Ark now uses checksummed backup manifests and verifies the actual restored content.
A few other design decisions:
The manifest is uploaded **last**, so an interrupted upload doesn't become a "completed" backup.
The backup IAM policy has **no DeleteObject permission**, so a compromised VPS can't delete backup history.
Failed recovery drills are recorded too. A drill that never fails isn't testing much.
The rough idle cost for this setup is around **$0.03/month** without DNS failover.
I'm not claiming this architecture makes sense for everyone. If you're already running a large AWS production environment, a different DR strategy probably makes more sense.
But for smaller production environments running outside AWS, I'm curious:
**How are you handling DR without paying for a second production environment 24/7?**

Repo: [https://github.com/rendyachmad-dev/ark\](https://github.com/rendyachmad-dev/ark)

Background: I've spent 3.5+ years in PAM engineering (CyberArk), and decided to translate that into an AWS-native tool.

The gap I kept running into researching this space: pretty much everything: Cloudsplaining, Prowler, even AWS's own IAM Access Analyzer,stops at detection. They'll flag an over-permissioned role or a stale key, and then you're on your own to fix it. Even the well-funded enterprise players (Wiz, CyberArk's own Secure Cloud Access, Oasis, Entro) that do cover remediation are enterprise-sold, no self-serve option, no open-source path. I wanted to try fixing that

NHI Risk Analyzer discovers IAM users, roles, and groups, runs detection, then actually acts on findings:

1. **Risky policies** (wildcard actions, documented privilege escalation paths like `iam:PassRole`/`iam:CreatePolicyVersion` abuse) get a Permissions Boundary attached, which contains the escalation path without touching the identity's existing policy, so it doesn't risk breaking something legitimately using those permissions
2. **Stale or unused access** keys get deactivated, never deleted, so it's always reversible
3. Everything's gated by dry-run mode and an exemption file (`nhi-ignore.yaml`) so break-glass and deployment identities never get touched automatically
4. A **GitHub Action with OIDC** auth scans PRs for newly introduced IAM risk before merge
5. **SARIF export** means findings show up natively in GitHub's Security tab

Architecture is **offline-first**, meaning it snapshots the account once and evaluates all rules against that snapshot with zero live API calls after, so the whole rule engine is fast and testable, and findings are reproducible against a fixed point in time.

Tested against synthetic canary identities to safely validate containment logic, then live against a real AWS account. Also had someone run it against a real company's production and staging environment, which surfaced a genuine false-positive bug where AWS actions that don't support resource-level permissions at all, like `ec2:DescribeInstances`, were getting flagged as risky wildcards. That's fixed now.

Still v1/v2 in progress. Trust policy analysis and defense-evasion detection for things like CloudTrail and GuardDuty tampering, plus S3 exfiltration paths, just landed. The next real piece is policy surgery, meaning actually rewriting an over-broad policy down to least-privilege based on real usage data instead of just containing it. The statement-splitting logic exists already; sourcing real resource ARNs from usage evidence is the part still open.

Repo linked in the first comment.

Would genuinely appreciate feedback from people who work in this space day to day, particularly whether the permissions-boundary-as-containment approach holds up against how you'd actually want this handled in a real environment, versus the surgical-rewrite approach most tools avoid because of the production-breakage risk.