← Back to Home
✍️ Yantratmika Solutions 📅 2026-07-06 ⏱️ 118 min read

Reduce the AWS Bills

We have hearda lot of people complain about growing AWS bills that eat into their numbers. Some blame AWS billing policies, others blame their fate. The fact is that the bill is high because of some common problems that can be identified and fixed.

This blog gives a detailed account of how we saved a startup, pulling their bill down to 60%. It gives precise details and checklists for how to go about the task. It covers the technical as well as non-technical problems that lead to a high AWS bill.

Ofcourse, we changed all the names!

A Field Guide to AWS Cost Optimization

Book One — The Infrastructure (fix it this quarter)

Book Two — The Architecture (refactor it over quarters)

Book Three — The Humans (the real reason)

Part 0 — The Bill That Broke the CFO

It was a Tuesday, which is when bad news at Nimbus traditionally arrives.

Priya Nair was three sips into her coffee when the all-hands calendar invite landed: “AWS Spend — Urgent — 30 min.” No agenda. Just the digital equivalent of a parent saying “we need to talk.” She knew, the way you know a pothole is coming, that this was going to be her problem soon.

Nimbus, for the uninitiated, is a Series-C consumer photo-sharing app. Forty million people use it to post pictures of their lunches, their dogs, and — increasingly — AI-generated portraits of their dogs eating lunch. It is a lovely business. It also, as of last month, spends $340,000 a month on AWS, a number that had grown so quietly and so consistently that nobody had thought to ask why until the CFO, doing the thing CFOs do, looked at a graph.

The graph went up and to the right. The revenue graph also went up and to the right, but noticeably less steeply. When two lines diverge like that, someone eventually gets a Tuesday meeting.

In that meeting, three things happened. First, Chad Ellison — Nimbus’s CTO, a man who owns more Patagonia vests than the average Patagonia store — explained that a big cloud bill is “a sign of scale” and that he’d literally bragged about it at a conference last month. (“We spend the GDP of a small island on AWS,” he’d said, to applause.) Second, Marcus Webb — Principal Architect, designer of the sacred Nimbus Platform 2.0 — insisted the architecture was correct and that any savings would compromise resilience “at scale.” Third, the CFO looked at Priya, the most senior engineer in the room who hadn’t yet said anything defensive, and said: “You. Figure out where the money’s going. You’re the FinOps lead now.”

Priya did not want to be the FinOps lead. Nobody wants to be the FinOps lead. It is a job title that sounds like a punishment and, for the first two weeks, feels like one. But she opened Cost Explorer, and what she found over the following ninety days is the reason this series exists.

What this series actually is

This is The Nimbus Chronicles — a field guide to AWS cost optimization told as the story of one company clawing back a runaway bill. It is meant to be funny, because the alternative to laughing at your cloud bill is crying at it, and it is meant to be accurate, because a joke that teaches you the wrong thing is just an expensive joke.

Every episode follows Priya through one category of waste: what it looked like at Nimbus, why it was happening, the real numbers, and exactly how she fixed it. The cast will become familiar. There’s Marcus, who can justify any expense with the phrase “we might need it at scale.” There’s Deepak, a genuinely brilliant engineer who has decided that the monolith — lovingly known as PhotoBlob — is finished and shall never be touched again. There’s Jenny, the junior engineer who actually reads the bill and keeps getting ignored. There’s Big Bertha, an m5.24xlarge running a single nightly cron job at 3% CPU. And there is the NAT gateway. Oh, we’ll get to the NAT gateway.

The map

The waste at Nimbus fell into three broad territories, and so does this series.

First, the infrastructure — the knobs you can turn this quarter. Oversized instances and the wrong chips (Part 1). Buying commitments like an adult (Part 2). Redundancy you’re paying for and will never need (Part 3). The invisible networking bill and its patron saint, the NAT gateway (Part 4). Storage that never forgets (Part 5). Databases behaving badly (Part 6). Serverless that turned out not to be costless (Part 7). Caching, or the art of not doing the same work twice (Part 8). Over-engineering, a.k.a. running a Kafka cluster for 500 messages a day (Part 9). And the boring governance plumbing that quietly saves millions (Part 10).

Second, the architecture — the technical debt you’ll refactor over quarters, not days. The distributed monolith and its friends (Part 11). And all the ways application code fights the cloud instead of using it (Part 12).

Third, the humans — because the real reason a bill stays huge is almost never technical (Part 13). Egos, incentives, vanity metrics, and the CTO who brags about the number that’s bleaking money.

We close with an epilogue: where Nimbus landed, what actually moved the needle, and the two changes that made everything else stick.

A note before we start: cost optimization is not about making everything as small and cheap as possible. A checkout service that falls over on Black Friday to save forty dollars is not a win. The goal is to stop paying for things that create no value — idle cores, duplicated data, redundancy nobody needs, work done twice — while keeping every dollar that actually buys you reliability, speed, or sleep. Priya’s north star, taped to her monitor by week three, read: “Cheap where it doesn’t matter, so we can spend where it does.”

Pour a coffee. Bertha’s been running all night for no reason. Let’s go find out why.

Part 1 — The Instance That Was Too Big to Fail

The email arrived at 8:47 a.m. Subject line: “Question about AWS.” No greeting. Just a screenshot of the Nimbus bill — $340,127.44 — and a single sentence from the CFO: “I was told this was a photo app, not a particle accelerator.”

Priya Nair read it twice. She was a Staff Engineer who, as of eleven minutes ago, was also the “FinOps lead” — a title she had not applied for and could not find in the org chart. She poured coffee, opened Cost Explorer, and filtered to EC2. Compute was 61% of everything. Somewhere in that number was a machine the team had affectionately named Big Bertha.

Big Bertha was an m5.24xlarge. Ninety-six vCPUs. 384 GiB of RAM. Roughly $4.60 an hour on-demand, which is about $3,350 a month, which is about $40,000 a year. Bertha ran exactly one thing: a nightly cron job that resized some thumbnails at 3 a.m. Priya pulled up CloudWatch. Bertha’s peak CPU utilization, ever, was 3%.

“We might need it at scale,” said Marcus Webb, appearing behind her like a compliance ghost. He had designed Bertha. He would defend Bertha.

Priya sipped her coffee. “Marcus. It’s running a cron job. At three percent. You could do this on a Raspberry Pi taped to the wall.”

The most expensive words in the cloud: “just in case”

Compute waste at Nimbus wasn’t one villain. It was a hundred small ones. Here’s the tour Priya took, clipboard in hand.

Instances that are simply too big

Bertha is the mascot, but she has cousins everywhere. Over-provisioned instances — sized for an imagined future — are the single most common form of cloud waste. The fix is boring and effective: right-sizing. AWS Compute Optimizer analyzes actual CPU, memory, and network history and tells you, per instance, “you could drop two sizes.” Dropping Bertha from a 24xlarge to, say, a 2xlarge isn’t a discount; it’s a 12x cut on that line. Trusted Advisor flags the low-utilization offenders too. Priya started a spreadsheet titled “Bertha and Friends.”

Instances that do nothing at all

Then there were the idle instances — dev boxes spun up for a demo in 2024, still humming, doing nothing but generating charges. Eight of them. t3.large, ~$60/month each. Nobody could say what they were for, which is the surest sign you can turn them off.

Old-generation instances

Nimbus was running a fleet of m4 instances. AWS’s newer generations — m5, m6 — are cheaper and faster per dollar. Same story for t2 → t3: t3 is a straight upgrade at lower cost. Moving an m4 to an m6i is often a ~10–20% price improvement for more performance. There is almost never a reason to stay on old gen; you’re paying more to run slower.

Intel/AMD vs. ARM Graviton

The bigger lever: Graviton. AWS’s ARM-based chips (the g suffix — m6g, c7g, r7g) run about 20% cheaper than the equivalent Intel/AMD instances, often with better performance-per-watt. If your workload compiles for ARM — and most modern runtimes (Java, Go, Node, Python) do — this is close to free money. Priya flagged the stateless web tier as candidate one.

The wrong instance family entirely

Instance families are shaped for different jobs: c (compute) is CPU-heavy and RAM-light; m is balanced; r is memory-heavy. Nimbus was running an in-memory cache on an m5 — starving for RAM, wasting vCPUs. Moving it to an r family gave more memory per dollar. Meanwhile a CPU-bound encoder sat on an r instance paying for RAM it never touched. They had it backwards. Matching family to workload shape is free performance.

Burstable T-family misuse

The T-family (t3, t4g) is burstable: it earns CPU credits when idle and spends them when busy. Brilliant for spiky, low-average workloads. But Nimbus had a steadily-busy API service on a t3.medium, blowing through credits and — because they’d left unlimited mode on — paying surcharge fees for sustained CPU. The T-family charges you both ways: idle-but-oversized wastes the baseline; busy-past-credits racks up overage. A steady workload belongs on an m or c, not a T.

The idle GPU

In us-west-2 sat a p3.2xlarge — a GPU instance at roughly $3/hour — spun up for an ML experiment that ended in Q1. Nobody killed it. That’s ~$2,200/month for a graphics card doing absolutely nothing. Idle GPUs are the most expensive idle you can own. Priya terminated it on the spot and felt something close to joy.

The storage nobody looks at

Compute has a shadow: the disks and images bolted to it.

Time is a resource, and you’re paying for all of it

Non-prod running 24⁄7

Nimbus’s dev, staging, and QA environments ran around the clock. Nobody works nights and weekends (allegedly). If you schedule non-prod off outside business hours — say 8 a.m.–8 p.m., weekdays only — you eliminate roughly 70% of those hours. Cut a $30,000/month non-prod footprint by 70% and you’ve found $21,000/month by doing nothing but turning things off when everyone’s asleep.

No autoscaling

The production web fleet was a static fleet sized for peak — enough servers for the busiest minute of the busiest day, running that many 24⁄7. That’s like renting a 40-seat bus to commute alone because once a year you carpool. Autoscaling adds and removes capacity as load changes, so you pay for the average, not the maximum.

Autoscaling, but misconfigured

Worse than no autoscaling is autoscaling that scales up fast and scales down never. Nimbus’s group had a timid scale-down policy — a 30-minute cooldown and a conservative threshold — so it ballooned during traffic spikes and then sat there, fat and happy, for hours. Tune the scale-down as deliberately as the scale-up, or you get all the cost of peak with none of the savings.

Not using Spot

For anything fault-tolerant, batch, or stateless — the thumbnail pipeline, the video transcoders, CI runners — Spot Instances offer up to 90% off on-demand. Yes, AWS can reclaim them with two minutes’ notice, which is exactly why you use them for interruptible work, not your database. Nimbus’s entire batch encoding fleet was on-demand. Moving it to Spot was a potential ~$18K/month swing.

Half-empty container nodes

Finally, the containers. Nimbus ran EKS, and Priya found nodes at 40% packed — big EC2 instances hosting a handful of small pods, the rest of the capacity billed and empty. This is a bin-packing problem. Tools like Karpenter or cluster-autoscaler consolidate pods onto fewer, right-sized nodes and drain the empties. For spiky container work, Fargate Spot runs tasks on spare capacity at a deep discount. Better bin-packing alone took the cluster from ~40% to ~80% utilization — halving the node count.

By lunch, Priya had a spreadsheet, a Graviton migration plan, and a very quiet GPU. Marcus was still muttering “at scale.” Big Bertha, mercifully, was scheduled for demolition.

Takeaways

Part 2 — Commitment Issues

Chad Ellison swept into the FinOps sync forty minutes late, Patagonia vest zipped to the chin, radiating the confidence of a man who had just told a conference audience that Nimbus spends “the GDP of a small island” on AWS. He said it like a boast. Priya had watched the recording. The audience had laughed. The CFO, she suspected, had not.

“I’ve been thinking,” Chad announced, which was never a phrase that saved money. “We should just buy three-year, all-upfront, everything. Lock in the discount. Maximum savings. Boom.” He mimed a small explosion with both hands.

Priya set down her coffee. “Chad. If we commit three years to our current fleet, and half that fleet is Big Bertha and eight idle dev boxes, we will have signed a thirty-six-month lease on machines we’re actively trying to delete.”

Chad’s hands lowered slowly. “…Go on.”

The problem with paying full price forever

Right now, Nimbus paid 100% on-demand for its entire compute footprint — including a rock-steady baseline that never, ever went away. That baseline was the mistake. On-demand is the flexible, pay-per-second, no-strings pricing. You pay a premium for the privilege of walking away at any moment. But Nimbus was never going to walk away from its core API servers. It was paying walk-away prices for machines that had been running continuously for two years.

That’s what Savings Plans and Reserved Instances are for. In exchange for a 1-year or 3-year commitment, AWS discounts your compute by up to ~72%. You promise to spend a certain amount (Savings Plans) or reserve specific capacity (RIs); AWS gives you a large discount on the part of your usage that was going to happen anyway.

Savings Plans vs. Reserved Instances

Both hit up to ~72% off on-demand at the 3-year, all-upfront end. But the flexibility differs enormously, and that’s where Chad’s plan went sideways.

The wrong commitment type

There are three flavors, and picking wrong is its own kind of waste:

  1. Compute Savings Plans — the most flexible. The discount follows your usage across instance family, size, region, OS, and even between EC2, Fargate, and Lambda. Migrate from m5 to Graviton m7g? Move regions? The Savings Plan just… keeps applying. Slightly smaller max discount, dramatically less risk.
  2. EC2 Instance Savings Plans — a deeper discount, but locked to a specific instance family in a specific region (e.g., “m-family in us-east-1”). You can change size and OS within that family, but stray outside it and you’re back to on-demand.
  3. Reserved Instances — deepest and most rigid; best reserved (pun intended) for stable workloads you’re certain won’t move, or when you specifically need that zonal capacity reservation.

Priya’s rule for Nimbus: default to Compute Savings Plans for flexibility, use EC2 Instance SPs or RIs only for the workloads so stable and so understood that the extra few points of discount are worth surrendering the ability to change your mind. Given that they were mid-migration to Graviton (see Part 1), locking into an EC2 Instance SP for the Intel m-family would have been paying extra to strand themselves on the chips they were trying to leave.

The one insight that matters: commit to the baseline, not the peak

Here is the whole game, on one napkin.

Plot Nimbus’s compute usage over a week. There’s a baseline — the floor the usage never drops below, maybe 50 vCPUs’ worth, humming 24⁄7. And there’s a peak — the Friday-night photo-upload surge that briefly triples it. The correct move is to commit to the stable baseline and leave the peak on on-demand (or Spot).

Say the baseline is a steady $12/hour of on-demand compute and the peaks spike to $30/hour for a few hours a day. If you buy a Savings Plan for $12/hour, you discount the part that’s always there — up to ~72% off that slice — and let the spiky top float on flexible pricing. Concretely: $12/hour on-demand is ~$8,760/month; at a ~50% effective Savings Plan discount that’s ~$4,380/month, saving roughly $4,380/month on the baseline alone**, with zero risk, because that usage is never going away.

Chad’s plan — commit to the peak — is the trap. If you buy a $30/hour commitment to cover the spikes, you’re now paying $30/hour all the time, including the 20 hours a day you only use $12. You’ve converted a discount into a penalty. Over-committing locks in unusable spend: you pay for reserved capacity that sits idle, and it’s often worse than the on-demand you were trying to avoid. Commit the floor. Never the ceiling.

The commitments you already have (and are wasting)

Nimbus wasn’t starting from zero. Somebody, at some point, had bought RIs. Priya opened the coverage and utilization reports in Cost Explorer and found the usual archaeology.

Unused and underutilized RIs and Savings Plans

Some RIs had low utilization — the team had migrated the underlying instances to a different family months ago, and the rigid Standard RIs were now attached to nothing, discounting usage that no longer existed. That’s pure waste: you prepaid for a discount and then deleted the thing it applied to. The utilization report shows this as a percentage; anything well under 100% is money evaporating. The coverage report shows the opposite gap — how much of your eligible on-demand usage isn’t covered by any commitment, i.e., where you’re still overpaying.

RIs expiring into coverage gaps

Worse, a batch of 1-year RIs was about to expire. When an RI or Savings Plan lapses and nobody renews it, the covered usage silently falls back to full on-demand — a quiet, automatic price increase that shows up as a nasty surprise on next month’s bill. Priya set calendar alerts 30 days before every expiration. Commitment coverage isn’t “set and forget”; it’s a rolling ladder you have to keep climbing.

Not sharing across the Organization

Nimbus had multiple AWS accounts under one AWS Organization — prod, staging, data, that one account nobody claims. By default, with RI/Savings Plan sharing enabled at the Organization level, a commitment bought in one account applies to matching usage across all linked accounts. Nimbus had sharing off, so an underused RI in the prod account sat idle while the data account paid full on-demand for the exact instance type that RI covered. Flipping on sharing let the discounts flow to wherever the usage actually was — no new purchase required, just a checkbox and a pooled commitment.

How Priya actually rolled it out

She didn’t buy three-year-all-upfront-everything. She did the responsible, slightly boring thing:

  1. Finish the right-sizing and Graviton migration first (Part 1), so the baseline she committed to was the optimized baseline, not the wasteful one.
  2. Buy a Compute Savings Plan covering ~80% of that stable baseline, on a 1-year term to start — enough discount to matter, short enough to adapt.
  3. Turn on Organization-wide sharing so every account drank from the same discounted well.
  4. Set expiration alerts and a monthly ritual of reading the coverage and utilization reports.

Chad, to his credit, came around. “So we’re still saving a ton,” he said, “we’re just… not signing our own hostage note.”

“Exactly,” said Priya. “Commit to what’s boring and permanent. Stay flexible on everything else.”

Takeaways

Part 3 — Redundancy Theater

The disaster recovery drill was Marcus’s idea. We would simulate the loss of an entire AWS region and prove that Nimbus, home to 40 million people’s brunch photos, could shrug off a meteor strike on us-east-1.

Priya sat in the back of the war room, laptop open to Cost Explorer, watching Marcus click through his architecture diagram. Everything was blue. Everything was doubled. Everything, somewhere, had a twin quietly humming in another Availability Zone, another region, another account, drawing breath and dollars in equal measure.

“See,” Marcus said, gesturing at a constellation of boxes, “if us-east-1 goes down, we fail over to us-west-2 seamlessly. Full replication. Multi-AZ on everything. We’re bulletproof.”

“We’re paying for bulletproof glass on the garden shed,” Priya said, not quite under her breath. Chad, mid-sip of cold brew, laughed. Marcus did not. “We might need it at scale,” he said, and Priya wrote the phrase on a sticky note and pressed it to the side of her monitor like a specimen.

The Theater, and What the Audience Actually Paid For

Here is the thing nobody wants to say out loud in the DR meeting: redundancy is not free, and most redundancy at most companies is theater. It looks like resilience. It costs like resilience. But nobody has ever tested whether the understudy can actually go on.

Nimbus was spending an enormous amount on standbys, replicas, and copies that existed because someone once checked a box labeled “highly available” and never checked it again. Let’s walk the set.

Multi-AZ Everywhere, by Default

Multi-AZ is the reflex. You spin up an RDS instance, you tick “Multi-AZ deployment,” and AWS provisions a synchronous standby in a second AZ. Your write latency ticks up slightly, your failover becomes automatic — and your instance cost roughly doubles, because you are now paying for two instances instead of one.

That is a genuinely good trade for your production user database. It is a terrible trade for the seven non-production databases Nimbus had also set to Multi-AZ out of pure muscle memory.

Concrete math: a db.r6g.2xlarge in us-east-1 runs about $0.968/hour on-demand for a single instance — roughly $706/month. Flip on Multi-AZ and you are at about $1,412/month for that one database. Nimbus had this configuration on a dev database that three engineers touched during business hours and that would be gently mourned, then rebuilt from a snapshot, if it ever vanished. That’s about $700/month of standby guarding a database whose real recovery requirement was “recreate it Tuesday.”

The rule Priya wrote down: Multi-AZ is for genuinely critical, stateful production services where an AZ failure would be a real incident. Dev, test, and staging can run single-AZ. If they fall over, you relaunch them. That is what dev is for.

Multi-Region Replication of Things Nobody Would Miss

Marcus’s crown jewel was cross-region everything. And some of it was legitimate: the core user metadata, the billing ledger — yes, replicate those, because losing them is an extinction event.

But Nimbus was also replicating thumbnail caches, transcoded video derivatives, and analytics event logs across regions “for resilience.” These are all regenerable. The thumbnails can be re-rendered from originals. The analytics logs are already in the data warehouse. Paying to keep a hot copy of them 2,000 miles away protects against a disaster that, if it happened, you’d solve by pressing “rebuild.”

The discipline here is to start from your actual DR and RPO requirements — Recovery Point Objective, how much data you can afford to lose — and replicate only what those numbers demand. If your RPO for thumbnail caches is “who cares, regenerate them,” they do not belong in a cross-region replication policy.

Over-Replicated Storage: S3 CRR, Ghost Replicas, Idle Standbys

This is where the money hides, because storage redundancy is quiet.

S3 Cross-Region Replication (CRR): Nimbus had CRR turned on for a 400 TB bucket of user uploads. CRR means you pay for the destination storage again (another ~$0.023/GB-month for S3 Standard), plus inter-region data transfer at roughly $0.02/GB for every object replicated. 400 TB of duplicated storage is about $9,400/month just for the second copy, before a byte of transfer. Some of that bucket genuinely needed geographic redundancy. A large chunk of it was derived data that did not.

Read replicas nobody reads: Nimbus had three RDS read replicas on the PhotoBlob database. Flow logs and Performance Insights showed one of them served zero query traffic for the trailing 90 days. It was created for a reporting feature that shipped, got deprecated, and left its replica behind like a tenant who moved out but kept paying rent. That’s another full instance-cost — call it $700+/month — for a machine answering no questions.

Unused standbys: the DR region had warm standby EC2 fleets that had never once taken production traffic. Warm standby is a valid strategy, but a warm standby you never test is just an expensive standby. If you’re paying to keep it warm, you should be failing over to it on a schedule to prove it works — otherwise downgrade to pilot-light (minimal always-on footprint, scale up on disaster) and pocket the difference.

Duplicate Environments Breeding in the Dark

Priya inventoried the non-prod accounts and found: a staging stack, a qa stack, a staging-2 stack from a migration that finished in 2024, a demo environment for a sales team that now used a SaaS demo tool, and something called perf-test-DO-NOT-DELETE that no one could explain and everyone was afraid of.

Five overlapping environments, each with its own load balancers, databases, and half-scaled compute. Most were 80% idle. This is redundancy nobody even chose — it accreted. The fix is unglamorous: consolidate. One well-maintained staging environment that mirrors production is worth more than five stale ones, and it costs a fraction. Nimbus collapsed the five into two (staging + a shared ephemeral QA that spins up per-branch and tears down nightly).

Backups: Over-Retained and Over-Frequent

Finally, backups — where good intentions become a storage line item that only grows.

Nimbus was taking hourly automated snapshots of PhotoBlob and retaining them for two years. Snapshots are incremental, but two years of hourly snapshots across a large database is a genuinely large pile of EBS snapshot storage (~$0.05/GB-month), and it was climbing every hour of every day.

Nobody had ever restored a snapshot older than about a week. The real recovery need was “roll back a bad deploy” (hours) and “satisfy the 90-day compliance window” (quarters, not the whole database, and not hourly granularity). Priya matched retention to actual recovery needs: hourly snapshots kept for 7 days, daily for 30, monthly for a year for compliance. Same protection for every recovery scenario that had ever actually occurred — a fraction of the storage.

How the Drill Ended

They ran Marcus’s meteor-strike drill. Failover to us-west-2 worked for the core services — the ones that genuinely warranted the redundancy. And the exercise revealed, gently and expensively, that two-thirds of the doubled infrastructure was protecting things that either regenerated themselves or nobody would notice were gone.

Marcus kept his cross-region replication on the user ledger and the billing data, where it belonged. Priya turned off CRR on the thumbnail bucket, killed the ghost read replica, consolidated the environment zoo, and put the backups on a diet. The sticky note — “we might need it at scale” — stayed on her monitor, but she added one underneath it: “at which scale, for which failure, tested when?”

The redundancy that survived was the redundancy that answered those three questions. Everything else was theater, and the audience was the CFO.

Takeaways

Part 4 — The Case of the Mysterious NAT Gateway

The line item just said “EC2-Other.” $31,247 of it, per month, and nobody could say what it was.

Jenny found it first, because Jenny reads Cost Explorer the way other people read the news. She wandered over to Priya’s desk holding a laptop like it was a small animal that had bitten her. “There’s a category called EC2-Other and it’s our fourth-biggest cost and I have no idea what’s in it.”

Priya knew, the way you know the sound of a specific creaking stair in a house you grew up in. “That’s the network,” she said. “That’s where the invisible bill lives.” And somewhere in that $31,247, purring contentedly, was the thing the team had named years ago in a postmortem and never stopped fearing: the NAT gateway from hell.

The Invisible Bill

Compute and storage have honest bills. An instance costs what it costs; a bucket bills by the gigabyte. Networking is different. Networking charges you for movement — bytes crossing invisible boundaries you didn’t know you’d drawn — and the charges only show up as cryptic usage types buried in “EC2-Other.” Nobody provisions a data-transfer charge. It just happens, quietly, a fraction of a cent at a time, forty million times an hour.

Let’s turn on the lights.

The NAT Gateway From Hell

A NAT Gateway lets resources in a private subnet reach the internet (to pull packages, call third-party APIs, hit public endpoints) without being publicly addressable themselves. Useful. But it bills two ways at once:

That second number is the killer. Nimbus routed a firehose of traffic through NAT gateways — including, it turned out, traffic to S3. Every thumbnail write, every upload to the PhotoBlob bucket, was leaving the private subnet, passing through the NAT gateway (at $0.045/GB), and then hitting S3. At Nimbus’s volume — call it 300 TB/month through NAT — that’s roughly $13,500/month in NAT processing alone. For traffic to a service AWS will let you reach for free.

The Free Win Everyone Misses: Gateway VPC Endpoints

Here is the single highest-leverage fix in this entire series.

S3 and DynamoDB have Gateway VPC Endpoints, and they are free. No hourly charge, no per-GB charge. You add a route in your route table, and traffic to S3/DynamoDB goes over the AWS private network instead of out through the NAT gateway. You stop paying $0.045/GB. You stop paying for the internet round-trip you never needed.

Nimbus’s 300 TB/month of S3 traffic through NAT dropped to $0 in processing the moment a Gateway Endpoint was added. That one route-table change was the biggest single win Priya found all quarter. Deepak, who’d built the original VPC, went a bit quiet. “It was one checkbox,” he said. “It was always one checkbox.”

S3 and DynamoDB get the free Gateway endpoints. Other AWS services — ECR, Secrets Manager, SQS, Kinesis, CloudWatch, SSM, and friends — use Interface VPC Endpoints (PrivateLink) instead. These keep traffic to those services off the NAT gateway and on the private network.

The catch: Interface endpoints aren’t free. Each bills $0.01/hour per AZ ($7.20/month per AZ) plus ~$0.01/GB processed. Still far cheaper than NAT’s $0.045/GB for high-volume services — Nimbus’s container fleet pulling images from ECR all day paid down its NAT bill fast by adding an ECR interface endpoint.

But there’s a trap on the other side, which brings us to sprawl.

Interface Endpoint Sprawl

After the ECR win, someone got enthusiastic and started adding interface endpoints for every service, in every AZ, across every VPC. Because each is billed per-hour, per-AZ, this adds up. Fifteen interface endpoints across three AZs is 45 × $7.20 = ~$324/month in hourly charges before any traffic — and several were for services with trivial traffic that would’ve been cheaper left on NAT.

Interface endpoints are worth it for high-volume service traffic. For a service you call twice a day, the endpoint’s hourly cost exceeds the NAT processing you’d save. Right-size them; don’t collect them.

Cross-AZ Traffic: The Tax You Pay for Sloppy Placement

Traffic between Availability Zones costs ~$0.01/GB in each direction — so ~$0.02/GB round-trip. This is small per gigabyte and enormous in aggregate when your services are chatty and scattered carelessly across AZs.

Nimbus’s PhotoBlob monolith talked constantly to a caching tier that had been placed, for no reason anyone remembered, in a different AZ. Every cache read hopped an AZ boundary and back. VPC Flow Logs showed 80 TB/month of this cross-AZ chatter — roughly $1,600/month to move data between two machines that could have been AZ-aligned. Inter-service chattiness is fine within an AZ (free) and expensive across one. Co-locate the talkers.

Public Subnets, Wrong Paths, and Forced NAT

Several of the mysterious charges traced back to subnet design. Resources that should have been in private subnets with private routing were sitting in public subnets, or were forcing traffic out through NAT because no private path existed. A misplaced resource in the wrong subnet turns a free private hop into a NAT-processed, cross-AZ, sometimes-internet-egressing mess.

The broader principle: the data access path matters as much as the data. Reaching another AWS service can go over the public internet route (you pay egress), through a private endpoint/PrivateLink (cheap or free), or over Direct Connect (for on-prem). Same bytes, wildly different bills, decided entirely by which route table entry wins.

Internet Egress: The Priciest Bytes

Data leaving AWS for the public internet is the most expensive transfer of all — roughly $0.09/GB for the first tier out of us-east-1, tapering with volume. Nimbus serves photos to 40 million users, so egress is unavoidable — but how you serve it isn’t.

Serving images straight from S3 or EC2 to users pays full egress every time. Putting CloudFront in front changes the economics: CloudFront’s egress rates are lower, transfer from S3/EC2 to CloudFront is free, and caching means you serve most requests without touching the origin at all. Moving photo delivery behind CloudFront cut Nimbus’s origin egress dramatically — the CDN both lowered the per-GB rate and slashed how many GB left the origin in the first place.

Cross-Region and Sustained High-Volume Transfer

Cross-region traffic (~$0.02/GB between regions) had crept in via the replication we trimmed back in Part 3 — another reason to replicate only what DR truly requires.

And for sustained, high-volume transfer — think a steady pipe to an on-prem data center or partner — the internet route is the wrong tool. Direct Connect provides a dedicated private connection with lower per-GB rates for consistent heavy traffic; a VPN over the internet is cheaper to stand up but still pays internet egress and gives less predictable throughput. For a Nimbus analytics feed shipping terabytes to a partner monthly, Direct Connect beat both VPN and raw internet egress once volume stayed high month over month.

Idle Load Balancers and Unattached Elastic IPs

Finally, the loose change that bills by the hour whether you use it or not:

Cracking the Case

The tools that solved it: Cost Explorer with usage-type filters (filter on NatGateway-Bytes, DataTransfer-Regional-Bytes, DataTransfer-Out-Bytes to see exactly which movement is costing what) and VPC Flow Logs (to see which services are talking, across which boundaries, in what volume). Between them, “EC2-Other” stopped being a mystery and became a to-do list.

Jenny presented the findings. Chad, who bragged that Nimbus spent “the GDP of a small island” on AWS, was delighted to learn that a meaningful chunk of that GDP had been spent moving photos in a circle through a gateway they didn’t need. The NAT gateway from hell still runs — every VPC needs some egress — but it now handles only the traffic that genuinely has to reach the open internet. Everything else found a cheaper, quieter road.

Takeaways

Part 5 — The Bucket That Ate the Budget

Priya found the bucket at 11:40 on a Tuesday, which is the hour of day when nothing good is ever discovered.

It was called nimbus-photoblob-prod, and according to S3 Storage Lens it held 4.1 petabytes. She read the number three times. Then she opened Cost Explorer, filtered to S3, and watched a single line item bloom across the chart like a bruise: $71K a month. For one service. For a company that, as Chad enjoyed reminding the all-hands, “spends the GDP of a small island” on AWS.

“That can’t be right,” said Jenny, leaning over. “Forty million users, sure, but the photos aren’t that big.”

“They’re not,” said Priya. “The photos are fine. It’s everything around the photos.” She scrolled. Thumbnails. Re-encodes. A folder named temp/ last touched in 2024. And, beautifully, a top-level prefix called bertha-migration-backup/ that nobody could explain and everybody was afraid to delete. Big Bertha, reaching out from the grave to bill us.

The photos are fine. The plumbing is a crime scene.

S3 pricing rewards you for putting cold data in cold places and punishes you for hoarding hot copies of things nobody looks at. Nimbus was doing the opposite with enormous discipline.

Lifecycle policies, or the absence thereof

Every object in nimbus-photoblob-prod was sitting in S3 Standard, which in us-east-1 runs about $0.023 per GB-month. That’s the right tier for a photo uploaded ten minutes ago that the user is actively sharing. It is an insane tier for a photo from 2019 that will be opened, on average, never.

S3 offers a temperature ladder, and lifecycle policies let objects walk down it automatically based on age:

Do the arithmetic on one petabyte of genuinely cold photos. In Standard that’s roughly $23,000/month. The same petabyte in Deep Archive is about $990/month. Nimbus had at least two petabytes that hadn’t been touched in over a year. A lifecycle rule — Standard for 30 days, IA at 30 days, Glacier at 90, Deep Archive at 365 — would have moved them without a human ever opening a ticket.

Priya wrote the rule that afternoon. Deepak asked whether restores would “make profile loads slow.” Priya explained, patiently, that you only transition data users aren’t loading, which is the entire point.

Intelligent-Tiering for the stuff you can’t predict

Lifecycle rules assume you know an object’s access pattern by its age. For a lot of Nimbus data — shared albums that go viral for a week two years after upload — you don’t. That’s what S3 Intelligent-Tiering is for: S3 monitors access per object and moves it between frequent and infrequent tiers automatically, for a small monitoring fee (about $0.0025 per 1,000 objects/month). No retrieval charges when data warms back up.

The catch is the monitoring fee: it’s death by a thousand cuts if your objects are tiny. Which brings us to Nimbus’s second crime.

A billion tiny objects

PhotoBlob stored every image as a family: original, plus thumbnails at six sizes, plus a JSON sidecar of EXIF data. Seven-plus objects per photo, most of them a few kilobytes. Across the fleet, roughly 1.5 billion objects.

Storage of tiny objects is cheap. Requests are not. S3 charges per request — about $0.0004 per 1,000 GET and $0.005 per 1,000 PUT in Standard. When your app renders a feed by issuing millions of GETs against millions of postage-stamp thumbnails, the request line becomes real money, and IA/Glacier tiers add per-object minimums and retrieval fees that make tiny cold objects worse, not better. The fix is architectural — sprite the thumbnails, cache aggressively at CloudFront, and stop treating S3 like a filesystem — but even naming the problem via Storage Lens’s object-count and request metrics was new information to the team.

The uploads that never finished

Then Priya found the quiet one. Multipart uploads let you send a big file in chunks. If a chunk fails and the client never calls Complete or Abort, those uploaded parts sit in the bucket billing you at full Standard rate, invisible in the normal object listing.

Nimbus’s mobile clients dropped connections constantly — subways, elevators, the void. Storage Lens reported incomplete multipart uploads totaling ~180 TB. That’s roughly $4,100/month for uploads that, by definition, no user ever completed.

The fix is one lifecycle rule: AbortIncompleteMultipartUpload after 7 days. It should be on literally every bucket you own. Priya added it everywhere and felt, briefly, like a person who had accomplished something.

Versioning without a curfew

nimbus-photoblob-prod had versioning enabled — sensible, protects against accidental deletes. But versioning without a non-current version expiration rule means every overwrite keeps the old copy forever, and every “delete” just drops a delete marker on top of a stack of retained versions you’re still paying to store.

Storage Lens showed non-current versions accounting for nearly 300 TB. Nobody could restore a two-year-old thumbnail version and nobody would want to. A lifecycle rule to expire non-current versions after 30 days and clean up expired delete markers reclaimed the lot — another ~$7,000/month of storage nobody knew existed.

Logs, temp, dupes, and abandoned buckets

The rest was housekeeping, but housekeeping at petabyte scale is money:

The block storage nobody looked at

S3 gets the headlines, but Priya ran the same audit on block storage:

The number at the bottom

By Friday, Priya’s spreadsheet said the storage changes — lifecycle transitions, aborted multiparts, expired versions, deleted duplicates, cold-tiered logs, right-sized EBS/EFS — would take the S3-and-friends line from ~$71K to roughly $28K a month. She didn’t delete a single photo a user cared about.

Chad, told the number, nodded gravely and said, “That’s the GDP of a smaller island.” Priya took the win.

Takeaways

Caution!

Part 6 — Full Scan of Shame

The Slack message from Deepak was seven words long: “Profile loads are slow again, not my code.”

Priya opened Performance Insights for the primary RDS instance, sorted by load, and there it was — a single query eating 60% of database time, running thousands of times a minute: SELECT * FROM photos WHERE user_id = ?, with no index on user_id. Every profile load in PhotoBlob triggered a full table scan across 900 million rows. Deepak’s code. Deepak’s very own code, scanning the entire photos table to render one avatar.

“It’s a full scan of shame,” said Jenny, who had recently learned what a full table scan was and was enjoying it enormously.

Marcus wandered over. “We might need the flexibility of not indexing,” he offered, “at scale.” Priya wrote an index. It took ninety seconds. The query went from 4 seconds to 3 milliseconds, and the database’s CPU graph fell off a cliff. Nobody needed the flexibility.

That was the appetizer. The database bill was $96,000/month, and Priya had opinions about all of it.

The relational layer

Over-provisioned RDS and Aurora

The primary was a db.r5.8xlarge (32 vCPU, 256 GB RAM) running at 11% average CPU. It had been sized during the Big Bertha panic of 2025 and never revisited. Right-sizing to a db.r6g.2xlarge — a Graviton instance, which delivers roughly 20% better price-performance than the equivalent x86 — cut the instance cost by more than half. On-demand, an r5.8xlarge runs about $4.00/hour ($2,900/mo); an r6g.2xlarge is about $0.80/hour ($580/mo). Same workload, once the index existed.

Storage was on old-style gp2. Moving to gp3 decoupled IOPS from capacity and let Priya provision baseline 3,000 IOPS without over-buying storage, saving another ~20% on the storage line.

Idle databases and Aurora Serverless v2

There were fourteen RDS instances. Two were the real workload. The other twelve were per-team “staging” and “analytics” databases averaging under 5% CPU, several idle overnight and all weekend. Priya consolidated the stragglers onto shared instances, and for the genuinely bursty internal tools she moved them to Aurora Serverless v2, which scales capacity in fine-grained ACUs and can idle down when nobody’s querying — so the finance team’s once-a-day dashboard stops paying for a full instance 24⁄7.

Missing indexes, N+1, and the writer taking all the reads

The full-scan avatar query was the headline, but the slow-query log and Performance Insights surfaced two more patterns:

No connection pooling

Every application container opened its own pool, and at peak the database hit its max_connections ceiling and started rejecting connections — which the team had “fixed” by scaling up the instance for more RAM. The actual fix was RDS Proxy, which multiplexes thousands of application connections onto a small pool of database connections, smooths failover, and let Priya keep the smaller instance without connection storms.

DynamoDB: the two classic mistakes

Nimbus’s session and activity-feed data lived in DynamoDB, and it was making both textbook errors.

Backups, snapshots, and unused replicas

The analytics and big-data layer

The data platform was a second, quieter budget fire.

Athena scanning too much

Athena charges ~$5.00 per TB scanned. The analytics team’s dashboards queried raw JSON event logs with SELECT *, unpartitioned, and each dashboard refresh scanned ~2 TB — about $10 a click, thousands of times a day. Priya’s fixes were the canonical Athena playbook: partition the data by date, convert from JSON to Parquet (columnar, compressed), and select only the needed columns. The same dashboard query dropped from 2 TB scanned to under 40 GB — roughly $0.20 instead of $10.

Redshift running idle

A ra3.4xlarge Redshift cluster ran 24⁄7 for reports generated between 6 and 9 a.m. Options Priya weighed: pause the cluster outside business hours (RA3 supports pause/resume), or move to Redshift Serverless so it bills only while queries run. For a 3-hours-a-day workload, serverless cut the line item by ~80%.

EMR clusters left running

Two EMR clusters had been spun up for one-off backfills months ago and never terminated — a persistent bill for zero jobs. The pattern for batch analytics is transient clusters that launch, run the job, and self-destruct: enable auto-termination, and use Spot instances for task nodes (60–90% off on-demand) since task nodes are stateless and interruption-tolerant.

Glue over-provisioned

The nightly ETL Glue job was configured with 20 DPUs and finished in 6 minutes using a fraction of them. Right-sizing to the DPUs the job actually needed (and using auto-scaling where available) cut its cost proportionally — you pay per DPU-hour, so over-provisioning is pure waste.

Kinesis over-provisioned shards

The ingestion stream was provisioned at 50 shards sized for a launch that never fully materialized, billing per shard-hour regardless of throughput. Switching Kinesis Data Streams to on-demand mode let it scale with actual traffic instead of paying for permanent headroom.

OpenSearch oversized

The log-search OpenSearch domain ran a fat hot-tier cluster holding 18 months of logs nobody searched past week two. Priya moved older indices to UltraWarm (and cold storage beyond that), set an index lifecycle policy to delete stale indices after 90 days, and right-sized the hot nodes. Hot storage is expensive; UltraWarm is a fraction of it for data you rarely query.

The number at the bottom

Right-sizing, Graviton, gp3, killed idle instances, RDS Proxy, reader-endpoint routing, DynamoDB mode fixes and the Query redesign, plus the analytics cleanup, took the data layer from $96K to about $41K a month. The single largest contributor to the performance win was a ninety-second index on user_id.

Deepak, informed that his profile-load query was the most expensive line in the database, was quiet for a moment. Then he said, “So it was my code.” Personal growth.

Takeaways

Part 7 — Serverless, Not Costless

Priya found the recursive Lambda the way you find a gas leak: by noticing something was very wrong before you understood what.

The CloudWatch dashboard showed a function named thumbnail-fanout invoking 14 million times an hour. Nimbus has 40 million users. Nobody uploads 14 million photos an hour at 2 a.m. on a Tuesday. Priya traced it: the function wrote a resized image back to the same S3 prefix that triggered it. Every write fired a new event. Every event resized the image. Every resize wrote another object. Big Bertha, the ingest bucket, was cheerfully feeding a function that ate itself.

“It’s serverless,” Chad had told the CFO in March. “It scales to zero. It’s basically free.” Chad was wearing the Patagonia vest. The bill for Lambda alone that month was $31K, and roughly $9K of it was one function calling itself in a circle like a dog after its tail.

Serverless is wonderful. Serverless is not free. Let’s go through the ways Nimbus paid for the difference.

Memory is CPU, and CPU is money

The first thing Priya fixed was the most common Lambda mistake in the world: someone set memory to 128 MB “to save money.”

Lambda memory is not just memory. Memory is the dial that also sets vCPU — proportionally, up to roughly 6 vCPUs at 10,240 MB. At 128 MB you get a sliver of a core. Nimbus’s image-resize function at 128 MB took 8,000 ms to process a photo. Bumped to 1,024 MB, it took 900 ms.

You bill for GB-seconds. Do the math:

The “cheap” 128 MB setting was more expensive per invocation and nearly nine times slower. Too low starves the CPU and stretches billed duration. Too high — say pinning everything to 10 GB — burns money the workload never uses. The sweet spot is empirical, not vibes.

Priya ran AWS Lambda Power Tuning (the open-source Step Functions state machine) across each function. It sweeps memory settings, plots cost against speed, and hands you the optimum. Half of Nimbus’s functions landed between 512 MB and 1,536 MB. The resize function’s true optimum was 1,769 MB — a number no human would have guessed.

Still on x86 in 2026

Every Nimbus function ran on x86. Switching the runtime architecture to arm64 (Graviton2) is a one-line config change for most Python and Node functions, and Graviton Lambda is about 20% cheaper per GB-second — often faster too. On a $31K Lambda spend, that alone was roughly $6K/month for a checkbox. Deepak grumbled that PhotoBlob “might have a native dependency.” It had one. It had an ARM wheel. It shipped.

Chatty functions and the self-eating loop

The recursive thumbnail-fanout got a guard rail: resized output now writes to a different bucket than the trigger, plus an object-tag check so a reprocessed image can’t re-fire. Recursion loops are the serverless equivalent of leaving the tap running — AWS now has a recursion-detection kill switch, but relying on it is like relying on the smoke alarm to cook dinner.

The subtler waste was chattiness. A downstream notify function fired once per photo — 14 events, 14 cold-ish invocations, 14 sets of overhead. Priya put SQS in front of it and set a batch size of 100. Same work, roughly 1/100th the invocations. For true firehoses (the activity feed), the same idea with Kinesis batched records into fat, efficient calls. You pay per invocation and per GB-second; amortizing overhead across a batch cuts both.

Paying to wait

One function, await-encoding, called an external video service and then sat there polling for up to 45 seconds until the job finished. Forty-five seconds of billed Lambda duration doing nothing but sleeping on I/O. At 1,536 MB that’s about 0.066 GB-s per second of nap — pure waste, times millions of jobs.

The fix is a Step Functions state machine: kick off the encode, then wait using a .waitForTaskToken callback or a poll-and-wait loop between short Lambda invocations. Step Functions bills for state transitions, not for the wall-clock time you spend waiting. You stop renting compute to watch a progress bar.

When serverless is the wrong tool

Here is the heresy Marcus needed to hear. Lambda’s per-invocation model is a rounding error at spiky or low load and a liability at constant high load.

Nimbus’s feed-render function ran flat-out, 24⁄7, at steady throughput — it never scaled to zero because traffic never dropped to zero. Model the crossover:

At sustained, predictable, high throughput, containers or EC2 win — often by 2–3x — because you stop paying the per-invocation premium for elasticity you aren’t using. The rule: model the crossover. Bursty and unpredictable → Lambda. Flat and constant → containers. Nimbus moved the feed renderer to Fargate and left the spiky upload path on Lambda.

Provisioned concurrency you’re not using

To kill cold starts on the checkout function, someone had set provisioned concurrency to 200. Actual peak concurrency was 35. Provisioned concurrency bills for reserved capacity whether or not it’s invoked — it’s a standing charge. Nimbus was paying to keep 165 execution environments warm for nobody. Priya dropped it to 50 with an application-auto-scaling schedule that ramps it up before the evening peak and back down overnight. Cold starts stayed rare; the standing bill dropped by two-thirds.

Fat packages, slow starts

The cold starts that remained were made worse by deployment package bloat. The upload function shipped a 240 MB zip that dragged in the entire AWS SDK, pandas (used by nobody), and three copies of an image library. Bigger packages mean slower cold-start initialization — you pay in latency and in init duration. Priya trimmed to the specific SDK v3 clients actually imported, moved shared code to a Lambda layer, and got the package under 20 MB. Cold starts dropped from ~2.1 s to ~400 ms.

The front door: REST vs HTTP vs URL

Every API at Nimbus went through API Gateway REST APIs at $3.50 per million requests. Most of those routes were dumb proxies that used none of REST’s fancy features (request validation, API keys, WAF wiring).

Nimbus moved the high-volume public routes to HTTP API and internal webhooks to Function URLs. On a few hundred million requests a month, the delta was real four-figure money.

Standard vs Express Step Functions

Finally, the workflow bill. Step Functions Standard charges per state transition — great for long, durable, human-in-the-loop flows. Nimbus’s image pipeline was neither long nor durable: it ran in under a second, millions of times a day, and used Standard. At per-transition pricing that’s brutal.

Express workflows bill on duration and memory instead, and for high-volume, short-lived orchestration they’re often 10x+ cheaper. Priya converted the sub-second pipelines to Express and kept Standard only for the multi-day moderation-appeal flow that genuinely needs durability and full execution history.

Chad still tells the CFO serverless “scales to zero.” It does. The bill did too — from $31K to about $11K — once someone stopped it from eating itself.

Takeaways

Part 8 — Cache Rules Everything Around Me

Jenny found it in Cost Explorer, which is where Jenny finds everything now. She had filtered by usage type, sorted by cost, and gone quiet.

“Priya,” she said. “We generate the same thumbnail forty thousand times a day. The same one. The cat picture on the trending page. Every single request re-reads the original from S3, resizes it, and throws it away.”

Priya looked. Jenny was right. The trending grid on the Nimbus home screen was rendering thumbnails on the fly, per request, from the original full-resolution masters in Big Bertha. Forty million users, one trending cat, and a Lambda dutifully re-resizing the identical 12-megapixel image tens of thousands of times because nobody had ever told the system to remember the answer.

Deepak’s take, as always, was that PhotoBlob “was designed for correctness, not caching.” Correctness is wonderful. Correctness that recomputes the same JPEG forty thousand times is just an expensive kind of amnesia.

The month’s data-transfer and compute bill for serving photos was the second-largest line item at Nimbus, right behind the NAT gateway from hell. Almost all of it was work the system had already done and immediately forgotten. This is the chapter about memory — the good kind.

No caching layer at all

Start with the database. Nimbus’s feed service queried DynamoDB and RDS for the same hot rows constantly: the trending post list, popular user profiles, the “photo of the day.” Every page load hit the database for data that changed maybe once an hour.

Repeated identical reads are exactly what an in-memory cache is for. Priya put ElastiCache for Redis in front of the hot read paths. The feed query — user profile plus recent-post metadata — went from a 40 ms round trip that consumed provisioned RDS capacity to a sub-millisecond Redis hit that consumed almost nothing.

The effect compounds two ways. Latency drops, and so does the database itself: RDS read replicas that had been added purely to absorb read load could be scaled down. For the DynamoDB paths, the same logic applied through DynamoDB Accelerator (DAX) — a purpose-built in-memory cache for DynamoDB that returns cached items in microseconds and lets you drop provisioned read capacity units. For simple string/blob key-value lookups with no clustering needs, Memcached would have done the job just as well; Redis won here because Nimbus wanted sorted sets for the leaderboard.

The DB layer’s read cost fell by more than half once the cache absorbed the repetitive reads.

No CDN in front of anything

Now the big one. Nimbus served photos, avatars, thumbnails, CSS, and JavaScript directly from S3 and from the application load balancer — straight out of the region, to users on six continents.

Every byte left an AWS region as egress, at full data-transfer-out rates. Every request hit the origin. A user in Jakarta pulling the home feed dragged image bytes all the way from us-east-1, and Nimbus paid regional egress for the privilege, over and over, for content that never changed.

Putting CloudFront in front of it changes the economics twice. First, egress: data transfer out to the internet through CloudFront is billed at CDN rates, which are lower than raw regional S3 egress, and — critically — origin-to-CloudFront transfer from S3 in the same partition is free. Second, offload: once an object is cached at the edge, the origin never sees the next request for it at all. S3 GET charges, ALB LCUs, and resize compute all evaporate for cache hits.

For a photo-sharing app this is not a nice-to-have. Static image bytes are the entire product. Nimbus fronted S3 and the app with CloudFront, and the origin request volume dropped by roughly 90% within a day as the edge warmed up.

A cache that doesn’t hit is just latency with extra steps

Except at first, it barely helped. Jenny pulled the CloudFront CacheHitRate metric and it was sitting at 41%. More than half of requests were still going to origin. A CDN with a bad hit ratio is decoration.

Two culprits, both classic:

Bad TTLs

The photos were being served with Cache-Control: max-age=0, inherited from a default nobody had touched. Immutable content — a thumbnail keyed to a specific photo version — should be cached for a long time. Priya set Cache-Control: public, max-age=31536000, immutable on versioned image URLs (the version is in the object key, so a new image gets a new URL and cache-busting is automatic). TTL went from zero to a year.

Cache keys that fragment

The other half of the problem was the cache key. CloudFront was including every query string and a pile of headers in the cache key. The trending cat thumbnail was requested as .../cat.jpg?utm_source=twitter, ?utm_source=email, ?ref=push… each a distinct cache entry, each a separate origin fetch, all returning the identical bytes. Marketing’s tracking parameters were shredding the cache.

Priya switched to a cache policy that keyed only on the parts that actually change the response and forwarded the tracking params without caching on them. Overnight the hit ratio climbed from 41% to 96%. Same infrastructure — the cache was just finally hitting.

Caching at the right tier

The deepest lesson was about where the cache lives. When Priya first started, the instinct was to cache inside the application — memoize the resized thumbnail in the Lambda, or in a Redis blob store the app checks before resizing.

That helps, but it still pays for the request to arrive: the user’s packet crosses the ocean, hits CloudFront (miss), hits the ALB, wakes a Lambda, which then checks a cache and returns bytes. You’ve saved the resize, but paid for the round trip, the load balancer, and the invocation.

An edge cache removes the request entirely. When the thumbnail lives in CloudFront’s edge cache, the request for the trending cat terminates in Jakarta. The origin — S3, the ALB, the Lambda, the whole us-east-1 machine — never hears about it. No egress from the region, no GET, no compute, no invocation. Nothing.

The rule of thumb Priya wrote on the whiteboard: cache as close to the user as the content’s freshness allows. Deep application caches save recomputation. Edge caches save the entire journey. For Nimbus’s thumbnails — immutable, versioned, wildly popular — the correct tier was the edge, not a clever memoization buried three layers down in PhotoBlob.

The trending cat is now generated exactly once, cached at 300-plus edge locations, and served to the world without ever waking the origin again. Chad described this to the CFO as “leveraging our global edge fabric.” It was, technically, true. Jenny just calls it remembering the answer.

Takeaways

Part 9 — Everything Is a Kafka, Nothing Is a Kafka

Priya found the cluster the way you find a raccoon in the attic: by following the noise of money disappearing.

It was line item 47 on the Cost Explorer breakdown — “Amazon MSK, us-east-1, *$1,190/month**” — and it belonged to a service called event-fanout-v3. She pulled the CloudWatch metrics. Over the trailing thirty days, the three-broker Managed Streaming for Kafka cluster had processed a grand total of roughly 15,000 messages. About 500 a day. One message every three minutes, give or take, humming along on infrastructure designed to move a million per second.

She walked over to Marcus’s desk. Marcus, Principal Architect, defender of Nimbus Platform 2.0, did not look up. “It’s the notification event bus,” he said. “We might need it at scale.”

“Marcus, it’s moving less traffic than my group chat.”

He gestured at the whiteboard, where a diagram of hexagons connected by arrows suggested a distributed system of real ambition. “Kafka gives us replayability, ordering guarantees, and durable retention. When we hit real scale, we’ll be glad it’s there.” Chad Ellison, passing in his Patagonia vest, nodded approvingly and said Nimbus was “building like a company that spends the GDP of a small island.” Which, Priya noted, was technically the problem.

The tool is not the trophy

Here is the uncomfortable truth about managed services: they are wonderful, and they are priced for the scale they were built to serve, whether or not you operate at that scale.

MSK bills you for broker instances that run 24⁄7 regardless of throughput. Marcus’s three kafka.m5.large brokers cost roughly $0.21/hour each, plus storage, plus data transfer — call it $1,190/month all-in — to deliver 500 messages a day. An SQS queue would have handled the same workload for essentially free: the first million requests per month are on the free tier, and after that it’s $0.40 per million requests. At 15,000 messages a month, Nimbus’s SQS bill would round to zero. Replace the whole thing with an SQS Standard queue feeding a Lambda consumer, and you keep durability, retries, and a dead-letter queue, and you delete $1,190/month.

This pattern repeats across the whole account like a recurring dream:

EMR for a spreadsheet’s worth of data

There was a persistent EMR cluster — three m5.xlarge nodes plus the EMR surcharge, about $600/month — running a nightly job that aggregated signup counts. The input was 2 GB of CSV. Two gigabytes. That is not a big-data problem; that is a laptop. The job was rewritten as a single Athena query (Athena is $5 per TB scanned, so this query cost about $0.01 per run) reading Parquet from S3. EMR cluster terminated. $600/month gone, and the job got faster because it stopped spending four minutes bootstrapping Hadoop.

OpenSearch as a very expensive grep

log-search-prod was a three-node OpenSearch domain, r6g.large.search instances plus EBS, around $900/month, that engineers used to grep application logs maybe twice a week during incidents. CloudWatch Logs Insights queries the same logs at $0.005 per GB scanned with zero standing cost. For a team that searches logs occasionally, the managed search cluster was a Ferrari kept idling in the driveway in case someone needed to check the mail.

Death by a thousand tiny services

The second act of over-engineering wasn’t a single monster — it was a swarm. Nimbus Platform 2.0 had been decomposed, per Marcus’s architecture, into 34 microservices. user-avatar-service. avatar-thumbnail-service. avatar-thumbnail-cache-invalidation-service. Each one deserving, in the diagram, of its own box.

The problem is that a microservice in the cloud is never just the code. Each of those 34 services had:

Fourteen of those 34 services averaged under one request per second. They were consolidated into three services grouped by actual bounded context, sharing a load balancer via path-based routing. Eleven ALBs deleted. The idle Fargate baseline dropped by two-thirds. The lesson: microservices are an organizational tool for letting teams deploy independently. If one team owns fourteen of them, that’s not microservices, that’s a monolith wearing eleven trench coats and paying eleven load-balancer bills.

Kubernetes, for four containers

Then there was EKS. Nimbus ran an EKS cluster to host — Priya counted twice — six containers. The EKS control plane alone is $0.10/hour, $73/month, before you run a single pod, plus the node group, plus the operational tax of an engineer who understands kubectl incidents at 2 a.m.

For six containers with no need for custom schedulers, service meshes, or multi-tenant namespacing, EKS was pure ceremony. Moving those workloads to ECS on Fargate eliminated the $73/month control-plane fee (ECS has no control-plane charge) and, more importantly, removed the person-cost of babysitting Kubernetes. This is not an anti-Kubernetes screed — at 300 containers with complex scheduling, EKS earns its keep and its control-plane fee vanishes into the noise. But “we might run it on Kubernetes someday” is not a reason to pay for Kubernetes today.

The observability tax

Nimbus was also being quietly billed for watching itself too closely.

None of this made the app better. It made the app observed, expensively, in places nobody was looking.

Managed vs. custom, evaluated honestly

The fair conclusion isn’t “managed bad, DIY good.” It cuts both ways.

Nimbus’s home-grown Redis-on-EC2 cluster — self-patched, self-monitored, with a failover script Deepak swore worked — cost about $400/month in instances but easily $2,000/month in the engineering time spent keeping it alive. Moving to ElastiCache Serverless cost more in raw AWS dollars and saved money on total cost of ownership. Managed won.

Meanwhile MSK, EMR, and OpenSearch lost on total cost, because the managed premium bought capabilities the workload never used. The rule is boring and unglamorous: match the tool to the actual scale and the actual operational burden, measured in real numbers, not in the scale you’re manifesting on a whiteboard.

Marcus kept the MSK cluster for exactly one more week, until the SQS-plus-Lambda replacement passed its load test at 200x the current traffic for $0/month. Then he quietly deleted it. On the whiteboard, one hexagon got erased. Chad never noticed.

Takeaways

Part 10 — The Boring Stuff That Saves Millions

Jenny found it on a Tuesday, which is when Jenny found most things, because Tuesday was when she read Cost Explorer for fun.

“There’s an account called sandbox-test-delete-me,” she said, “and it’s spending $2,100 a month.”

Priya looked up. sandbox-test-delete-me had been created eighteen months ago by an intern who had since graduated, gotten a job, and possibly gotten married. Nobody had deleted it. Inside it, running patiently in the dark, was a p3.2xlarge GPU instances someone had spun up to try a machine-learning demo and never turned off. At roughly $3.06/hour, it had quietly burned about $44K across eighteen months to compute absolutely nothing.

This is the thing nobody tells you about cloud cost: the expensive part is rarely the clever architecture. It’s the boring part. It’s the stuff no one owns, no one tags, and no one is watching. This installment is about that stuff — governance, the quiet ancillary services, and the mess that multi-account setups create. None of it is glamorous. All of it saves millions.

The graveyard of** orphaned resources

Priya ran a sweep across every account. The findings read like an archaeological dig:

None of this was anyone’s fault, exactly. That was the point: it was nobody’s job.

You can’t manage what you can’t see

The root cause was that Nimbus had almost no cost governance. Specifically:

No tagging. Cost Explorer could show spend by service, but not by team, environment, or product feature, because resources weren’t tagged. Activate cost-allocation tags (team, env, service, owner) and suddenly you can answer “who spent this?” — which is the question that ends 90% of cost arguments.

No budgets, no anomaly detection. There was no AWS Budget alerting anyone when spend crossed a threshold, and no Cost Anomaly Detection — a free, ML-based service that would have emailed someone the day those GPU instances spun up in the sandbox, instead of eighteen months later. Both are effectively free. Not using them is like having a smoke detector and removing the battery.

Compute Optimizer and Trusted Advisor, ignored. Compute Optimizer had been sitting there for a year recommending downsizes on over-provisioned instances. Trusted Advisor’s cost-optimization checks had flagged the idle load balancers and low-utilization instances. Nobody read either. Free advice, unopened.

No CUR. Nimbus used Cost Explorer casually but had never enabled the Cost and Usage Report — the hour-by-hour, resource-by-resource billing firehose that lets you actually audit spend. Without it, deep analysis is guesswork.

No FinOps ownership. The real bug. Until Priya took it up, cost was everyone’s responsibility, which meant no one’s! And the support plan and AWS Marketplace charges — a $15,000/year Business Support tier and a forgotten $2,000/month third-party monitoring SaaS bought through Marketplace — sat unexamined because they weren’t anyone’s line to defend.

The silent killer: CloudWatch Logs

Then Priya found the single biggest surprise on the whole bill. Not compute. Logs.

CloudWatch Logs charges about $0.50/GB to ingest and $0.03/GB-month to store, forever, if you never set retention. Nimbus’s log groups had no retention policy, so logs from 2023 were still accruing storage charges. Worse, the applications logged at DEBUG in production, emitting full request/response bodies. Total ingestion was roughly 9 TB/month — about $4,500/month just to ingest logs, plus a growing storage tail.

The fix was unglamorous and enormous:

Ingestion dropped from 9 TB to under 2 TB/month. That one change saved more than the entire microservices consolidation from Part 9.

The other ancillary line items

Once you start looking at the boring services, they add up:

The multi-account mess

Finally, the org-level waste — the kind you only see from the management account:

Priya deleted sandbox-test-delete-me herself. It felt, she said later, like closing a very expensive raccoon back out into the night.

Takeaways

Part 11 — The Distributed Monolith and Other Horror Stories

Priya found the architecture diagram taped to the wall of the war room. Someone — the handwriting suggested Marcus — had drawn PhotoBlob in the center, a single friendly rectangle labeled “PhotoBlob (core).” Around it orbited eleven more rectangles, each connected to PhotoBlob by a bidirectional arrow, and each connected to the same database, drawn as a cylinder someone had lovingly shaded.

“So it’s a monolith,” said Priya.

“It’s a services architecture,” said Marcus.

“Every one of these services calls PhotoBlob synchronously, and they all share one Postgres instance.”

“We’re on a journey,” said Marcus.

Deepak, who had not looked up from his laptop, said: “PhotoBlob is fine. PhotoBlob has served us well.” He said this the way people talk about a dog that has bitten several children but is still, technically, a good boy.

Priya opened Cost Explorer. Compute was $180K a month. She had a feeling that most of it was going to waste in ways that had nothing to do with pricing and everything to do with the diagram on the wall.

The instances are mostly asleep

PhotoBlob’s image-processing worker ran on a fleet of c6i.8xlarge instances — 32 vCPUs each. Priya pulled up CloudWatch and found CPU utilization hovering around 4%. Not 40. Four.

The worker was a single-threaded Python process. One process, one core, on a 32-core machine. They were paying for 32 cores and using one.

One process is not a plan

Single-core executables on multi-core instances are one of the purest forms of cloud waste, because the cloud bills you for the whole instance whether you use it or not. On a laptop, an idle core is free. On a c6i.8xlarge, thirty-one idle cores cost real money, twenty-four hours a day.

The fix is not exotic. Run one process per core with a supervisor, or a worker pool, or — better — package the worker as many small containers and let the scheduler pack them onto instances until the cores are full. Bin-packing thirty-two single-threaded containers onto a 32-core box turns a 4% machine into a 90% machine, and 90% is the number you actually want to pay for.

Blocking, busy-waiting, and the polling tax

The worker also had no parallelism in a second, sneakier way: it processed images with blocking synchronous I/O. Fetch an image from S3 (wait), resize it (work), write it back (wait). During both waits, the core did nothing, but it was still checked out and unavailable.

Worse, PhotoBlob’s queue consumer didn’t use long-polling. It ran a busy-wait loop — poll SQS, get nothing, sleep(0.1), poll again — spinning the CPU ten times a second forever. On EC2 you at least pay a flat hourly rate for the spinning. Priya had also found the same pattern in a Lambda that polled a DynamoDB table in a loop until a flag flipped. In serverless, a busy-wait loop is a metered crime: you pay per millisecond of execution, so a function that spends four seconds spinning is four seconds of billed compute doing nothing. Blocking waits should yield; loops should long-poll or use event triggers, not spin.

Fat runtimes, slow starts

Half the “microservices” ran on the JVM inside 1.4 GB container images. The oversized runtime — a heavy JVM footprint for a service that did little more than validate a JSON body — meant every container reserved more memory than its work required, and every cold start dragged, because a fat container has more to pull and a JVM has more to warm up. On Lambda-backed services this showed up as cold-start latency; on ECS it showed up as slow, expensive scale-out during traffic spikes. Right-sizing the runtime to the work — a slim base image, a lighter runtime where the JVM wasn’t earning its keep — shrank both the memory reservation and the time-to-ready.

The diagram on the wall

Now to the wall.

One big server that does everything

Historically, PhotoBlob was one huge server running everything: uploads, resizing, feed generation, notifications, billing. This is a single point of failure — when it fell over, all of Nimbus fell over — and, just as expensive, you cannot scale its components independently. Feed generation needed more CPU during peak hours; notifications needed more memory during campaigns. But because they shared one process, the only lever was to scale the entire machine up, paying for peak feed capacity even at 3 a.m. when nobody was scrolling.

The distributed monolith: all the cost, none of the benefit

Then the team “broke it up,” which brings us to the diagram. What they built is the distributed monolith: services split across servers but still tightly coupled, sharing one database, calling each other in synchronous chains. You pay the full price of distributed systems — network hops, serialization, partial failures, cross-AZ data charges — and receive almost none of the benefits, because nothing can be deployed, scaled, or reasoned about independently. A change to the shared schema still requires coordinating eleven services.

Chatty chains multiply latency, failure, and cross-AZ bills

Because the calls are synchronous chains — the upload service calls the metadata service, which calls PhotoBlob, which calls the tagging service — latency adds up and failure multiplies. If each of five hops is 99.9% available, the chain is only about 99.5% available; each hop is another chance to time out. And on AWS every one of those hops that crosses an Availability Zone incurs cross-AZ data transfer charges at roughly one cent per gigabyte each way. Chatty services gossiping across AZs turn a rounding error into a line item.

But don’t overcorrect

Priya was careful here, because the opposite mistake was also on the table. Marcus’s “Platform 2.0” proposed splitting the tagging service into four services. That is premature, over-granular microservices — the cost of the network and the operational overhead without a real boundary to justify it. And the shared database across services is the coupling that makes the whole thing a monolith regardless of how many rectangles you draw. The rule Priya wrote on the whiteboard: split on real boundaries, give each service its own data, and prefer async messaging over synchronous chains.

Pets, snowflakes, and the local disk

Autoscaling was the next fight, and it was losing to biology.

Pets, not cattle

PhotoBlob’s servers were stateful. Each one held in-memory session state and cached user data that existed nowhere else. You could not just terminate one — it was a pet, with a name and feelings. Stateful application servers block autoscaling (you can’t scale in without losing state) and block Spot (you can’t tolerate a two-minute interruption notice). Nimbus was paying full On-Demand for a fleet it couldn’t safely shrink.

Two culprits made the servers stateful. First, session affinity / sticky sessions: the load balancer pinned each user to one server, so that server couldn’t be replaced without logging people out. Second, local filesystem dependence: uploads were written to the instance’s local disk before processing. On ephemeral compute, local disk vanishes when the instance does, so this both created state and lost data on every scale-in. Sessions belong in a shared store (Redis, DynamoDB); files belong in S3. Do that and the servers become cattle — interchangeable, disposable, Spot-eligible.

Snowflakes and the vertical-only ceiling

Every server was also a snowflake — hand-configured over years, no two quite alike, impossible to reproduce. The fix is Infrastructure as Code plus immutable images: bake an AMI or container, deploy it identically, replace rather than patch. And because the servers were pets, the team’s only scaling move had been vertical — buy a bigger box. Vertical scaling has a hard ceiling (there is a biggest instance) and it’s a SPOF by construction. Horizontal scaling — more small cattle behind a load balancer — is what actually lets you follow demand and turn machines off when it’s gone.

The database is doing chores it shouldn’t

Finally, the shared Postgres, which was on fire.

Pagination, scans, and the N+1

The feed API used application-layer pagination: it fetched every photo for a user into application memory, then sliced out page 3. For a power user with 40,000 photos, that’s 40,000 rows pulled across the network to return 20. The fix is to push the limit into the query — LIMIT/keyset (seek) pagination and projections that select only the needed columns — so the database returns the page, not the library. The same rule holds in DynamoDB: use a paginated Query with LastEvaluatedKey, not a Scan of the whole table.

Which was the next problem: full table scans and missing indexes. The “search by caption” feature had no index, so every search scanned the whole photos table. On a shared database, one unindexed query starves every other service on the box.

And the feed had a classic N+1 query pattern: one query to list 50 photos, then 50 more queries to fetch each photo’s author. Fifty-one round trips where a join or a batched IN query would do one. Every extra round trip is latency and, across AZs, transfer cost.

One database, no cache

There was no caching layer anywhere — every feed render hit Postgres cold, even though feeds barely change between refreshes. A read-through cache (ElastiCache) in front of the hot paths would have taken enormous load off the database. And underneath it all, the deepest anti-pattern: one database for every workload. Relational feed data, full-text search, session state, and a time-series of view counts were all crammed into one Postgres instance, so the instance was sized for the sum of every access pattern and tuned for none. Purpose-built stores — Postgres for relational, OpenSearch for search, DynamoDB for key-value sessions, a cache for hot reads — each right-sized, each independently scalable.

Priya took the diagram off the wall.

“We’re keeping PhotoBlob,” she told Deepak. “We’re just going to stop asking it to be the entire company.”

Deepak nodded slowly, like a man being told his dog would be allowed to stay, but must, from now on, stop biting the children.

Takeaways

Part 12 — It Compiles, Ship It

At 2:14 a.m., the entire Nimbus feed went dark because the thumbnail service was slow.

Not down. Slow. The thumbnail service took eight seconds to respond instead of the usual two hundred milliseconds, and because the upload path called it synchronously with no timeout, every upload request held its worker thread open for eight seconds, and the worker pool filled, and then the health check requests couldn’t get a thread either, and then the load balancer decided the whole fleet was unhealthy, and then 40 million people saw a spinner.

Priya got the page. She traced it in eleven minutes and then sat in the dark for a bit, because the thing that took down Nimbus was that one non-critical dependency — thumbnails — got a little slow, and nothing in the system was built to survive a dependency being a little slow.

Chad’s Slack message arrived at 2:40 a.m.: “why are we down, I told the board we spend the GDP of a small island on infra.” Priya typed, deleted, and retyped a reply four times before settling on: “Working on it.”

Things that fall over when one thing gets slow

No timeouts, no retries, no circuit breakers

The root cause was that the upload service called thumbnails with no timeout. A call with no timeout waits forever; forever is longer than your thread pool. Every outbound call needs a timeout shorter than the caller’s own budget.

Then there was the opposite failure. The notification service did retry — naively, immediately, three times, with no backoff. When the downstream service got slow, every caller retried, tripling the load on the thing that was already struggling. That’s a retry storm: retries kick a system while it’s down and keep it down. The cure is timeouts plus retries with exponential backoff and jitter, wrapped in a circuit breaker that trips open after repeated failures and stops calling the sick dependency entirely until it recovers.

Blocking the user on work the user doesn’t care about

The deeper sin was synchronous coupling to non-critical work. The upload request blocked on thumbnail generation. It also blocked on sending a confirmation email and on writing an analytics event. None of these are things the user is waiting for — they tapped “post,” they want to see “posted.” Emails, analytics, and thumbnails should be offloaded to a queue (SQS, EventBridge) and processed asynchronously. The user request returns in milliseconds; the slow, non-critical work happens later, on its own schedule, and its slowness can no longer take down the critical path.

One of everything

Priya drew the failure domains and found single points of failure stacked like dominoes: the primary database ran in one AZ with no replica; all egress went through one NAT gateway (the NAT gateway from hell returns); several services ran as a single instance with no second copy. Any one of them dying meant an outage. Multi-AZ databases, redundant NAT gateways per AZ, and at least two instances behind every load balancer are not gold-plating; they are the difference between a blip and an incident.

Dead nodes and dropped deploys

Two more gaps made every incident worse. There were no health checks deep enough to matter, so the load balancer kept routing traffic to nodes that were technically running but internally broken. And there was no graceful shutdown — on every deploy, instances were killed mid-request, so every routine deployment dropped a fistful of live requests. Real health checks pull dead nodes out of rotation; connection draining and graceful shutdown let a terminating node finish its in-flight work before it goes.

Idempotency, so retries are safe

Underlying all of it: the payment webhook was not idempotent. When a retry did fire, it could charge a card twice. Retries are only safe if the operation is idempotent — an idempotency key so that the same request processed twice has the same effect as processing it once. Without it, you must choose between resilience and correctness, and that is not a choice anyone should have to make at 2 a.m. **

Fighting the cloud instead of using it

The next morning, calmer, Priya went looking for why the bill was $340K, and found a theme: Nimbus kept rebuilding AWS inside AWS.

Reinventing managed services

The queue that should have decoupled uploads from thumbnails didn’t exist because someone had self-hosted RabbitMQ on a pair of EC2 instances — which they patched, monitored, and occasionally lost sleep over. Next to it: a self-managed Redis cluster and a self-hosted Postgres on raw EC2, backups scripted by hand. Every one of these reinvents a managed service. SQS, ElastiCache, and RDS exist, are cheaper once you count the human time, and don’t page anyone at 2 a.m. for a failed backup cron. Running your own message broker on EC2 is paying for the instance and paying an engineer to be the on-call for it.

Hardcoded secrets and hardcoded assumptions

Grepping the repo, Priya found database passwords committed in a config file — hardcoded configuration and secrets. These belong in SSM Parameter Store (for config) and Secrets Manager (for credentials), fetched at runtime, rotated without a redeploy, and kept out of git history forever.

She also found code that assumed instances live forever: cron jobs pinned to a specific hostname, an in-memory cache expected to survive for weeks, a “warmup” that ran once at boot and never again. This is ignoring ephemerality — on the cloud, instances come and go, and anything that assumes a long-lived host breaks the moment autoscaling or a Spot reclaim does its job.

A fleet sized for a peak that happens twice a day

The compute fleet was fixed: 80 instances, running 24⁄7, sized for the Friday-evening peak. Traffic at 4 a.m. was a tenth of that. This is hardcoded capacity with no elasticity — paying for peak around the clock. Autoscaling on a real metric (queue depth, CPU, request rate) would run 80 at peak and 12 overnight, and the difference is enormous when it’s compounded across 720 hours a month.

Not knowing where the bytes go

Finally, region and AZ blindness. Services were scattered across AZs with no thought to which called which, so requests hopped AZs constantly, each hop billing cross-AZ transfer and adding a millisecond of latency. One analytics job in us-east-1 pulled its source data from a bucket in eu-west-1 on every run — cross-region transfer, the most expensive kind, hidden in a nightly job nobody watched. Data has gravity and a price tag; keeping chatty components in the same AZ and data in the same region is free money.

Flying the plane with the instruments taped over

No pipeline, no IaC, no idea

Deployments were manual — someone SSH’d in and pulled the latest build, service by service. There was no CI/CD, so deploys were slow, inconsistent, and terrifying, and there was no Infrastructure as Code — the whole environment had been clicked together in the console over three years and had drifted so far that nobody could recreate it. Releases were big-bang and coupled: all eleven services shipped together because they were entangled, so any one bug rolled back all of them.

The fix is the boring, load-bearing stuff: a CI/CD pipeline that builds, tests, and deploys automatically; IaC (Terraform, CloudFormation, CDK) as the single source of truth so the console matches the code; and decoupled, independently deployable releases so a thumbnail fix doesn’t require redeploying billing.

Flying blind

The reason the 2 a.m. incident took eleven minutes to diagnose and could easily have taken two hours was no observability. No distributed tracing, no useful metrics, no dashboards. You cannot right-size what you cannot measure, and you cannot diagnose what you cannot see. Metrics, traces, and structured logs are how you find the idle instance and the slow dependency.

But observability has its own trap, which Priya found in the bill. CloudWatch Logs ingestion was a top-five line item. Every service logged every request at DEBUG, in unstructured free text, and it all shipped to CloudWatch at ingestion cost per gigabyte. Over-logging turns your telemetry into a cost center. Log at sensible levels, emit structured (JSON) logs you can actually query, sample the high-volume paths, and set retention so logs don’t live forever at full price.

The debt with interest

Two last categories, cheap to fix and expensive to ignore.

Security debt

IAM was a disaster of convenience: services shared over-broad roles (“just give it AdministratorAccess, we’ll tighten it later” — later never came) and authenticated with long-lived access keys hardcoded into containers. Least privilege — scope each role to exactly what it needs — and IAM roles over static keys (the container assumes a role and gets short-lived, auto-rotated credentials) close the biggest blast radius on the account.

The network was flat and public. Databases sat in public subnets with security groups open to 0.0.0.0/0, because it was easier during a hackathon in 2024. Data stores belong in private subnets with tight security groups, and traffic to AWS services should go through VPC endpoints — which also, pleasingly, keeps that traffic off the NAT gateway from hell and off the transfer bill.

Image hygiene

The container images were huge — the 1.4 GB JVM images from Part 11 — which slowed every pull, every scale-out, every cold start. And ECR had no lifecycle policy, so every image build for three years was still sitting in the registry, accruing storage charges for artifacts no deployment would ever use again. Slim base images and an ECR lifecycle policy that expires old, untagged images are ten minutes of work that pay rent every month.

Priya closed her laptop. The list was long, but none of it was mysterious. It was just debt — taken on one reasonable-at-the-time shortcut at a time, and now, at $340K a month, coming due all at once.

“It compiles,” Deepak had said, three years ago. “Ship it.”

It had, in fact, shipped.

Takeaways

Part 13 — The Real Reason - Humans not Servers

The lights come up on the main stage at CloudScale Summit, and there is Chad Ellison, CTO of Nimbus, in the vest. The Patagonia vest. He clicks to a slide that just says $340K/MONTH in a font large enough to read from orbit.

“People ask me how we know we’ve made it,” Chad says, pausing for the room. “And I tell them — we spend the GDP of a small island on AWS. Every single month.” Warm laughter. A ripple of applause. Someone in row three actually whoops.

In the fourth row, Priya Nair — Staff Engineer, freshly and involuntarily crowned “FinOps lead” — sits very still, the way you sit when you have just watched someone describe a fever as a personality trait. She has spent six weeks proving that at least a third of that number is pure waste: the NAT gateway from hell, PhotoBlob’s idle midnight fleet, Big Bertha the over-provisioned analytics cluster running at 4% utilization. And here is her CTO, on a stage, celebrating the symptom as if it were the diagnosis.

She writes one line in her notebook: The bill isn’t the problem. The applause is.

This is the last post in the series, and it’s the one no tool can fix. Every previous installment was about instances and gateways and lifecycle policies. This one is about people, incentives, and the strange gravity that keeps waste alive long after everyone can see it. Because here’s the uncomfortable truth: your architecture didn’t get expensive by accident. It got expensive because, for a lot of people, expensive was working exactly as intended.

The Ego Tax: When the Design Is Someone’s Identity

Marcus Webb designed Nimbus Platform 2.0. He will tell you this within ninety seconds of meeting him. He will also tell you, when Priya suggests collapsing three of its services into one, that “we might need it at scale” — a phrase that has never once been followed by evidence.

Sunk cost wearing a lanyard

The problem isn’t that Marcus is wrong. The problem is that Marcus can’t afford to be right about this. His comp band, his principal title, and roughly two years of his professional identity are welded to that design. Asking him to admit it’s over-built is asking him to file for a demotion. “We invested two years in this” is not an engineering argument; it’s grief. And sunk cost is the most expensive emotion in the building — those two years are gone whether you keep the design or not.

How to address it: separate the person from the artifact. Praise the ambition, retire the parts that didn’t pay off, and make “we simplified Marcus’s platform” a win Marcus gets credit for — not a verdict against him. Nobody rewrites their own monument. They’ll happily co-author the renovation.

Not-invented-here and the résumé that ate the roadmap

There are two related diseases here. The first is not-invented-here: “we built it, we keep it,” regardless of whether a managed service now does the same thing for a tenth of the cost and none of the on-call. The second is its louder cousin, résumé-driven development — the reason Nimbus runs Kubernetes to schedule what is functionally a nightly cron job, Kafka to move messages a queue would carry, and microservices to serve forty million users who would never notice a monolith.

None of these were chosen because the problem demanded them. They were chosen because they look magnificent on a CV, and because in 2023 you couldn’t get promoted talking about a well-tuned Postgres.

How to address it: require an ADR — an Architecture Decision Record — for anything load-bearing, and make one section mandatory: “Why not the simpler option?” If the honest answer is “because it’s boring,” that’s your signal. You’ve just caught a career decision masquerading as a technical one.

The Scoreboard Is Wrong

Which brings us back to Chad on that stage. Chad isn’t a villain. Chad is responding, rationally, to the scoreboard he’s been handed — and the scoreboard says a big AWS bill is a trophy.

Vanity metrics and the gross-spend brag

Gross spend is a vanity metric. It tells you nothing about health. A $340K bill serving 40M users at healthy margins is a triumph; the same bill serving them at 4% cluster utilization is a slow-motion fire. But “gross spend” is the number that makes it into the keynote, so gross spend is the number that gets optimized upward.

How to address it: change what’s on the scoreboard. Stop celebrating total spend; start celebrating unit economics — cost per active user, per upload, per GB served, per transaction. When Priya finally got Chad a slide that read “cost per active user: $0.0085, down 22% this quarter,” his brag changed overnight. Same ego, better target. He now brags about efficiency at scale, which, mercifully, is a thing you can be proud of without lighting money on fire.

Empire-building, use-it-or-lose-it, and the orphaned cost

Two more incentive traps live here. Empire-building — where status is measured in headcount and budget, so no leader ever has a reason to shrink either. And use-it-or-lose-it budgeting, where a team burns its remaining cloud allocation in Q4 specifically so it isn’t cut next year. Both reward consumption for its own sake.

And underneath all of it: at Nimbus, cost was nobody’s job. Everyone assumed someone else was watching the meter. Nobody was. That’s why Priya exists now.

How to address it: give cost an owner — a named FinOps function, even if it’s one person part-time. Reward returning budget, not spending it. And measure leaders on outcomes-per-dollar, not dollars commanded.

The Culture That Protects Waste

Deepak and the fortress of “it works”

Deepak maintains PhotoBlob, the original monolith, and his position is theological: “it works, leave it alone.” He’s not lazy — he’s scared and busy, which look identical from the outside. Refactoring PhotoBlob has no allocated time, no reward, and considerable risk. So of course it never happens.

How to address it: make improvement part of the actual job, not a heroic side quest. Allocate explicit time for it. Reward the refactor that halves PhotoBlob’s idle footprint the same way you’d reward a shipped feature. What gets celebrated gets done.

Fear-driven “don’t touch it”

Deepak’s fear is rational, though. The reason nobody touches Big Bertha or the NAT gateway from hell is that there’s no safety net — no tests, no canary, no clean rollback. In that world, “don’t touch it” is the correct strategy.

How to address it: build the net first. Tests, canary deploys, one-click rollback. Fear-driven caution isn’t a character flaw; it’s a missing seatbelt. Install the seatbelt and watch how much braver everyone gets.

Hero culture, blame culture, and alert fatigue

Nimbus has a hero culture: the engineer who stays up until 3 a.m. firefighting a cost spike gets the plaque. The engineer who quietly prevented the spike gets nothing, because you can’t throw a party for an incident that didn’t happen. This trains people to value the fire, not the fire-proofing.

It’s compounded by blame culture — when the post-incident question is “whose fault?”, people hide problems instead of surfacing them. And by alert fatigue: Jenny, the FinOps-curious junior, has been flagging anomalies in Cost Explorer for months. She was right about Big Bertha in March. Nobody listened, because the dashboards cry wolf so often that a real wolf reads as noise, and because Jenny is junior and the org has quietly normalized the waste as just “what the bill is.”

How to address it: reward prevention as loudly as you reward heroics. Run blameless postmortems so problems come out into the light. Tune alerts so the ones that fire actually mean something — and when your Jenny is right three times in a row, promote her, don’t ignore her.

The Governance Vacuum

Nobody can see their own bill

At Nimbus, no team could see what they cost. The bill arrived as one undifferentiated $340K boulder, owned by everyone and therefore no one. You cannot ask people to manage a number they never see.

How to address it: tag everything, then implement showback (here’s what your team cost) or chargeback (and here’s the invoice). Visibility alone changes behavior — teams optimize what they can see attributed to their own name.

Decisions made once, never revisited

The always-on schedule for PhotoBlob’s fleet was set in 2023, for a traffic pattern that no longer exists. Nobody chose to keep it running at 3 a.m.; they just never chose not to. Decisions calcify into facts.

How to address it: put a review cadence on the big ones. Quarterly, ask of every major cost: “if we were deciding today, would we decide this?”

Analysis paralysis and the approval swamp

The flip side of never-revisiting is never-deciding. Priya’s first proposal to right-size Big Bertha sat in an approvals queue for five weeks — three sign-offs for a change that saved $9K/month and could be undone in ten minutes.

How to address it: delegate with guardrails. For reversible decisions, default to yes and let engineers act. Save the ceremony for the one-way doors. An approval process that costs more than the thing it’s approving is itself waste.

Optimization as a project, not a practice

The deepest governance failure: Nimbus treated cost work as a project — a heroic quarter, a spreadsheet, a victory lap, then back to normal until the CFO panics again. Waste doesn’t work in sprints. It accrues continuously, so it has to be fought continuously.

How to address it: make it a standing practice. Continuous FinOps — a small recurring rhythm, not an annual crisis.

Skills, Silos, and the Translation Gap

Half of Nimbus’s “cloud architecture” is really a data-center mindset transplanted to AWS: lift-and-shift habits, always-on servers, capacity provisioned for a peak that arrives twice a year. That’s a skills gap, and it’s fixable with training, not blame — nobody unlearns fifteen years of on-prem instinct by osmosis.

Then there’s key-person risk: only Deepak understands PhotoBlob, only Marcus understands Platform 2.0’s wiring. Knowledge silos make every system too scary to change and every vacation a liability. The fix is unglamorous — docs, pairing, and deliberate rotation so no single brain is load-bearing.

And there’s the translation gap: engineers talk in vCPUs and IOPS; the CFO talks in margin. They are describing the same reality and cannot hear each other. Unit economics is the shared language — “this change improves gross margin by 1.5 points” lands in a boardroom in a way “we downsized the r5 fleet” never will.

Finally, vendor over-reliance: Nimbus pays a consultancy handsomely, and the consultancy is paid by the hour — which quietly rewards complexity, not thrift. Align those contracts to outcomes (a share of realized savings) and watch the recommendations get simpler.

Structure and Politics

Conway’s Law is having a field day here. Nimbus’s system mirrors its org chart, and the org chart is a mess, so there are three services doing image resizing because three teams each built their own rather than share. Turf wars over shared resources mean nobody will adopt a common component they didn’t own.

Above it all sits short-termism — next-quarter thinking that never funds the boring efficiency work — and reorg churn, which reassigns ownership so often that no one holds a system long enough to care about its bill.

How to address it: fund a standing efficiency-and-tech-debt allocation so the long game has a budget. Keep ownership stable long enough for pride to form. And occasionally, reshape a team to fix a redundant service — you can run Conway’s Law forwards instead of letting it run you.

Takeaways

The closing insight: almost every problem in this post reduces to just two root causes — misaligned incentives and invisible cost. Every villain in the series is really one of those two wearing a costume. So the two highest-leverage fixes aren’t technical at all. First, make cost visible and owned: showback, unit-economics dashboards, a named FinOps owner, anomaly alerts that people trust. Second, change what gets rewarded: efficiency, prevention, and simplicity — measure cost-per-unit instead of gross spend, and let teams keep a share of the savings they verifiably create. Get those two right and the individual behaviors start correcting themselves — not because anyone became a better person, but because, at long last, doing the right thing finally pays. Chad will still wear the vest. He’ll just brag about a different number.

Epilogue — Ninety Days Later

The second “AWS Spend” meeting was also on a Tuesday, because the universe enjoys symmetry.

This time Priya presented. The bill had gone from $340K/Month to $196K/Month. That is a 42% cut — and, crucially, Nimbus was serving more traffic than it had in the quarter before. The line that made the CFO smile wasn’t the gross number, though. It was the one Priya had learned to care about - cost per monthly active user had dropped from $0.0085 to $0.0049. They were serving each person for a little over half of what it used to cost.

Nobody had turned off anything customers noticed. There were no outages. The checkout path still had its multi-AZ redundancy, the critical database still had its standby, and the on-call rotation actually slept better than before, because half of the fragile, hand-tuned, snowflake infrastructure had been replaced with things that healed themselves.

Here’s what actually happened, and in roughly what order — because the order matters more than people expect.

The receipts

The quick, embarrassing wins came first. Big Bertha — the m5.24xlarge running a 3%-CPU cron — became a scheduled t4g.medium and saved more than most people’s salaries. Priya deleted 40 TB of orphaned EBS snapshots, released a dozen unattached Elastic IPs that had been billing hourly for a year, and flipped every gp2 volume to gp3 for the same performance at ~20% less. None of it required a meeting. All of it should have happened two years ago. This is the recurring lesson of cost work: the first 20% is almost free and slightly humiliating.

Then the structural stuff. The NAT gateway from hell got Gateway VPC Endpoints for S3 and DynamoDB — which are free — and its data-processing charge fell off a cliff. Right-sizing compute off Compute Optimizer’s recommendations, plus moving the fleet to Graviton, took a big bite. S3 lifecycle policies started aging cold photos down to Glacier instead of paying Standard rates to store a selfie from 2019 that no one will ever open again.

Then the commitments — but only after right-sizing. Priya covered the now-smaller steady baseline with Compute Savings Plans, deliberately not the peak, and let Spot and on-demand absorb the spikes. Buying commitments before right-sizing would have locked in the waste; she’d have been proud of a discount on things she was about to delete.

Then the slow, valuable refactors. PhotoBlob is still mostly a monolith — that’s fine, a well-run monolith is cheaper than a badly-run mesh of microservices — but its worst full-table-scan queries got indexes, its session state moved to Redis so the app tier could finally autoscale, and Marcus’s 3-broker MSK cluster serving 500 messages a day was quietly replaced with an SQS queue that costs approximately nothing. These didn’t land in week one. They landed over the quarter, on a standing “efficiency and debt” allocation that got its own slice of every sprint.

The two things that made it stick

If you take nothing else from the entire series, take these, because they’re the reason the savings didn’t evaporate the moment Priya looked away.

One: make cost visible and owned. Every team at Nimbus now sees its own spend, thanks to a tagging policy Priya enforced with the enthusiasm of a customs officer. There’s a dashboard showing cost-per-MAU, not just gross dollars. There are budgets and Cost Anomaly Detection alerts with a named human attached to each. Cost stopped being “finance’s problem” or “the platform team’s problem” and became everyone’s, visibly. You cannot fix what nobody can see and nobody owns.

Two: change what gets rewarded. This was the hard one, and it was Chad’s to solve, not Priya’s. The scoreboard changed. Nimbus stopped celebrating the size of the AWS bill and started celebrating unit economics. Teams that shipped efficiency wins got the same visible credit as teams that shipped features. And — the masterstroke — teams got to keep and redeploy a share of the savings they verified, which turned “cut the bill” from a chore imposed from above into a game people actually wanted to win. Chad, to his credit, updated his conference talk. The new version is about margin. It got fewer laughs and more job offers.

The prioritization framework, on one napkin

For anyone starting their own ninety days, here’s the order Priya wishes she’d known on day one:

  1. Kill waste. Orphaned resources, idle instances, unattached everything. Turn on Compute Optimizer, Trusted Advisor, budgets, and anomaly alerts. Free, fast, no risk.
  2. Grab the quick wins. gp2→gp3, NAT→Gateway Endpoints, non-prod scheduling, Graviton migration.
  3. Right-size compute and databases from real utilization data.
  4. Commit the stable baseline — never the peak — to Savings Plans, and only after right-sizing.
  5. Go structural. Networking topology, redundancy you don’t need, caching, and over-engineering. Durable savings, a bit more effort.
  6. Refactor the debt. Statelessness first (it unlocks autoscaling and Spot), then concurrency, then data-access patterns, then decoupling — scored by blast radius × recurring cost × effort, burned down over quarters.
  7. Fix the humans. Visibility and incentives. Without this, you’ll do steps 1–6 again next year.

Score each candidate by how much it costs you every month times how easy it is to fix, and start at the top of that list. Boring, repeatable, and it works.

Last word

Nimbus’s bill will creep back up — that’s what bills do, because businesses grow and entropy is undefeated. The difference is that now someone is watching, everyone can see, and the person who finds the next Big Bertha will get a high-five instead of a shrug.

Priya still doesn’t love being the FinOps lead. But she’s made her peace with it, mostly because of a sentence she now repeats in every architecture review, usually while looking directly at Marcus: “We can absolutely build it that way. Let’s just be honest about what it costs — and whether anyone will ever need it at scale.”

He hasn’t got a comeback for that one yet.

— End of series. Go read your bill. Bertha’s counting on you.