CloudWatch Log Ingestion Quadrupled in Two Months

The number

In August 2024 the CloudWatch line on the AWS bill of a US fintech I worked for was about $2.4k. In October it was about $6.4k, and it stayed around $6.2k a month after that.

One API operation explained most of the jump. PutLogEvents went from about $1.3k to about $5.3k a month. That is log ingestion: the price of sending log lines into CloudWatch Logs. It grew about four times in two months, and nobody had decided it should.

I found it during a cost review in early 2025, where I went through six months of Cost Explorer, August 2024 to February 2025, one service at a time. CloudWatch was the line that jumped most, but it was not the only thing on that bill nobody had chosen on purpose.

The service view hides it

If you look at Cost Explorer grouped by service, you see "CloudWatch went up". That is not enough to act on. CloudWatch bills for metrics, alarms, dashboards, Logs Insights queries, log storage and log ingestion, and each has a different fix.

What made it obvious was grouping by API operation. Filter the service to CloudWatch, group by API operation, and set the range to six months. One bar grows and the rest stay flat. In this case the bar was PutLogEvents.

From there the next step is to find which log groups are sending the bytes. CloudWatch Logs publishes an IncomingBytes metric per log group. Sort the log groups by incoming bytes over the last month, and compare with the month before the jump. Usually two or three log groups hold most of the growth.

The usual causes

When I see this pattern, these are the first things I check:

• A log level set to debug during an incident and never set back.
• A new service, sidecar or agent that logs every request, including health checks.
• Lambda functions that print whole payloads.
• Retries in a loop, where each failed attempt writes a stack trace.
• Access logs sent to CloudWatch when they only needed to land in S3.

None of these is exotic. They are normal engineering choices that nobody revisits because the cost lands on a shared line weeks later, and the person who changed the log level is not the person who reads the bill.

What else was on the same bill

The same review turned up a set of smaller items. None of them was as large as the logging jump, but together they added up, and each was the kind of cost that stays because nobody owns it.

gp2 volumes still running. EBS gp2 volumes cost about $760 a month next to about $520 for gp3. gp3 is about 20% cheaper per GB and lets you set IOPS separately from size. Moving gp2 to gp3 is one of the few changes that is cheaper and usually no worse.

AWS Config in non-production. The AWS Config line went from about $700 to about $1.3k a month. Most of that growth came from staging, which went from about $20 to about $440 (about 19 times), and other non-production accounts, from about $40 to about $270. Config charges per configuration item recorded, so an environment where resources churn a lot, or where a new controller keeps updating resources, can produce a large number of items. Recording everything in staging is often more than the compliance scope asks for.

An always-on SFTP endpoint. AWS Transfer Family cost about $1.2k to $1.4k a month. Most of that is the endpoint being on, every hour, whether files arrive or not.

NAT gateways. NAT gateway data cost about $1.2k a month, plus NAT hours in two regions with very little traffic. An idle NAT gateway still charges for every hour.

Business Support. The support plan went from about $2.5k to about $3.0k a month. Business Support is priced as a percentage of the bill, so every dollar of log ingestion also adds a little to the support line.

The options for log ingestion

The fixes for log ingestion are well known, and they are about deciding what each log is for:

• Set log levels per environment, and make debug something that expires.
• Stop logging health checks and other high-volume, low-value requests at the source.
• Send logs that are only kept for audit to S3 instead of CloudWatch Logs.
• Use the Infrequent Access log class for log groups you rarely query. It charges about half per GB ingested, with fewer features.
• Set a retention period on every log group. "Never expire" is the default, and it is rarely what anyone wants.

Retention fixes storage, not ingestion. If PutLogEvents is the line that grew, the fix is fewer bytes sent, not shorter storage.

A monthly check

Once a month, I look at these in Cost Explorer:

1. Top five services by month-over-month change, not by size.
2. For each one that grew, group by API operation or usage type.
3. For CloudWatch, sort log groups by IncomingBytes and compare with last month.
4. gp2 volumes still running.
5. AWS Config cost per account, especially non-production.
6. Endpoints that charge by the hour: NAT gateways, Transfer Family, load balancers with no traffic.

The point is not to find every dollar. It is to catch a 4x jump in the month it happens, not six months later.

Put a unit next to the total

A total bill only tells you it went up. Later I built a Grafana billing dashboard at the same company that divided month-to-date AWS spend by the number of transactions. In October 2025 it showed about 1.03 million transactions for the month so far and about 5 cents of AWS spend per transaction. That one number let the VPs see where they stood: if the bill grows with the business, the unit cost stays flat, and if it grows on its own, the unit cost moves. A 4x jump in log ingestion shows up there too.

If your AWS bill has a line that grew and nobody can say why, I can help with that: Cloud Cost & FinOps.