Managing Prometheus at 900 GB Without Disrupting Trading Services

When Metrics Become an Operational Burden

At more than 900 GB of time series data, Prometheus had become an operational burden. Queries timed out unless scoped to an hour, while labels added for debugging kept driving cardinality higher. Those symptoms point to a monitoring system whose usefulness is being squeezed by its own growth: engineers need historical metrics to investigate problems, but the cost of asking broader questions is rising.

Prometheus stores measurements as time series: sequences of values associated with metric names and labels. A label might identify a service, host, region, or other property. This structure makes it possible to filter and compare metrics in useful ways. It also creates a trade-off. Every distinct combination of metric and label values can represent another series for the system to track. As those combinations multiply, storage and query work can grow even when the number of metric types appears stable.

That distinction matters in practice. A debugging label can look harmless when considered on its own, yet vary across many events or objects. If each variation creates a distinct series, the added detail can raise cardinality substantially. The same detail may be valuable during an investigation and much less useful once that incident has passed. Keeping it indefinitely can turn short-term diagnostic convenience into a continuing cost.

The Query Problem Is Also a Data Problem

A query that has to be limited to an hour is more than an inconvenience. It can leave an engineer without the time window needed to understand whether a fault began suddenly, followed a gradual change, or recurred after an earlier event. Narrow queries can also force people to run several searches and piece together the results, which adds friction precisely when a clear view of system behavior matters most.

Queries over time series data generally need to find the relevant series, read samples across a selected interval, and perform the requested calculation. A broader time range can mean more samples to process; high cardinality can mean more series to consider. The exact cost depends on the query and the system’s data layout, so the reported timeouts alone don’t identify one cause. They do show that the current combination of query scope and stored data had become difficult to operate.

This is a common tension in monitoring. More history can help with comparisons and retrospective investigations, but retaining more data increases the work involved in storage and retrieval. More labels can make a metric easier to isolate, but can also make the underlying dataset harder to manage. There isn’t a single setting that resolves both pressures. Teams have to decide which measurements and dimensions are worth preserving, for how long, and at what level of detail.

Cost and Runtime Risk Are Linked

A partial copy of metrics also went to an external SaaS vendor, and its bill climbed each quarter. That creates a second pressure alongside the difficulty of querying the local Prometheus system: the organization was paying to send some monitoring data elsewhere while the main store itself was becoming harder to use. The source doesn’t specify the vendor’s pricing model or the reasons for each increase, so the bill’s trajectory can’t be attributed to any particular charge. Still, rising expense makes the choice of what to collect and transmit harder to treat as an afterthought.

Sending only part of a metrics stream can reduce the amount of data leaving a system, but the value of that arrangement depends on what’s included and how the external service is used. If the copy supports dashboards, alerts, or investigations that the local stack can’t handle as effectively, reducing it could create operational gaps. If it duplicates information with little practical use, continuing to send it may preserve cost without providing much benefit. The available details don’t settle that question.

The monitoring stack ran alongside latency-sensitive trading services, leaving little tolerance for extra CPU use. That constraint changes the shape of the problem. A query or data-processing task may be acceptable on a general-purpose monitoring host and still be undesirable when it competes for resources near services that respond to time-sensitive workloads. The concern isn’t merely whether a new design can process the data; it’s whether doing so adds load at a moment when the trading services need predictable performance.

Here’s the thing: monitoring is meant to help operators see trouble, but the monitoring machinery can itself become a source of operational risk. That risk tends to appear in ordinary work: a wide query, a burst of ingestion, or background processing that needs more CPU than expected. (And this is the part most people overlook.) The system’s location and workload neighbors can matter as much as its storage format.

Why Changes Need to Be Evaluated Together

Any change to the system had to account for query performance, data growth, cost and the risk of adding load at the wrong moment. These are connected constraints. Reducing retention may ease storage pressure but leave less history for investigations. Removing labels may lower cardinality but make some debugging questions harder to answer. Shifting work to another system could alter local resource use while adding transfer or service costs.

That doesn’t mean every label should be removed or every old sample discarded. It means the value of each kind of data needs to be considered against its continuing footprint. Teams can ask whether a label helps answer a recurring operational question, whether that answer requires fine-grained detail over long periods, and whether a less costly summary would serve the same purpose. Such questions are general design choices; the source doesn’t say which changes were made or what their results were.

Testing also has to reflect the workload that caused the problem. A system can appear responsive with short queries and modest data, then struggle when asked to scan a longer interval across many series. Likewise, a change that reduces one kind of processing may shift work elsewhere. In this case, any evaluation would need to consider both query behavior and resource use near the trading services, since an improvement in one measure could still carry a cost in another.

Implications for the Monitoring Strategy

This is more significant than it looks. The issue isn’t simply that a large database needs more space; the monitoring setup was under pressure from data volume, query limits, external spending, and a strict CPU constraint at once. Treating each symptom as a separate tuning task could miss how a fix in one area affects the others.

If you’re working in this space, the practical lesson is to treat labels, retention, query patterns, and data export as parts of one operating model. A debugging dimension can be useful, but its value should be weighed against the number of distinct series it creates and how long that detail needs to remain available. External copies deserve the same scrutiny: what operational purpose do they serve, and does the data sent match that purpose?

There’s no basis here to claim that one monitoring architecture would solve the problem, or that the team selected a particular replacement. The useful conclusion is narrower: once a metrics system becomes difficult to query, expensive to replicate, and costly to run beside sensitive workloads, changes need to be judged across all those pressures. Better visibility is valuable only if the system providing it remains dependable under the conditions it’s meant to observe.