Network monitoring tends to start small. One engineer sets up a time series database, adds a few exporters and a dashboard, and the team is happy. A year later the same instance is scraping hundreds of switches, flow data has been added, and every query takes long enough for people to stop looking at the graphs.
At that point the question is no longer which tool to use. It is how to design and size the stack so that it stays fast as the network grows. This guide covers the three pressures that decide this: metric cardinality, flow volume and retention.
Understand What Actually Grows
Monitoring systems rarely fail because of the number of devices. They fail because of the number of unique data series those devices produce.
A switch with 48 ports does not produce one metric. It produces traffic in and out, errors, discards and status for each port, and each combination of labels forms its own series. Add per-VLAN breakdowns or per-customer tags and the count multiplies again. The Prometheus storage documentation gives the basic capacity relationship: the disk needed is roughly the retention time multiplied by the ingested samples per second and the bytes per sample. Active series, meanwhile, drive memory.
Before buying anything, measure these numbers on your current system:
- Active series and how quickly the count is growing.
- Samples ingested per second, which depend on series count and scrape interval.
- Flow records per second at peak, if you collect NetFlow, IPFIX or sFlow.
- Query patterns, including how far back people really look.
The last point often surprises teams. Most dashboards read the last few hours, while long-range queries come from a small number of capacity reports. That gap is what makes tiered retention worthwhile.
Treat Metrics and Flows as Two Separate Problems
Metrics and flow records look similar on a dashboard but behave very differently on hardware.
|
Characteristic |
Device metrics |
Flow records |
|
Data shape |
Regular samples per series |
Bursty records, one per conversation |
|
Main bottleneck |
Memory for active series |
Write throughput and disk space |
|
Compression |
Very good on regular values |
Moderate, depends on fields kept |
|
Typical query |
A few series over hours or days |
Top talkers and aggregations across many records |
|
Cost driver |
Label cardinality |
Sampling rate and retention |
|
Good storage engine |
Time series database |
Columnar or search-oriented database |
Keeping them separate pays off. A sudden flow burst during a DDoS event is precisely the moment you need your device metrics to keep working. If both share the same database and disks, the incident you are trying to investigate can also degrade your view of it.
Flow sampling deserves an honest conversation as well. Sampling one packet in a thousand may be perfectly adequate for capacity planning, but it will miss short, low-volume events. Agree on what the data is for before you set the rate, because that single setting can change your storage needs many times over.
Design Retention in Tiers
Keeping everything at full resolution forever is the most common way to overspend. A tiered policy keeps recent data detailed and older data cheap.
Before you commit, check whether any compliance rules apply to flow records. In some jurisdictions and industries, these records count as sensitive or subject to retention limits, so the longest tier is sometimes a legal decision and not an engineering one.
Sizing the Server
Once you know your series count, samples per second and flow rate, the hardware follows from them. Monitoring workloads are mostly about memory and storage, with parallel processing for ingestion and queries. A single fast core will not rescue a system whose problem is too many series in memory or too few disk operations.
A consolidated monitoring node for a mid-sized network typically needs:
- Enough RAM to hold all active series with room to grow, plus memory for the flow database.
- Several NVMe drives, separating write-ahead logs from long-term data where the software supports it.
- Enough cores to handle ingestion, compaction and concurrent dashboard queries at once.
- A network port with headroom for incoming telemetry, because flow exports and scrapes add up.
That combination of high core counts, memory capacity and NVMe connectivity is why operators often look at a dedicated server amd epyc configuration when they move monitoring off shared virtual machines. Dedicated hardware also has a practical benefit in this role: your telemetry is not competing with other tenants during the exact moment when everything else is on fire.
Whichever platform you choose, place the monitoring system outside the failure domain it observes. If the monitoring server sits behind the same core switch it is watching, you will lose visibility exactly when you need it. An out-of-band path or a second location is worth the extra cost.
Keep the Stack Healthy
A monitoring system needs monitoring too, and these habits prevent most slow degradation:
- Alert on cardinality growth. A new label on one exporter can add millions of series overnight, so track the active series count and alert on sudden changes.
- Drop what nobody uses. Review which metrics dashboards and alerts actually read. Unused interface counters are usually the easiest saving.
- Test restores. Back up configuration and dashboards as code, and restore them on a clean machine at least twice a year.
- Run a dead man’s switch. Send a constant alert to an external service, so that silence itself triggers a page.
- Load test before growth. When adding a new site, replay its expected telemetry against the stack first.
Done well, this work makes monitoring boring, and that is the goal. Engineers should be able to open a dashboard in the middle of an incident and trust that it loads quickly and tells the truth. Measure your series and flow rates, split the two workloads, tier your retention, and size the server for the growth you can already see in your own data.