Replacing SigNoz with VictoriaMetrics on a 4 GiB budget

If your telemetry backend is using more memory than the services it monitors, you aren't alone and it's not necessarily misconfigured. ClickHouse-backed observability stacks have a floor, and below that floor they degrade in a specific and dangerous way: they stay up and drop data.

We replaced SigNoz with VictoriaMetrics, VictoriaLogs and VictoriaTraces on 14 August 2026. The applications didn't change - not one line - because everything already spoke OTLP to a collector, and swapping a backend behind a collector is a config change. The whole migration was an afternoon.

Here are the numbers, measured on 16 August.

Every application container in production, added up:

28 containers, 1,456 MiB. A dozen products, one of them running in five regions.

The telemetry stack watching them had a 4 GiB cap, and was still dropping metrics and traces. Nearly three times the memory budget of everything it monitored, and not enough.

The failure mode that matters

It didn't fall over. That's the part worth internalising, because a crash is a page and this isn't.

A memory-starved ingest pipeline sheds load. Metrics went first, then traces. The API kept returning 200. Dashboards kept rendering. A dashboard with a hole in it and a dashboard where nothing happened look identical, so the system quietly told us slightly less than the truth for an unknown period.

If you're running ClickHouse-backed telemetry near a memory ceiling, this is the thing to alert on - ingest rate against expected rate, not process liveness. Liveness will tell you nothing.

What we moved to, exactly

Three storage services, one collector, Grafana. No ClickHouse, no ZooKeeper, no JVM.

Measured two days after cutover:

~1.2 GiB for store, collect and query. Another 113 MiB buys things SigNoz never did at all: blackbox probes, alertmanager, vmalert, and an ntfy bridge that puts alerts on a phone.

Retention is set with -retentionPeriod: 60 days on metrics, 3 months on logs and traces.

The collector wiring

This is the part that's annoying to work out from the docs, so here it is. The three Victoria services take OTLP directly - you don't need a Prometheus remote write path or a Jaeger sidecar:

exporters:
  otlphttp/vm:
    metrics_endpoint: http://victoria-metrics:8428/opentelemetry/v1/metrics
  otlphttp/vl:
    logs_endpoint:    http://victoria-logs:9428/insert/opentelemetry/v1/logs
  otlphttp/vt:
    traces_endpoint:  http://victoria-traces:10428/insert/opentelemetry/v1/traces

service:
  pipelines:
    metrics/apps: { receivers: [otlp], processors: [batch], exporters: [otlphttp/vm] }
    logs/apps:    { receivers: [otlp], processors: [batch], exporters: [otlphttp/vl] }
    traces/apps:  { receivers: [otlp], processors: [batch], exporters: [otlphttp/vt] }

Point the collector's OTLP receiver at the port the old collector used and the applications never know. Ours kept :4318 and :4317.

Keep the app pipelines free of resource processors. Telemetry arriving over OTLP already carries its own service.name and deployment.environment; tag only what you collect locally, or you'll overwrite the identity of every service with the identity of the collector host.

What it cost

One query surface became three. SigNoz gives you one place to move between a trace, its logs and the metrics around it. Victoria gives you three HTTP APIs: :8428 for metrics in MetricsQL, :9428 for logs in LogsQL, :10428 for traces via the Jaeger API. Grafana covers most of the join. Not all of it, and the trace waterfall is genuinely worse.

Every hand-written query broke. MetricsQL renders dotted OTLP names in Prometheus style, so system.cpu.load_average.1m becomes system_cpu_load_average_1m. Budget an afternoon for dashboards and alert rules.

Three query languages instead of one. MetricsQL, LogsQL and Jaeger. If your on-call rota is more than a couple of people, that's a real training cost and worth weighing against the memory saving.

When SigNoz is still the right answer

If you have the memory, SigNoz is the better product and I'd still recommend it. The correlated view is worth real money at 3am, and none of the above is an argument that it's badly built - it's an argument that it has a floor and we were under it.

Specifically, don't do this if you need a single correlated pane more than you need the RAM back, if your team is large enough that three query languages costs more than a memory upgrade, or if you're already paying for managed telemetry and it works.

And there's a ratio worth checking before anyone argues about architectures: the monitoring should cost less than the thing being monitored. Ours had it backwards, which is the whole story above - a stack with a bigger memory budget than every application it was watching, and still dropping data. So the crossover is where that ratio flips. Once your application estate genuinely outweighs the telemetry serving it, ClickHouse stops being overkill and starts being right, because you're buying query power with resources you can spare. Below it you're paying for an engine to store observations about a system smaller than the engine.

Do consider it if you're sizing telemetry against a hard cap, if your ingest is modest, or if you've started seeing gaps you cannot explain.

I deleted the before-graph

I wanted to show the memory graph across the cutover. The "before" was in ClickHouse, and I deleted ClickHouse, so the history of how much ClickHouse was using went with it. VictoriaMetrics' first datapoint is the day I installed it.

Everything above is the after, measured, and the before is a 4 GiB cap and what I watched it do inside that. Any ratio I gave you would have a made-up numerator.

If you're about to replace an observability stack, export the memory and ingest history first. I didn't.

The size limit nobody quotes

Self-managed infrastructure has a size limit, and it isn't "will it run". It's "will it run in what's left after the things that pay for it". SigNoz ran. It ran the way software runs when you're squeezing it, which is that every API call succeeds and the job quietly doesn't get done.

The replacement is still roughly the same size as everything it watches - 1,343 MiB of observability against 1,456 MiB of product. I don't think that's wrong exactly, watching a dozen things costs something, but it's not a ratio I'd have guessed before measuring it.

If you're running something comparable and your ratio looks better, I'd genuinely like to know what you're doing differently.