Table of Contents

You picked a vendor for observability three years ago. It was the obvious choice: polished UI, simple onboarding, their sales team knew your CTO by name. Now you’re locked in. A 10% price increase lands every January. Your bill is $40K a month. You can’t leave without rewriting every dashboard and losing months of historical data. And the worst part? You could have had this visibility for the cost of your own infrastructure, with zero vendor dependency.

That’s the observability trap. And it’s entirely avoidable.

The open-source observability stack—Prometheus for metrics, OpenTelemetry for instrumentation, Grafana for visualization—gives you production-grade visibility without the vendor handcuffs. This post is for CTOs and senior engineers who’ve felt that squeeze. We’ll walk through how to build it, when it makes sense, and how to migrate off proprietary tools without losing a single data point.

Why Vendor Lock-in Matters for Observability

Observability is not a feature. It’s foundational infrastructure. And when it’s proprietary, you’re renting access to your own production visibility.

Here’s what happens with a vendor-locked observability platform: You instrument your code around their APIs. You build dashboards in their UI. You write alerts in their query language. Your team learns their system. After two years, you have thousands of dashboards, millions of data points, and your infrastructure is woven into their product. The switching cost becomes astronomical—not because the technology is hard to migrate, but because the business risk is unacceptable. You can’t afford downtime while you replatform observability.

Vendors know this. And they price accordingly. The negotiation power shifts entirely to them. A startup that started at $500/month grows to $15K/month at scale, then $50K/month at the size where you actually need sophisticated monitoring. By the time you realize you’re overpaying, you’re too entangled to leave. This is a known problem in the industry—observability vendors have some of the highest net revenue retention rates in SaaS, not because their product gets better every year, but because the switching cost is prohibitive.

The open-source alternative is different. You own your observability stack. You can run it on your own infrastructure, migrate it between cloud providers, switch visualization tools, or replace components without rewriting anything. The instrumentation layer—OpenTelemetry—is becoming a standard. Your code doesn’t depend on any single vendor’s API. If you decide to run Prometheus on Kubernetes instead of a managed service, you change a YAML file. If you want to add a new visualization tool alongside Grafana, you point it at the same Prometheus instance. Vendor independence isn’t a nice-to-have for large engineering organizations. It’s a requirement.

The trade-off is real: you’re responsible for uptime, scaling, and maintenance. But that’s a different kind of cost—one you control, one you can staff for, one that doesn’t surprise you on a quarterly bill.

Prometheus as Your Foundation

Prometheus is the industry standard for metrics collection. It’s not the only metrics database, but it’s the one that won the market—and for good reasons that matter in production.

Prometheus works on a pull model, not push. Your services expose metrics on an HTTP endpoint (typically `/metrics`). Prometheus scrapes that endpoint every 15 seconds (configurable) and stores the data. This is different from vendors like Datadog, which require your services to push metrics to their infrastructure. The pull model means your services don’t need to send traffic outbound to report their health—Prometheus comes to them. It also means if your service crashes, Prometheus knows immediately (the scrape fails). You get a cleaner signal.

Prometheus stores metrics as time-series data. Each metric is a sequence of (timestamp, value) pairs. A single service might emit hundreds of metrics: request latency, error rate, database connection count, garbage collection pause duration. Each one is a separate time series. Prometheus’ query language, PromQL, lets you aggregate, transform, and correlate these series in real time. For example:

rate(http_requests_total[5m]) > 100

This query asks: “How many HTTP requests per second, averaged over the last 5 minutes, are we seeing? Alert me if it’s above 100.” Simple, powerful, and fast.

Prometheus is not infinitely scalable. A single Prometheus instance can handle roughly 1 million time-series before performance starts to degrade (this depends on your hardware and query complexity). For very large organizations—think 100+ microservices, each emitting 500+ metrics—you’ll need Prometheus federation (multiple Prometheus instances that scrape each other) or remote storage (writing metrics to a long-term database like Thanos or Cortex). But for 90% of companies, a single Prometheus instance running on moderate hardware will handle years of growth. That’s the economics of open source: you’re not paying for unlimited scale. You’re paying with the engineering effort to scale when you need it.

Setup is straightforward. Download the Prometheus binary, write a simple YAML config file that lists your services’ metrics endpoints, and start it:

global:
scrape_interval: 15s
scrape_configs:
- job_name: 'api-service'
static_configs:
- targets: ['localhost:8080']
- job_name: 'worker-service'
static_configs:
- targets: ['localhost:8081']

That’s it. Prometheus starts scraping `/metrics` from both services. Within minutes, you have metrics flowing in. Run it in Docker, in Kubernetes, or on bare metal—it works everywhere.

The hard part isn’t running Prometheus. It’s deciding what to measure and writing the instrumentation code in your services. That’s where OpenTelemetry comes in.

OpenTelemetry: The Instrumentation Standard

OpenTelemetry (OTel) is the open standard for instrumenting code. It handles metrics, traces, and logs—the three pillars of observability. For this post, we’re focusing on metrics, but OTel’s power is that you instrument once and can export to any backend.

The key insight: Your application code should not depend on your observability vendor. If you write metrics directly to Prometheus format, you’re locked into Prometheus. If you write to Datadog’s API, you’re locked into Datadog. OpenTelemetry breaks that dependency. You instrument your code using the OTel SDK, and you can export metrics to Prometheus, Datadog, Jaeger, or any backend that supports OTel. Change backends? Just change your exporter configuration. The code stays the same.

Here’s what instrumentation looks like in Node.js with OpenTelemetry:

const { MeterProvider, PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');

const exporter = new PrometheusExporter({}, () => {
console.log('Prometheus server started on port 8888');
});

const meterProvider = new MeterProvider({
readers: [exporter],
});

const meter = meterProvider.getMeter('my-app');
const requestCounter = meter.createCounter('http_requests_total');
const latencyHistogram = meter.createHistogram('http_request_duration_ms');

// In your request handler:
requestCounter.add(1, { method: 'GET', status: 200 });
latencyHistogram.record(responseTime, { method: 'GET', endpoint: '/api/users' });

The exporter handles translating OTel metrics into Prometheus format. Your code doesn’t care about Prometheus. If you later decide to use a different backend, you swap the exporter. The instrumentation stays.

OpenTelemetry has SDKs for every major language: Go, Python, Java, .NET, Node.js, Ruby. There are also automatic instrumentation packages that hook into popular frameworks (Express, Django, Spring) and capture metrics without requiring code changes. For example, the Node.js auto-instrumentation will automatically track HTTP request latency, database query duration, and error rates if you add a single require statement at the top of your app.

The challenge with OTel is choosing what to instrument. You don’t want to measure everything—that creates noise and burns through storage. You want business metrics (signup rate, checkout success rate) and system metrics (request latency, error rate, resource usage). Some frameworks provide these automatically. Others require you to decide. A good rule: instrument at the boundaries. Track incoming requests, outgoing database queries, calls to external APIs. Track the metrics that correlate with user impact.

Grafana as Your Visualization Layer

Prometheus is your metrics database. Grafana is your visualization and alerting layer. Grafana connects to Prometheus (or any metrics database), lets you build dashboards, and fires alerts when metrics cross thresholds.

Grafana is open source and free to self-host. You can run it in Docker, Kubernetes, or on a VM. It has a web UI for building dashboards without code, and it supports version control for infrastructure-as-code teams. Dashboards are JSON files. You can store them in Git, diff them, review changes, and roll back if needed.

Here’s what a Grafana dashboard might look like for an API service:

Panel 1: Request Rate
Query: rate(http_requests_total[5m])
Shows requests per second, broken down by endpoint and status code.

Panel 2: Latency (P95)
Query: histogram_quantile(0.95, rate(http_request_duration_ms_bucket[5m]))
Shows the 95th percentile latency over time. If 95% of requests are under 200ms, you’re in good shape.

Panel 3: Error Rate
Query: rate(http_requests_total{status=~"5.."}[5m])
Shows 5xx errors per second. An alert fires if this exceeds 0.1 errors/sec.

Panel 4: Database Query Duration
Query: histogram_quantile(0.99, rate(db_query_duration_ms_bucket[5m]))
Shows the 99th percentile database query latency. Helps catch slow queries creeping in.

Alerting in Grafana works by defining rules that evaluate PromQL queries. If the query result exceeds a threshold, Grafana can send notifications to Slack, PagerDuty, email, or any webhook-compatible system. Example alert rule:

alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
for: 5m
annotations:
summary: "Error rate is {{ $value }} errors/sec"
runbook: "https://wiki.internal.com/runbooks/high-error-rate"

This rule fires if the error rate exceeds 0.1 errors/second for 5 minutes straight. The 5-minute grace period prevents flaky alerts—a single blip won’t page you at 3am. The annotation includes a link to your runbook so the on-call engineer knows what to do.

The visualization layer is where Grafana really shines compared to Prometheus alone. Prometheus has a basic UI for querying metrics, but it’s not suitable for dashboards. Grafana gives you variable templating, multi-series overlays, heatmaps, and all the visualization tools ops teams expect. And because it’s open source, you can self-host it, modify it, or integrate it with your internal tooling.

Self-Hosted vs Managed Trade-offs

You can run Prometheus and Grafana on your own infrastructure, or you can use managed services like Grafana Cloud. Understanding the trade-offs is critical for sizing the decision correctly.

Self-hosted (you run Prometheus and Grafana on your infrastructure):
Pros: No monthly fees beyond infrastructure cost (negligible). Complete control over data. No vendor dependency. Can integrate with internal authentication systems. Can customize Prometheus scrape configs on the fly.
Cons: You’re responsible for uptime, backups, and scaling. Prometheus data is stored on disk—if your Prometheus instance dies, you lose historical data (though you can add remote storage). Requires operational overhead (monitoring the monitor, as they say).

Managed (Grafana Cloud for dashboards, Prometheus remote storage service like Datadog, New Relic, or Grafana’s own Prometheus service):
Pros: No infrastructure to manage. Automatic backups and high availability. Scales without you thinking about it. Easier for distributed teams (no need to tunnel to an internal instance).
Cons: Monthly costs that scale with your data volume. Vendor dependency on the managed provider. Data lives in their infrastructure (compliance concern for some organizations). Harder to customize.

For most companies, hybrid is the sweet spot: self-hosted Prometheus for metrics collection and short-term storage (7-30 days), with remote storage for long-term retention. Grafana can query both the local Prometheus (fresh data) and the remote storage (historical queries). You get the operational simplicity of self-hosting the hot path and the scalability of managed storage for cold data.

If you go hybrid, tools like Thanos (open source) or Cortex (also open source, now part of Loki) provide long-term metrics storage. They’re designed to work with Prometheus and handle multi-tenancy, high-cardinality metrics, and querying across retention boundaries. Setup is more complex than vanilla Prometheus, but the trade-off is clear: you get unlimited data retention and multi-region querying for the cost of managing another service.

One more consideration: metrics cardinality. High-cardinality metrics (metrics with many unique label combinations) consume disproportionate storage and query resources. For example, if you emit a metric like http_request_duration_ms{endpoint, method, status, user_id} and user_id is a unique identifier, you’ve created one time-series per user per endpoint per method per status code. With 10K users, 100 endpoints, 5 methods, and 10 status codes, that’s 50 million time-series. Prometheus will choke. The fix: don’t include high-cardinality labels like user_id in metrics. Use them in traces instead (OpenTelemetry handles this). Only use low-cardinality labels (method, status, endpoint) in metrics.

Migration Path from Proprietary Tools

If you’re currently using a proprietary observability platform, migrating to open source is feasible but requires planning. You can’t flip a switch and move everything overnight without losing visibility.

Phase 1: Parallel Instrumentation (2-4 weeks)
Start instrumenting new code with OpenTelemetry instead of your vendor’s SDK. Existing code continues to ship metrics to the vendor. Gradually migrate instrumentation as you touch code. This isn’t a rewrite—you’re simply using OTel instead of Datadog/New Relic/whoever. The exporter sends metrics to Prometheus while you’re still running the vendor’s agent.

Phase 2: Prometheus Setup (1-2 weeks)
Deploy Prometheus to collect metrics from services now using OTel. Run it alongside your vendor platform. Get your team familiar with PromQL. Start building dashboards in Grafana that mirror your vendor dashboards. Don’t delete the vendor dashboards yet—keep them for comparison.

Phase 3: Gradual Dashboard Migration (4-8 weeks)
Build out Grafana dashboards for critical systems. Test alerting. Validate that your Prometheus data matches your vendor data (lag in scrape intervals might cause small differences—that’s normal). Once you’re confident, cut over alerting to Grafana. Vendor dashboards become read-only references.

Phase 4: Sunset Vendor (2-4 weeks)
Stop paying the vendor. Archive their dashboards and data. Keep Prometheus running. You’ve migrated.

The key risk during migration is losing alerting coverage. Avoid this by running both systems in parallel for 2-3 weeks. Make sure every alert you have in the vendor system exists in Grafana and is firing correctly before you turn off the vendor alerts.

For historical data, you can’t migrate it (vendor data is proprietary). But you can keep your vendor account in read-only mode for 90 days if you need to refer back to old dashboards. After that, the cost is negligible.

One practical example: A SaaS company we worked with was paying $35K/month to a vendor for observability across 40 microservices. They migrated to self-hosted Prometheus + Grafana over 8 weeks. The infrastructure cost for Prometheus (a 4-core, 16GB instance in Kubernetes) was $400/month. Grafana added another $200/month (managed Grafana Cloud for dashboards, since they preferred not to manage another service). Total cost: $600/month. Savings: $34.4K/month. The migration required 300 engineering hours (roughly 2 senior engineers for 8 weeks), which the company calculated as a 6-month payback. After that, it was pure savings. Plus, they owned their observability stack.

The open-source observability stack—Prometheus, OpenTelemetry, and Grafana—is production-ready and battle-tested at scale. You’re not experimenting with bleeding-edge tools. You’re using what Netflix, Kubernetes, and thousands of other organizations depend on in production. The cost difference is dramatic. The freedom is tangible. And the complexity is entirely manageable if you staff it correctly.

If your observability bill has become a line item that makes your CFO wince, or if you’ve felt the squeeze of vendor lock-in, this is the path forward. The engineering cost to migrate is real, but it’s a one-time cost. The savings are indefinite. We help teams through this migration during our Sprint and Build engagements—it typically maps to a Sprint (2-4 weeks, one shipped outcome: Prometheus + Grafana operational, alerting migrated, vendor contract canceled). If you’re evaluating whether to move, apply for an engagement. The application takes ten minutes, and we can scope the migration in a conversation.