Introduction
Effective monitoring is the difference between discovering incidents through user complaints and catching them proactively through dashboards and alerts. The three dominant platforms in the observability space--Grafana, Datadog, and New Relic--each take distinct approaches to metrics, logging, tracing, and alerting. This article provides a technical comparison to guide your selection.
Dashboarding Capabilities
Grafana
Grafana excels at visualization with support for dozens of data sources:
{
"dashboard": {
"title": "Production Overview",
"panels": [
{
"title": "HTTP Request Rate",
"type": "timeseries",
"datasource": "Prometheus",
"targets": [{
"expr": "sum(rate(http_requests_total[5m])) by (service)",
"legendFormat": "{{ service }}"
}]
},
{
"title": "Service Latency (p99)",
"type": "stat",
"datasource": "Tempo",
"targets": [{
"query": "{.name = \"HTTP GET\"} | stats p99(duration_ms) as p99 by service"
}]
},
{
"title": "Error Budget",
"type": "gauge",
"datasource": "Prometheus",
"targets": [{
"expr": "(1 - (sum(rate(http_requests_total{status=~\"5..\"}[30d])) / sum(rate(http_requests_total[30d])))) * 100"
}],
"thresholds": {
"steps": [
{"value": null, "color": "green"},
{"value": 99.9, "color": "yellow"},
{"value": 99.99, "color": "red"}
]
}
}
]
}
}
Datadog
Datadog provides a more opinionated dashboarding experience with integrated template variables:
{
"title": "Service Overview",
"widgets": [{
"definition": {
"type": "timeseries",
"requests": [{
"q": "avg:http.requests{service:payment} by {endpoint}.as_rate()",
"display_type": "line",
"style": {"palette": "warm"}
}],
"yaxis": {"scale": "linear", "min": "auto"}
}
}]
}
New Relic
New Relic uses NRQL, a SQL-like query language for dashboards:
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\-- NRQL query
SELECT percentile(duration, 99) AS 'p99'
FROM Transaction
WHERE appName = 'Payment Service'
TIMESERIES auto
SINCE 1 hour ago
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\-- Error rate query
SELECT count(*) AS 'errors'
FROM TransactionError
WHERE appName = 'Payment Service'
FACET error.message
LIMIT 10
Alerting Configuration
Grafana Alerting
Grafana managed alert rule
apiVersion: grafana/v1
kind: AlertRule
metadata:
name: HighErrorRate
spec:
for: 5m
annotations:
summary: "Error rate above threshold for Payment Service"
runbook_url: "https://runbooks.internal/payment-high-errors"
labels:
severity: critical
team: platform
data:
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- ref: A
datasourceUid: prometheus
model:
expr: |
sum(rate(http_requests_total{
service="payment", status=~"5.."
}[5m])) / sum(rate(http_requests_total{
service="payment"
}[5m])) > 0.05
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- ref: B
datasourceUid: prometheus
model:
expr: "1"
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- ref: C
datasourceUid: expr
model:
expression: "$A && $B"
type: math
Datadog Monitors
Datadog monitor via API
monitor:
name: "[Payment] High Latency Alert"
type: metric alert
query: "avg(last_5m):p99:trace.servlet.request.duration{service:payment} > 1"
message: |
{{#is_alert}}
Payment service p99 latency is {{value}}s (threshold: 1s)
@slack-alerts
{{/is_alert}}
options:
thresholds:
critical: 1.0
warning: 0.5
notify_no_data: true
evaluation_delay: 60
new_group_delay: 300
APM and Distributed Tracing
Datadog APM
from ddtrace import tracer, patch_all
Auto-instrument supported libraries
patch_all()
Custom instrumentation
@tracer.writer(service_name="payment-service")
def process_payment(order_id, amount):
with tracer.trace("payment.charge") as span:
span.set_tag("order_id", order_id)
span.set_metric("amount", amount)
result = gateway.charge(amount)
span.set_tag("transaction_id", result.id)
return result
New Relic APM
import newrelic.agent
Custom transaction
@newrelic.agent.background_task()
def process_refund(transaction_id):
with newrelic.agent.FunctionTrace(name="refund.process"):
refund_result = refund_gateway.process(transaction_id)
newrelic.agent.record_custom_metric(
"Custom/RefundAmount", refund_result.amount
)
return refund_result
Log Integration
| Feature | Grafana + Loki | Datadog Logs | New Relic Logs |
|---|---|---|---|
| Structured parsing | LogQL | Grok parser | NRQL parsing |
| Ingestion cost | Low (S3-based) | Medium | Medium |
| Retention | Configurable | 15 days default | 30 days default |
| Live tail | Yes | Yes | Yes |
Example Loki query for log correlation:
{service="payment"} |= "ERROR"
| logfmt
| duration > 1s
| line_format "{{.timestamp}} {{.message}} (duration: {{.duration}})"
Pricing Comparison
| Tier | Grafana (self-hosted) | Grafana Cloud | Datadog | New Relic |
|---|---|---|---|---|
| Free | Unlimited | 3 users, 10k series | 5 hosts, 15d retention | 100GB/month, 1 user |
| Entry | Server cost only | $49/month | ~$15/host/month | ~$0.55/GB |
| Enterprise | Support cost | Custom | Custom | Custom |
Grafana self-hosted is the most cost-effective at scale because you only pay for infrastructure. Datadog and New Relic pricing scales with data volume and can become expensive for high-cardinality metrics or verbose logging.
Self-Hosted vs SaaS
-
Grafana : Excellent self-hosted option with Prometheus, Loki, and Tempo forming a complete open-source stack. Grafana Cloud offers a managed alternative.
-
Datadog : SaaS-only with strong integrations but vendor lock-in. No self-hosted option exists.
-
New Relic : Cloud-first but offers a data-ingestion API that allows hybrid collection patterns.
For startups and small teams, Grafana self-hosted provides the best balance of capability and cost. As teams grow to 20+ engineers, Datadog's out-of-the-box integrations reduce operational overhead. New Relic is compelling for organizations already in the Oracle/AWS ecosystem that value NRQL's analytical power.
Enjoy this article? Share your thoughts, questions, or experiences in the comments below — your insights help other readers too.
Join the discussion ↓