Overview
As Darwinium's Akamai solution is an active service in its own right, we recommend it to be monitored in order to ensure the service remains available at all times.
This document describes the Prometheus metrics collected from Darwinium-deployed Linode Kubernetes Engine (LKE) clusters and provides a suggested set of queries for key metrics. Prometheus runs in agent mode inside the cluster, scrapes metrics from local targets, and pushes them via remote_write to the customer's observability platform.
All suggested queries in this document are in PromQL and need to be translated to your platform. Metric names and labels will remain the same as documented.
How Remote Write Works
A Prometheus agent runs inside the cluster as a single lightweight stateless pod. It only scrapes targets and forwards metrics outbound to your platform’s remote_write endpoint via HTTPS.
Consult your vendor's docs for integration:
Grafana Cloud
New Relic
DataDog (via Vector)
Configuration
Receiver endpoints can be configured through the Darwinium portal, under Node Settings -> Edge Deployment -> Your Target Name -> Deployment Config -> Prometheus Shipper Paths
- Add a new Endpoint (an unlimited number may be added).
- Enter URL of receiver
- Select appropriate auth method (basic or barer token)
- Enter Username+Password or Barer Token
Key characteristics
The following standard metrics are available
cadvisor (container resource usage, CPU, memory)
node_exporter (node-level metrics, CPU, memory)
kube-state-metrics (Kubernetes object state, deployments, pods, nodes)
dwn-orca-linode (core application metrics, traffic, errors, latency)
-
Push-based, no inbound network access or VPN tunnels required. Lightweight and stateless.
-
If your remote_write endpoint is temporarily unavailable, the agent buffers in its WAL for up to ~2 hours. After that, unsent data is lost.
-
Metrics arrive as standard Prometheus counters, gauges, and histograms. Your platform handles any type conversion needed (e.g. cumulative-to-delta).
-
Low-value metrics (go_* and process_*) are dropped before sending to reduce ingest volume.
We recommend a critical alert when no metrics arrive from a given dwn_cluster for 5 minutes. This covers any issues that result in metrics interruption from a cluster.
# Should always return data. If absent, the pipeline is broken.
up{dwn_cluster="$dwn_cluster"}
Examples causing no metrics could include:
-
Prometheus agent pod crashed or insufficient resources
-
Network connectivity lost between cluster and your platform
-
Authentication credentials expired or revoked
-
Your platform's ingestion endpoint is down
You can also monitor remote-write health:
# Samples queued but not yet sent (growing means endpoint is slow or unreachable)
prometheus_remote_storage_samples_pending{dwn_cluster="$dwn_cluster"}
# Samples permanently lost
rate(prometheus_remote_storage_samples_dropped_total{dwn_cluster="$dwn_cluster"}[5m])
# Lag in seconds behind real-time
time() - prometheus_remote_storage_highest_timestamp_in_seconds{dwn_cluster="$dwn_cluster"}
These agent self-metrics are also sent via remote-write. If the pipeline is fully broken, you won't see them either, so the "no data" alert on your platform is critical.
We recommend a warning alert if the prometheus remote_write queue is growing, samples are being dropped, or the time lag is more than expected.
Core Service Health
Deployment Replica Status
Except during a rollout/upgrade or scale up/down, the available replicas for dwn-orca-linode should always equal the desired replicas in the Kubernetes deployment, and should equal the available and desired replicas for nginx-ingress-controller
# Available replicas for dwn-orca-linode
kube_deployment_status_replicas_available{deployment="dwn-orca-linode", dwn_cluster="$dwn_cluster"}
# Desired replicas for dwn-orca-linode
kube_deployment_spec_replicas{deployment="dwn-orca-linode", dwn_cluster="$dwn_cluster"}
# Unavailable replicas (desired - available) for dwn-orca-linode
kube_deployment_status_replicas_unavailable{deployment="dwn-orca-linode", dwn_cluster="$dwn_cluster"}
# Available replicas for nginx
kube_daemonset_status_number_available{dwn_cluster="$dwn_cluster", daemonset="nginx-ingress-controller"}
# Desired replicas for nginx
kube_daemonset_status_desired_number_scheduled{dwn_cluster="$dwn_cluster", daemonset="nginx-ingress-controller"}
We recommend a critical alert if the available replicas deviates from the expected number of replicas for dwn-orca-linode or nginx-ingress and it is not an expected rollout or scaling event. dwn-orca-linode pod will report not-ready if it cannot communicate with your origin.
Pod Health
# Pods in crash loops
kube_pod_container_status_waiting_reason{reason=~"ErrImagePull|ImagePullBackOff|CrashLoopBackOff"}
# Pod restart count (exited pods will be automatically restarted by Kubernetes, high rate indicates instability, such as OOM kill)
increase(kube_pod_container_status_restarts_total{dwn_cluster="$dwn_cluster"}[10m])
We recommend a critical alert if any pod is in a crash loop or high restart rate. Examples include continuous OOM kills.
Node Health
The number of ready nodes should always equal the number you have set for your LKE cluster, except during upgrade or scale up/down
# Node readiness
kube_node_status_condition{condition="Ready", status="true", dwn_cluster="$dwn_cluster"}
# Number of nodes
count(kube_node_info{dwn_cluster="$dwn_cluster"})
We recommend a critical alert if the healthy Ready nodes deviates from the expected number of nodes for your cluster and it is not an expected rollout or scaling event.
Core Application Performance and Health
These metrics are exposed by the dwn-orca-linode application.
Request Errors and Failures
# Origin fetch failure rate
rate(failed_fetch_origin_total{dwn_cluster="$dwn_cluster"}[$__rate_interval])
# Origin mapping failure rate (could not match a request to an origin)
rate(failed_map_origin_total{dwn_cluster="$dwn_cluster"}[$__rate_interval])
# Origin 5xx response rate
sum(rate(origin_response_status_code_total{status_code=~"5..", dwn_cluster="$dwn_cluster"}[$__rate_interval]))
# 5xx as percentage of all origin responses
sum(rate(origin_response_status_code_total{status_code=~"5..", dwn_cluster="$dwn_cluster"}[$__rate_interval]))
/
sum(rate(origin_response_status_code_total{dwn_cluster="$dwn_cluster"}[$__rate_interval]))
We recommend a critical alert if the error rate of connecting to your origin, or the error rate of responses from your origin is too high. Origin mapping failures imply that misconfigured traffic that Linode does not expect to receive is being received, and errors are being returned.
Request Traffic
# Incoming request rate (requests/sec per pod)
rate(incoming_request_total{dwn_cluster="$dwn_cluster"}[$__rate_interval])
# Requests with a matched Darwinium Journey step
rate(incoming_request_step_matched_total{dwn_cluster="$dwn_cluster"}[$__rate_interval])
# Currently in-flight requests (gauge)
ongoing_requests{dwn_cluster="$dwn_cluster"}
# Active scheduler count (gauge)
total_scheduler_count{dwn_cluster="$dwn_cluster"}
We recommend a warning alert if the request rate deviates too high or low from expected volumes. High values for ongoing_requests may indicate approaching saturation or slow origin responses.
Origin Response Codes
# Origin response status codes (by status_code and origin)
sum by (status_code, origin) (rate(origin_response_status_code_total{dwn_cluster="$dwn_cluster"}[$__rate_interval]))
Latency (Histograms)
# Origin fetch duration (microseconds) - P95 or pick your quantile. Time to fetch response from the customer's origin server
histogram_quantile(0.95, sum(rate(fetch_origin_duration_microseconds_bucket{dwn_cluster="$dwn_cluster"}[$__rate_interval])) by (le))
# Linode processing duration (microseconds) - P95 or pick your quantile. Time spent in Darwinium processing logic
histogram_quantile(0.95, sum(rate(linode_processing_duration_microseconds_bucket{dwn_cluster="$dwn_cluster"}[$__rate_interval])) by (le))
# Total request duration (microseconds) - P95 or pick your quantile. End-to-end request duration
histogram_quantile(0.95, sum(rate(linode_total_duration_microseconds_bucket{dwn_cluster="$dwn_cluster"}[$__rate_interval])) by (le))
We recommend a warning alert if latencies exceed acceptable values.
Container Resource Usage (cAdvisor)
You can adjust the namespace and container labels to retrieve metrics for any pod in the LKE cluster. The key Darwinium deployed pods processing customer traffic are:
-
dwn-orca-linode (container and namespace dwn-orca-linode)
-
nginx (container and namespace nginx-ingress)
Other namespaces are LKE deployed system components, in kube-system namespace, and the Prometheus agent itself, in prometheus-agent namespace
CPU and Memory
# CPU usage per pod (cores) for dwn-orca-linode
rate(container_cpu_usage_seconds_total{namespace="dwn-orca-linode", container="dwn-orca-linode", dwn_cluster="$dwn_cluster"}[$__rate_interval])
# Memory working set per pod (bytes) for dwn-orca-linode
container_memory_working_set_bytes{namespace="dwn-orca-linode", container="dwn-orca-linode", dwn_cluster="$dwn_cluster"}
We recommend a warning alert if CPU or memory usage for dwn-orca-linode (which is the largest resource consumer on the cluster) exceeds planned capacities for your cluster. Normally we would not recommend consuming more than 50% of node CPU/memory.
Container Network
# Network receive rate for dwn-orca-linode
rate(container_network_receive_bytes_total{namespace="dwn-orca-linode", pod=~"dwn-orca-linode-.*", interface="eth0", dwn_cluster="$dwn_cluster"}[$__rate_interval])
# Network transmit rate for dwn-orca-linode
rate(container_network_transmit_bytes_total{namespace="dwn-orca-linode", pod=~"dwn-orca-linode-.*", interface="eth0", dwn_cluster="$dwn_cluster"}[$__rate_interval])
Node Resource Usage (node_exporter)
CPU, Memory, load averages
# CPU usage per node
avg(sum by (instance, cpu) (rate(node_cpu_seconds_total{mode!="idle", dwn_cluster="$dwn_cluster"}[$__rate_interval]))) by (instance)
# Memory used per node (bytes)
node_memory_MemTotal_bytes{dwn_cluster="$dwn_cluster"} - node_memory_MemAvailable_bytes{dwn_cluster="$dwn_cluster"}
# Load averages
node_load1{dwn_cluster="$dwn_cluster"}
node_load5{dwn_cluster="$dwn_cluster"}
### node_load15{dwn_cluster="$dwn_cluster"}
We recommend a warning alert if CPU or memory usage for the node exceeds planned capacities for your cluster.