Twelve PoPs across 5 continents, reporting request rate, error rate, and latency percentiles every 500ms, produce at least 72 metric values per second before adding dimensions, logs, traces, or deployment events. That is 4,320 values per minute and 259,200 per hour from the basic telemetry stream alone. The hard part is not collecting those values. It is preserving the signal when one region is quiet, another is degraded, and the control plane is receiving a constant stream of normal-looking data.
At Symbiosis, our PoPs are in Stockholm, London, Amsterdam, Frankfurt, Paris, New York, Chicago, Los Angeles, Sao Paulo, Singapore, Tokyo, and Sydney. Designing observability for this footprint has meant treating telemetry as a decision system rather than a warehouse of measurements.
Start with a small, consistent telemetry model
Every PoP reports the same core measurements on the same 500ms cadence. Request rate tells us whether traffic is arriving. Error rate tells us whether requests are being served successfully. Latency percentiles show how the experience is changing for users, including the slower requests that an average can hide.
Consistency matters more than collecting every possible field at the edge. If each location reports different names, intervals, or percentile definitions, the control plane spends its time reconciling incompatible data. A common model makes regional comparisons possible and gives downstream systems a stable input.
The raw stream should not be the only view used for diagnosis. We retain high resolution for active conditions, while also producing time-windowed views for patterns over minutes and longer periods. Aggregation is not just a cost control. It gives an operator a way to distinguish a short spike from a sustained change.
Aggregate by question, not just by time
A useful aggregation answers a specific operational question:
- Is a PoP receiving traffic?
- Are failures isolated to one location or present across the fleet?
- Is latency increasing for all requests or only for the slowest percentile?
- Did the condition begin at one PoP or at the control plane?
Those questions suggest different views of the same stream. A fleet-wide error rate can reveal a broad incident, while a per-PoP error rate can identify the location. A regional request-rate view helps prevent a quiet location from being mistaken for a failed one. Latency percentiles should remain associated with the PoP and relevant time window, because a healthy global percentile can conceal a severe local problem.
Labels also need restraint. Location is essential. Additional dimensions should earn their place by supporting a real diagnostic question. Every extra dimension multiplies the number of time series and makes dashboards harder to read. High-cardinality data can still be valuable, but it belongs in a more targeted diagnostic path than in the primary fleet view.
Design alerts around evidence
An alert should express a condition and the evidence that makes it actionable. “Latency is high” is a useful observation, but it is not enough on its own. A better alert combines a latency change with a meaningful request volume and a defined duration. The exact thresholds depend on the service, but the structure is portable.
Traffic absence needs similar care. A zero request rate at a PoP can mean that the PoP is unavailable. It can also mean that the region has no traffic at that moment. A traffic-based alert without an independent health signal will page on normal geography.
We separate alerts into two broad categories. Condition alerts describe user impact, such as elevated error rate or latency. Availability alerts describe whether a PoP is reporting and reachable. Comparing those categories is more informative than either one alone. A missing telemetry report with no user traffic is different from a missing report while requests are expected and errors are rising.
Scenario: one PoP goes dark
Consider a synthetic incident in which the Singapore PoP stops reporting. The control plane sees no new telemetry from that location. At the same time, request rate for Singapore is low because of the time of day.
The first question is whether this is a reporting problem, a PoP failure, or simply a quiet region. We compare the last report timestamp with the expected 500ms cadence, then inspect the independent signals available to the platform. If request rate was already near zero and neighboring locations show normal behavior, the evidence for a user-impacting outage is weak. If Singapore had steady traffic immediately before telemetry stopped, or if errors and latency were rising, the incident deserves a different priority.
The alert should state that distinction. “Singapore telemetry missing” is precise. “Singapore unavailable” is a stronger claim and requires stronger evidence. Good alert wording prevents the first responder from treating an inference as a fact.
Export without creating another source of confusion
Our observability integrations include Datadog and Grafana. Export is useful when teams already have dashboards, notification policies, and investigation habits in those systems. The exported data should retain the same identity and aggregation rules as the control plane. Otherwise, operators compare a platform view and an external view that appear to disagree because they used different windows or definitions.
The control plane also uses the telemetry stream for AI traffic prediction. That makes data quality operationally important beyond incident response. Missing reports, inconsistent labels, and unexamined regional gaps can affect both what an operator sees and how future traffic is anticipated.
Alert fatigue is not solved by alert silence
There is a real tension between too many alerts and missed signals. Reducing notification volume can make a system feel calmer while hiding a slow degradation. The answer is not to alert on every threshold crossing, nor to suppress everything until a major failure is obvious.
We prefer alerts that have a clear owner, a clear time horizon, and enough context to support the next decision. Everything else can remain visible in dashboards and exported telemetry. The boundary is important: observability can show that a condition exists, but it cannot by itself explain every underlying cause or replace an investigation.
For a distributed edge platform, useful observability is disciplined compression. Keep the 500ms stream for timely detection, aggregate it into views that answer operational questions, and preserve the distinction between missing data, missing traffic, and actual failure. That is how twelve locations become one system that can still be understood when the signals stop agreeing.