This incident has been resolved.
The impact began at 2:00 UTC causing a small fraction of queries in the affected region to be slower. At 11:00 UTC, the issue escalated and became more apparent as p95 latency increased. By 15:00 UTC, the degradation became widespread, with median query performance showing high variance and p95 consistently elevated.
Higher than usual connection times could also be perceived during the incident.
No components marked as affected
Resolved
This incident has been resolved.
The impact began at 2:00 UTC causing a small fraction of queries in the affected region to be slower. At 11:00 UTC, the issue escalated and became more apparent as p95 latency increased. By 15:00 UTC, the degradation became widespread, with median query performance showing high variance and p95 consistently elevated.
Higher than usual connection times could also be perceived during the incident.
Monitoring
We identified and replaced a faulty node in the sa-east-1 region. We are seeing query and connection times return to normal and will continue to monitor.
Investigating
We are investigating degraded supavisor performance in the sa-east-1 region.