Resolved -
This incident has been resolved.
Nov 20, 01:18 UTC
Update -
Things are continuing to look healthy, but out of an abundance of caution, we're going to keep this incident open and continue to monitor.
Nov 20, 00:11 UTC
Monitoring -
A fix has been implemented and we are monitoring the results.
Nov 19, 23:14 UTC
Update -
We're continuing to scale up load across multiple regions. Managment API responsiveness should be improving as we continue this process.
New project creation, Configuration updates, and certain dashboard actions should now be working, but you may still see occasional slowness.
All existing projects, APIs, and databases continue to operate normally.
Nov 19, 22:42 UTC
Update -
The underlying issues are improving, and we are spreading load across globally distributed services. This should result in lower latencies as we continue to spread the load again.
You may still see occasional issues with New project creation, Configuration updates, and Certain dashboard actions until this is completed. All existing projects, APIs, and databases continue to operate normally.
Nov 19, 22:06 UTC
Update -
The team is rolling out another change which we hope to reduce error rates, but will result in longer latencies for responses on affected activities.
This will be done gradually to not cause any further disruption.
Nov 19, 21:04 UTC
Update -
The team is continuing to work on recovery.
New project creation, Configuration updates, and certain dashboard actions are still affected.
All existing projects, APIs, and databases continue to operate normally.
Nov 19, 20:45 UTC
Identified -
We have identified the source of the issue and a fix has already been pushed. We are seeing much lower load on the management layer, and are keeping an eye on its effects on recovery.
Nov 19, 20:03 UTC
Investigating -
We are currently investigating an issue affecting some management-layer services, which may impact: New project creation, Configuration updates, and Certain dashboard actions
All existing projects, APIs, and databases continue to operate normally.
Our engineering team is actively working to resolve this. We will provide updates as soon as more information is available.
Nov 19, 19:37 UTC