API Requests to projects being blocked at the API Gateway layer

Incident Report for Supabase

Postmortem

Between 16:02 UTC and 17:15 UTC on March 22nd 2025, customers who were accessing Supabase services from Next.js middleware had their Supabase requests blocked by our upstream CDN provider.

The impacted requests received a “Sorry, you have been blocked” error response. The primary service impacted was Supabase Auth, as performing auth in Next.js middleware is a common and recommended usage pattern. A smaller number of requests to other Supabase services were also impacted.

We sincerely apologise for the negative effects our customers experienced. Here is some additional detail about what happened and what we will do to mitigate future outages of this nature.

Who was affected?

Customers who access Supabase services from Next.js middleware, or otherwise use a x-middleware-subrequest header were affected. In total, 9.59M requests were blocked across our customers endpoints.

What happened?

On March 21st 2025, Vercel/Next.js published a security advisory for CVE-2025-29927, which allowed for Authorization Bypass in Next.js Middleware. In addition to advising customers to upgrade to a patched version of Next.js, the security advisory contained a workaround that noted “If patching to a safe version is infeasible, it is recommend that you prevent external user requests which contain the x-middleware-subrequest header from reaching your Next.js application.”

In an effort to protect their customers from this CVE, our CDN provider implemented a new managed WAF rule to block requests including the x-middleware-subrequest header, effectively implementing the suggested workaround for the CVE and rolled it out to their customers, which includes Supabase.

The incident timeline was as follows:

  • SAT 22 MAR 14:53 UTC: Our CDN provider posted in our shared slack channel informing us they were rolling out a patch for CVE-2025-29927 and that we would be the customer most impacted by this change. Slack is not monitored or part of our internal escalation mechanisms and our on-call engineers were not aware of the impending change.
  • SAT 22 MAR 16:02 UTC: The WAF rule was applied to the Supabase CDN.
  • SAT 22 MAR 16:09 UTC: We began to receive a high volume of customer reports indicating problems with the CDN blocking requests.
  • SAT 22 MAR 16:17 UTC: We declared an incident and response teams were paged and assembled.
  • SAT 22 MAR 17:07 UTC: We had identified the issue was related to the new WAF rule.
  • SAT 22 MAR 17:15 UTC: We worked with our CDN provider to disable the WAF rule.

Once the WAF rule was disabled, customers were once again able to use the x-middleware-subrequest header and the incident was resolved.

What will we do to mitigate problems like this in the future?

  1. During the incident, we discovered gaps with vendor relationships and we found it difficult to engage directly with our CDN provider. We have taken action to ensure that we have clearer methods of communication in the future for incidents that may impact our mutual customers.
  2. We have added alerting which will help us identify anomalies in the volume of customers requests that are blocked by our CDN provider, which will help us identify and resolve issues with harmful WAF rules faster in the future.
  3. We are investigating fallback CDN providers and other options for our CDN.

What actions do you need to take?

All customers using Next.js are encouraged to patch to the latest version if you are on a version that is affected by CVE-2025-29927.

If you are hosting your Next.js application with Vercel or Netlify you do not need to take any action as these platforms have built in protection from this CVE, however it is still a good idea to upgrade to a patched version of Next.js.

If you are hosting your Next.js application with a provider that is not Vercel or Netlify, we encourage you to urgently upgrade your application to a safe Next.js version as noted in the Security Advisory.

Posted Mar 26, 2025 - 12:27 UTC

Resolved

This incident has been resolved.
Posted Mar 22, 2025 - 17:54 UTC

Monitoring

The fix has been rolled out. Block rates at the API layer have returned to normal levels, and things are looking stable now. API requests to your supabase projects should now be working as expected.
Posted Mar 22, 2025 - 17:22 UTC

Identified

With the assistance of our API gateway partner, we have identified the configurations resulting in the Blocking and are working to update them.
Posted Mar 22, 2025 - 17:14 UTC

Update

Our team has reached out to our API gateway partners and are continuing to investigate the periodic Block errors that folks are receiving across all API endpoints.
Posted Mar 22, 2025 - 17:03 UTC

Investigating

We are receiving reports of user API requests receiving API gateway errors when making Auth and other API calls. The team is currently looking into the reason and for mitigating steps.
Posted Mar 22, 2025 - 16:21 UTC
This incident affected: API Gateway.