This incident has been resolved.

Incident Summary Between 22:30 CET and 02:00 CET, access to a subset of Umbraco Cloud websites hostnames was inadvertently blocked for visitors. Impact During the incident, affected users received HTTP 403 (Forbidden) responses when attempting to access impacted websites. Root Cause The issue was caused by a misconfigured security rule in the web application firewall. Resolution The configuration was corrected at 02:00 CET (28/10), and normal access was restored immediately. Preventive Actions We are reviewing our deployment and validation processes to prevent similar misconfigurations in future security rule updates.

Incident Summary Between 08:31 CET and 14:14 CET, access to Umbraco Cloud websites using custom or internal hostnames was inadvertently blocked for visitors from the following countries: Afghanistan, Belarus, China, Cuba, Iran, North Korea, Russia, and Syria. Impact During the incident, affected users received HTTP 403 (Forbidden) responses when attempting to access impacted websites. Root Cause The issue was caused by a misconfigured security rule in the web application firewall, which incorrectly applied geographic blocking to these regions. Resolution The configuration was corrected at 14:16 CET, and normal access was restored immediately. Preventive Actions We are reviewing our deployment and validation processes to prevent similar misconfigurations in future security rule updates.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

On Friday 22nd August 2025, around 2pm CEST the GraphQL API of Umbraco Heartcore returned some incorrect responses when querying either multiple variants of a single field. Only a small number of customers were affected. This was due to a regression inadvertently introduced when adding support for a minor feature - Using fragments to alias fields of different types to the same name. Following reports of the issue occuring, the following Tuesday we rolled forward with a hotfix that both corrected the regression and retained the feature behaviour.

Oct 11, 2025 - The licensing services have been updated with a built-in safeguard to handle invalid responses. Incident Summary – July 3, 2025 On July 3, 2025, between 1:15 PM and 1:35 PM UTC, a configuration change to the licensing service caused it to return invalid responses during license revalidation. Impact: During this period, affected products behaved as if unlicensed: - Commerce – Unable to process new orders - Engage – Stopped tracking, personalizing, and A/B testing - UI Builder, Workflow, Deploy – Minimal impact; related CMS features were unavailable Resolution: Engineering identified and resolved the misconfiguration at 1:35 PM UTC, shortly after customer reports surfaced. Functionality was restored without further intervention. We apologize for the disruption and are reviewing safeguards to prevent similar incidents.

# **Post-Mortem Report: Umbraco Cloud Outage – June 12, 2025** On **June 12, 2025**, Umbraco Cloud experienced a service disruption affecting several parts of our platform. We want to share a transparent overview of what happened, what caused it, and how we’re taking action to reduce the risk of similar incidents in the future. ## **What Happened** Between **20:08 and 23:18 CEST**, Umbraco Cloud Websites, Cloud & Heartcore services became unresponsive and returned **500 Internal Server Errors**. Affected systems included: * Umbraco Cloud Websites internal hostnames \(`{project}.{region}.umbraco.io`\) * Umbraco Cloud Portal * Heartcore REST and GraphQL APIs ## **Root Cause** Several Umbraco Cloud & Heartcore components depend on Cloudflare KV for normal operations, this upstream failure caused our services to return 500 errors. A failure in **Cloudflare Workers KV \(Key Value\) Store** - which we use for routing, configuration, and other operational needs - caused read/write operations to stop functioning. The root of this failure was in Cloudflare’s underlying storage infrastructure. As a result, key services in Umbraco Cloud could no longer complete requests and returned errors instead. ## **Trust in Cloudflare & Managing Risk** Cloudflare is a **key dependency** in our architecture, and we rely on their commitment to a **100% SLA-backed uptime**. In their [public incident report](https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/), Cloudflare acknowledged the architectural weakness that led to the outage and shared the work already underway to reduce such risks in the future. This includes strengthening their infrastructure and minimizing reliance on any single provider. We trust Cloudflare to follow through on these commitments and continue to invest in the resilience of their platform. ## **Steps Forward & Lessons Learned** This incident highlights the need to design systems that can tolerate failures—even in services considered highly reliable. ### **What We’re Doing** * We will evaluate alternative request routing strategies and graceful handling of errors in our Cloudflare deployments that would lessen the impact of Cloudflare KV outages in the future.  * Exploring whether critical configuration data can be mirrored * Exploring reducing the reliance on real-time KV access during its outage. * Investigating **graceful degradation mechanisms** to maintain partial service availability during similar events. ### **Lessons Learned** * Even highly available platforms can be impacted by upstream single points of failure. * Core functions like routing and API access should include fallback mechanisms where possible. * Our existing **monitoring and incident response processes** worked as intended, enabling us to detect, escalate, and track the incident effectively. ## **Final Thoughts** We understand that any disruption in service can be frustrating. We sincerely apologize for the inconvenience caused and thank you for your continued trust and patience. We're committed to improving the resilience and reliability of Umbraco Cloud and will continue to share updates as we strengthen our systems and processes. _— The Umbraco Cloud Team_

This incident has been resolved.

This incident has been resolved.

The incident has been resolved.

This incident has been resolved.

This incident has been resolved.

The Heartcore GraphQL API underwent a degraded quality of service for a period of approximately two hours, beginning at 13:00 on March 21st (UTC). During this time, some queries returned a 429: Too Many Requests HTTP response. The issue was organically resolved around 15:00 UTC. We determined the cause to be increased traffic volume to the service. Additional capacity has been provisioned to prevent the issue happening again.

This incident has been resolved. Please reach out to our support team if you were affected by this issue.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

We experienced a brief outage of the Cloud Portal for around 10 minutes due to an erroneous deployment. Our Cloud Team identified the cause and rolled back to the previous version.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

Umbraco Cloud hosting has not been affected by the CrowdStrike Falcon agent incident.

The incident has been resolved/.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

We've identified an incident from 07:30 AM to 8:30 AM GMT+1 . Heartcore APIs were unavailable during the period. We have investigated and resolved the issue.

The Heartcore APIs are continuing to operate as normal. This incident is now resolved.

Our supplier was able to identify the root cause of the issue and implement a fix. All Umbraco Cloud services should now function as normal. This issue has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

The root cause has been identified and a fix has been deployed and confirmed working. The root cause was due to an expired certificate for internal services used in the Umbraco Cloud Portal.

This incident has been resolved.

The incident has been resolved. Cloudflare's incident has been marked as resolved and we have not seen any further issues in the Cloud Portal.

On October 30, 2023, Umbraco Cloud experienced a service disruption due to an incident at our network service provider, Cloudflare. The incident began at 20:03 UTC, impacting the availability and functionality of various services. The Umbraco Cloud Portal was down during this period, and also a limited number of cloud projects. Timeline: - 20:03 UTC: Cloudflare reports an investigation into availability issues impacting several services. - 20:05 UTC: First update from Cloudflare, the investigation continues. - 20:23 UTC: Second update from Cloudflare, the investigation continues. - 20:33 UTC: The issue is identified, and a fix is being implemented by Cloudflare. - 20:34 UTC: A fix has been implemented, and monitoring begins to ensure stability. - 20:35 UTC: Cloudflare reports the incident as resolved. Impact: The incident affected the availability of the Umbraco Cloud Portal, hindering access to certain cloud projects. Authentication processes, as well as loading functionalities, were particularly impacted, causing inconvenience to our users. Resolution and Mitigation: Upon identifying the root cause, Cloudflare implemented a fix at 20:34 UTC, which restored the services to their normal functionality. Our engineering team remained in close contact with Cloudflare throughout the incident, ensuring that necessary steps were taken to mitigate the impact on Umbraco Cloud services. Alternative routing solutions were explored to minimize service disruption.

This incident has been resolved.

# Summary On Wednesday 26th of July, we had an incident impacting operations for adding new environments, changing plans, and creating projects, including trial flow, resulting in instances where customers were unable to do necessary operations on their projects. The affected component was a third-party vendor responsible for delivering a package feed to source additional applications that are used by Umbraco Cloud to run and manage environment websites. The incident was registered between 8:23 CET and resolved at 22:47 on the 26th of July # Root Cause Analysis Umbraco Cloud is using Myget to host a type of application called site extensions, which is a form of symbiotic component to an Umbraco Application running Umbraco Cloud. Pending on tier and product we utilize different site extensions to function as entry points for managing, and working with Umbraco under the hood, and is a required component on Umbraco Cloud.  Late on the 25th of July throughout the 26 and hours into the 27th Myget seized to respond to pulling Nuget Packages on Request \([https://status.myget.org/787400873](https://status.myget.org/787400873)\) and did so for around 40 hours.  MyGet was a known Single Point of Failure \(SPoF\), on Umbraco Cloud, and we have previously relied on stable uptime for that third-party provider. Internally the process for handling such an issue was in place, and we followed the playbooks for incident management. With the very limited knowledge regarding MyGet recovery, we initiated working on a fallback solution at 13:37CET and began testing it on our internal Development environments thereafter. At 16:09 CET we deployed our fallback solution to Production, where we saw an instant recovery of services.  At 17.35 We discovered that while our US and UK region was recovering from the incident gracefully, customers in the European region were still experiencing issues. At 20:28 The European region was recovering as expected and we changed from Actively fixing the issues to monitoring.  At 22:47 The Issue was resolved. **What happened to MyGet?** Unfortunately, we have yet to hear from Myget regarding the issue, and at this point in time, no efforts have been made on their behalf to ensure that this will be resolved moving forward. **Actions based on root cause analysis** First, we’ve already identified the provider as a SPoF. We have updated our risk assessment from negligible \(based on historical high uptimes\) to _critical_. We are actively working on solidifying the applied fallback solution, with the end goal of replacing Myget as a Nuget Package Provider in favor of multiple alternative fallback sources and ensuring a redundant Nuget Feed for Umbraco Cloud.  If you have any questions related to the above please feel free to contact your partner manager, reach out through our support channels or the Umbraco Cloud issue tracker on GitHub: [https://github.com/umbraco/Umbraco.Cloud.Issues/issues](https://github.com/umbraco/Umbraco.Cloud.Issues/issues)