kinsta.cloud domain name disruption
Incident Report for Kinsta
Postmortem

On Saturday, May 21, 2022, we experienced a service disruption that impacted a subset of customer live websites as well as multiple areas of our hosting product. In this postmortem, we will recap the incident and explain what we are doing in response.

Incident Recap

At 09:29 UTC, the domain registration status on the kinsta.cloud domain was updated by the .cloud registry to include a serverHold status. When this occurred, the domain was dropped from root DNS servers across the internet. This domain registration update resulted in potential downtime for customer live sites with domains pointed to Kinsta via a CNAME record. Additionally, all temporary URLs and phpMyAdmin URLs were unreachable since these are based on the kinsta.cloud domain name.

Once the source of the issue was confirmed, our team began multiple plans of attack over the course of the next ten hours.

  • We initiated and held conversations with our domain registrar to learn more about why the action was taken and what was needed from us in order to resolve the issue.
  • We initiated conversations with the .cloud registry and ICANN to learn more about why the action was taken and what was needed from us in order to resolve the issue.
  • As the kinsta.cloud domain's nameservers are pointed to Cloudflare, we initiated conversations with them to ensure the zone file itself would not be removed from their platform through their automated processes. Cloudflare also assisted us with a direct line of communication to the executive team at our registrar to help facilitate resolution.
  • We began discussing the feasibility and possibility of numerous temporary resolution options internally that would result in bringing as many live sites back online as possible.
  • We began identifying and testing ways customers could bring their live sites back online through ways within their control.

We were able to identify steps customers could take to bring their live sites back online and communicated these options to customers via open chats. As we were preparing to update our status page with these steps, our registrar informed us at 21:42 UTC that they had removed the serverHold status from the domain and within minutes all affected live sites, temporary URLs, and phpMyAdmin URLs were back online.

What We're Doing In Response

This incident has resulted in two action items.

The first action item involves our responsibility and selection of domain registrar. There are a wide variety of accredited registrars available for domain owners, and we will be transferring our domain registrations to a registrar that specializes in the security and protection needs of the enterprise market. We will begin this process by transferring the kinsta.cloud domain first with the rest of our company domains to follow.

The second action item we have outlined is to engage in communication with the .cloud registry to learn what we can do to ensure notifications occur before similar actions might be taken in the future. This has already proved to be valuable as the .cloud registry has placed kinsta.cloud on their allowlist to help prevent potential future recurrences.

We Never Stop Learning

Kinsta exists to serve our customers, and we deeply regret that a subset of customer live sites along with all staging sites and database access via phpMyAdmin were affected by this situation. We are committed to learning and growing as a team with the aim of living up to the trust you continue to place in us.

Posted May 27, 2022 - 19:53 UTC

Resolved
We are no longer receiving reports related to this DNS issue at this time. Service has been fully restored. If you do continue to have any difficulty however, please do not hesitate to reach out to our Support team. Thank you for your patience and we do apologize for any disruption to your business.
Posted May 21, 2022 - 21:28 UTC
Monitoring
At this time, our Engineers have worked with the domain registrar to find a solution. We are seeing domains resolving properly again. We are continuing to monitor and ensure this is no longer an issue. If you do continue to have any difficulty however, please do not hesitate to reach out to our Support team.
Posted May 21, 2022 - 20:10 UTC
Update
We are continuing to work on a fix for this issue. We appreciate your continued patience while we work with the domain registrar to resolve the issue.
Posted May 21, 2022 - 19:40 UTC
Update
We are continuing to work with the domain registrar to resolve this issue. Thank you again for your continued patience while we work towards a solution.
Posted May 21, 2022 - 17:54 UTC
Update
We are continuing to work on a fix for this issue. We appreciate your continued patience while we work towards a solution.
Posted May 21, 2022 - 16:33 UTC
Update
We are continuing to work on a fix for this issue.
Posted May 21, 2022 - 15:13 UTC
Update
We continue working with Domain registrar to resolve this issue. At the moment the issue seems to affect all staging sites, and 3-5% of live sites on our platform.
Posted May 21, 2022 - 13:35 UTC
Update
We are continuing to work on a fix for this issue.
Posted May 21, 2022 - 12:05 UTC
Update
We are continuing to work on a fix for this issue.
Posted May 21, 2022 - 10:48 UTC
Update
While we are seeing reports of some domains starting to propagate and resolve, we continue to work on a fix for the underlying issue.
Posted May 21, 2022 - 09:43 UTC
Identified
The issue has been identified as being related to an issue with our parent domain registrar, we are currently working with our domain Registrar to resolve the issue as quickly as possible.
Posted May 21, 2022 - 09:06 UTC
Investigating
We are investigating reports of various domains associated with our Cloudflare integration experiencing inconsistent DNS resolution, which has led to these domains not loading their associated sites on our network.
Posted May 21, 2022 - 08:42 UTC
This incident affected: Kinsta DNS and Cloudflare Integration.