» Published on Wed, 11 Mar 2020 11:22:00 +0000

• Resolved

  All databases are back online. Please contact the support if you are still experiencing issues.

  » Updated Wed, 11 Mar 2020 15:15:00 +0000

• Update

  Most databases restarted. We are handling the edge case. Some databases didn't successfully restarted and we are handling them manually.

  » Updated Wed, 11 Mar 2020 13:35:00 +0000

• Update

  75% of the databases restarted.

  » Updated Wed, 11 Mar 2020 13:10:00 +0000

• Update

  We actually suffered three concomitant incidents today:

  1. the impacted node was actually lost. We had to provision a new node and restore the databases on it.
  2. peering issue which may have impacted you to reach our services. This incident also impacted the node provisioning as the access to rubygems.org was impacted and we had a hard time installing our dependencies.
  3. a certificate expiration issue for our etcd impacted PostgreSQL clusters with about 10 minutes of downtime.

  We are still in the process of restarting the databases.

  » Updated Wed, 11 Mar 2020 12:55:00 +0000

• Update

  15% of the databases restarted.

  » Updated Wed, 11 Mar 2020 12:50:00 +0000

• Update

  The node successfully restarted, we are restarting the databases.

  » Updated Wed, 11 Mar 2020 12:48:00 +0000

• Update

  The node restart should be over in a few minutes.

  » Updated Wed, 11 Mar 2020 12:32:00 +0000

• Update

  The network seems to be stable again, our API and authentication service are back online.

  » Updated Wed, 11 Mar 2020 12:24:00 +0000

• Update

  It seems the network is also blinking which leads to our API and authentication service being hard to reach. Our team is still working hard on fixing the solution.

  » Updated Wed, 11 Mar 2020 12:05:00 +0000

• Update

  The restart of the node is being longer than expected, we're still on it.

  Your database data is safe, the issue lies on the node restart.

  » Updated Wed, 11 Mar 2020 11:47:00 +0000

• Update

  We are restarting the node.

  Multi-nodes databases will automatically failover. The automatic failover procedure may induce a short downtime (a few seconds).

  Single-node databases hosted on it will automatically be restarted after the successful restart of the node.

  » Updated Wed, 11 Mar 2020 11:25:00 +0000

• Investigating

  A node hosting databases is behaving weirdly. You may experience trouble accessing your database.

  » Updated Wed, 11 Mar 2020 11:22:00 +0000