[osc-fr1] Database host mis-behaving

» View Event Details | Created Sat, 03 Dec 2022 00:12:00 +0000

Resolved

The situation is back to a normal state. If you still have issue with your database, please contact the support.

Posted: Sat, 03 Dec 2022 00:28:00 +0000

Update

The node is back up and running. Our team is running post incident checks.

Posted: Sat, 03 Dec 2022 00:22:00 +0000

Investigating

One of our database node is mis-behaving. Some databases in sandbox or starter plan might be unavailable while business plan might experience a failover.

Our team is currently investigating the issue.

Posted: Sat, 03 Dec 2022 00:12:00 +0000

[osc-fr1] Database instance is mis-behaving

» View Event Details | Created Tue, 06 Dec 2022 15:22:00 +0000

Resolved

All databases has been restarted. If you still have an issue, please contact us on the support.

Posted: Tue, 06 Dec 2022 15:42:00 +0000

Update

95% of the impacted databases are up. The last one should be up and running in a few minutes.

Posted: Tue, 06 Dec 2022 15:42:00 +0000

Update

Databases are restarting on the impacted node.

Posted: Tue, 06 Dec 2022 15:25:00 +0000

Investigating

One of our database instance is mis-behaving. Our team has been alerted.

Posted: Tue, 06 Dec 2022 15:22:00 +0000

[osc-fr1] Database node mis-behaving

» View Event Details | Created Wed, 07 Dec 2022 16:14:00 +0000

Resolved

All databases has been restarted. If you still have an issue, please contact us on the support.

Posted: Wed, 07 Dec 2022 16:54:00 +0000

Update

Databases are restarting on the impacted node.

Posted: Wed, 07 Dec 2022 16:24:00 +0000

Investigating

One of our database node is misbehaving. We are taking a lookg at it.

Posted: Wed, 07 Dec 2022 16:14:00 +0000

[osc-fr1] Database node mis-behaving

» View Event Details | Created Fri, 09 Dec 2022 01:57:00 +0000

Resolved

All databases has been restarted. If you still have an issue, please contact us on the support

Posted: Fri, 09 Dec 2022 03:48:00 +0000

Update

95% of the impacted databases are up. The last one should be up and running in a few minutes.

Posted: Fri, 09 Dec 2022 02:53:00 +0000

Update

75% of databases are up.

Posted: Fri, 09 Dec 2022 02:33:00 +0000

Update

Node is up. Databases are rebooting on the node.

Posted: Fri, 09 Dec 2022 02:03:00 +0000

Investigating

One of our database node is misbehaving. We are taking a lookg at it.

Posted: Fri, 09 Dec 2022 01:57:00 +0000

[osc-fr1][osc-secnum-fr1] Database gateways planned restart

» View Event Details | Created Mon, 12 Dec 2022 13:00:00 +0000

Resolved

This maintenance is over. Please reach the support if you face an issue to reach your database.

Posted: Wed, 14 Dec 2022 17:09:00 +0000

In Progress

The maintenance is starting.

Reminder about the impact on your database:

• Starter/Sandbox: ~10 seconds, the database won't be reachable the time to restart the gateway,
• Business: a connection may be cut during the floating IP failover between the two gateways.

Posted: Mon, 12 Dec 2022 14:04:00 +0000

Scheduled

TL;DR; All database gateways of the platform will be restarted to deploy a new version. It will be performed automatically on Monday the 12th of December.

Description

PostgreSQL, Elasticsearch, InfluxDB and Redis databases hosted on the platform are behind a gateway instance. These gateways have been updated to improve security, and all new databases created starting today are already using it. However, a restart is required for the existing gateways to benefit from these security improvements. They will be automatically restarted on Monday the 12th of December 2022.

Impact on Your Database

The impact on your database depends on the plan you use:

• Starter/Sandbox: ~10 seconds, the database won't be reachable the time to restart the gateway,
• Business: a connection may be cut during the floating IP failover between the two gateways.

Posted: Mon, 12 Dec 2022 13:00:00 +0000

[osc-fr1] Performance issues on databases nodes

» View Event Details | Created Tue, 13 Dec 2022 09:21:00 +0000

Resolved

Databases performance has been stable for 12 hours, we are closing the incident.

Posted: Thu, 15 Dec 2022 13:45:00 +0000

Update

The performances of our volumes is getting better. The impacted databases should recover their full performance in the coming minutes.

Posted: Tue, 13 Dec 2022 15:40:00 +0000

Update

Our infrastructure provider is migrating our volumes to a less loaded storage bay in order to improve the performance we get out of our volumes. We will keep you informed of the evolution of the situation.

Posted: Tue, 13 Dec 2022 13:37:00 +0000

Update

Our team is still in contact with our infrastructure about these performance issues. Some databases are completely unreachable due to these performance issues. We are working hard to improve the situation.

Posted: Tue, 13 Dec 2022 12:11:00 +0000

Update

We are in contact with our infrastructure provider who acknowledged the issue and is currently working on it.

Posted: Tue, 13 Dec 2022 10:30:00 +0000

Update

Only 2 nodes are impacted. A failover will be triggered on business databases that are on the impacted nodes to move the workload on non-impacted nodes.

Posted: Tue, 13 Dec 2022 09:31:00 +0000

Investigating

Our probes returns a high disk latency on some database hosting nodes. This will impact performances of databases.

Our team is currently looking at it.

Posted: Tue, 13 Dec 2022 09:21:00 +0000

[osc-fr1][osc-secnum-fr1] Scalingo authentication service is currently down

» View Event Details | Created Tue, 13 Dec 2022 10:58:00 +0000

Resolved

The dashboard and the CLI are usable again.

This incident was a consequence of https://scalingostatus.com/details/6c871d6f7619c. The follow up will be written in the root cause incident.

We are considering this incident as over. Please reach our support if you still face issues to connect to the dashboard.

Posted: Tue, 13 Dec 2022 11:34:00 +0000

Update

The authentication service is recovering, the dashboard should be available soon.

Posted: Tue, 13 Dec 2022 11:24:00 +0000

Investigating

We have been alerted that the Scalingo authentication service is currently down. The dashboard is currently unavailable and using the CLI does not work.

Posted: Tue, 13 Dec 2022 10:58:00 +0000

[osc-fr1] Network degradation on osc-fr1

» View Event Details | Created Thu, 15 Dec 2022 09:39:00 +0000

Resolved

Network access from/to the platform in region osc-fr1 has been more stable for ~1h. Our team, with the help of our infrastructure provider, keep analyzing the source of this incident.

Posted: Thu, 15 Dec 2022 13:48:00 +0000

Update

→ End of the emergency maintenance of the status page provider, keeping on incident updates in this post.

TL;DR Situation is more stable but not perfect, our team is still working on the incident.

Our provider has achieved a maintenance operation which lead to a complete networking cut of the 'osc-fr1' from 11:25 to 11:32 CET

From that point, network quality got better, but our monitoring tool has continued detecting packet loss from and to the infrastructure, leading to potential connectivity problem from/to the applications.

Deployments are working as expected again, older deployments may have been aborted, you're invited to trigger a new one to update your applications.

Posted: Thu, 15 Dec 2022 11:48:00 +0000

Update

The issue has been escalated to our network provider. Their team is also working on it.

Posted: Thu, 15 Dec 2022 09:57:00 +0000

Investigating

Network performance is degraded on the osc-fr1.

This impacts performances access to all applications hosted on that region.

This also impacts deployment performances.

Posted: Thu, 15 Dec 2022 09:39:00 +0000

[osc-fr1][osc-secnum-fr1] Database gateways planned restart

» View Event Details | Created Fri, 23 Dec 2022 08:00:00 +0000

Resolved

This maintenance is over. Please reach the support if you face an issue to reach your database.

Posted: Fri, 23 Dec 2022 17:14:00 +0000

Update

The maintenance is starting. Reminder about the impact on your database:

• Starter/Sandbox: the database won't be reachable for around 10 seconds,
• Business: a connection may be lost due to high availability mechanisms that are part of the business plan.

Posted: Fri, 23 Dec 2022 08:45:00 +0000

Scheduled

TL;DR; All database gateways of the platform will be restarted to deploy a new version. It will be performed automatically on Friday the 23rd of December.

Description

PostgreSQL, Elasticsearch, InfluxDB, and Redis databases hosted on the platform are behind a gateway instance. These gateways have been updated to improve security, and all new databases created starting today are already using it. However, a restart is required for the existing gateways to benefit from these security improvements. They will be automatically restarted on Friday the 23rd of December 2022.

Impact on Your Database

The impact on your database depends on the plan you use:

• Starter/Sandbox: the database won't be reachable for around 10 seconds,
• Business: a connection may be lost due to high availability mechanisms that are part of the business plan.

Posted: Fri, 23 Dec 2022 08:00:00 +0000

[osc-fr1] Degraded network performances

» View Event Details | Created Tue, 27 Dec 2022 09:16:00 +0000

Resolved

We consider this incident over. If you still face network degradation, please contact our support team.

Our team is still in touch with the infrastructure provider to identify the root cause of this issue.

Posted: Tue, 27 Dec 2022 16:02:00 +0000

Update

We observe an improvement in network performances. Our team keeps monitoring and investigating the root cause of the issue.

Posted: Tue, 27 Dec 2022 09:49:00 +0000

Investigating

It seems that our network is degraded. Our engineers are taking a look at the issue.

Posted: Tue, 27 Dec 2022 09:16:00 +0000