[osc-fr1] A DB hosting node is misbehaving

» View Event Details | Created Tue, 01 Oct 2024 12:35:00 +0000

Resolved

All databases are now back up and running. If you still experience an issue with your database please contact our support.

Update

Almost all the databases have been restarted, we're dealing with the last ones that are causing problem.

Update

The impacted node has been restarted, database instances hosted on this machine are being restarted.

Investigating

Our monitoring system notified us that a database hosting node is misbehaving. Our operators are working on restarting it. We will keep you updated.

[osc-fr1] A Database hosting node is misbehaving

» View Event Details | Created Thu, 03 Oct 2024 00:49:00 +0000

Resolved

Incident is now considered as over, if you are still facing issues please contact our support team.

Update

Almost all the databases have been restarted, we're dealing with the last ones that are causing problem.

Update

Databases are being restarted.

Update

The decision was made to restart the VM: No impact for databases using a Business plan and hosted on this node Databases using a starter plan and hosted on this node will be down for the duration of the operation.

Investigating

Our monitoring system notified us that a database hosting node is misbehaving. We will keep you updated

[osc-fr1] Databases issue

» View Event Details | Created Tue, 08 Oct 2024 12:26:00 +0000

Resolved

All impacted databases have been fixed. If you still have an issue, please contact our support

Update

New databases are now booting correctly. We are fixing impacted databases.

Update

We found the root cause and are deploying a fix

Investigating

Our team has been notified of issues booting new databases, we are investigating

[osc-fr1] A DB node is misbehaving

» View Event Details | Created Wed, 09 Oct 2024 12:59:00 +0000

Resolved

The incident is considered as over on our side. If you are still facing issues, please contact our support.

Update

Databases hosted on the impacted node have been restarted, our operators are running post-incident checks.

Update

The node has been restarted and databases are being restarted. We will keep you updated.

Investigating

One of our database hosting node is misbehaving. Our team has been alerted.

[osc-fr1] Planned Maintenance on a Database hosting node

» View Event Details | Created Thu, 10 Oct 2024 18:00:00 +0000

Resolved

All operations has been done successfully. Maintenance is now complete. If you have any problems, please contact our support team.

Update

All databases have been restarted properly, we are now performing post-maintenance checks.

Update

The Database hosting node has been restarted properly, the databases concerned are being restarted.

In Progress

The maintenance has started on the Database hosting node. Databases will be stopped in a few minutes in order to restart the node.

Scheduled

A planned maintenance on a Database hosting node will occur in the osc-fr1 region on Thursday October 10, from 20:00 to 22:00 (CEST). Concerned customers were informed individually by email.

Business Plan Databases: No impact

Starter Plan Databases: Unavailable for the duration of the operation.

We will keep you informed when the operation will start through the status page.

[osc-fr1] Internet access to applications is degraded

» View Event Details | Created Tue, 15 Oct 2024 10:33:00 +0000

Resolved

All traffic is back to normal since 1:00pm CEST.

Update

The IP is working again as expected, we're observing the situation.

Investigating

One of our public IP is unresponsive. Our team is analyzing the situation.

The IP has been removed from our address pool.

Application with root domains having configured their DNS with A fields may experience a degraded access for their users.

[osc-fr1] A database hosting node is misbehaving

» View Event Details | Created Thu, 17 Oct 2024 12:50:00 +0000

Resolved

The incident is considered as over on our side. If you are still facing issues, please contact our support.

Update

Databases hosted on the impacted node have been restarted, our operators are running post-incident checks.

Update

The node has been restored and databases are being restarted. We will keep you updated.

Investigating

One of our database hosting node is misbehaving. Our team has been alerted.

[osc-fr1] Some databases are unavailable

» View Event Details | Created Fri, 18 Oct 2024 12:18:00 +0000

Resolved

All impacted databases are back online. If you are still facing issues, please contact our support.

Update

We're still working on the impacted databases

Investigating

We received reports about some databases being unavailable, we are investigating

[osc-fr1] A database hosting node is misbehaving

» View Event Details | Created Mon, 21 Oct 2024 16:15:00 +0000

Resolved

The incident is considered as over on our side. If you are still facing issues, please contact our support.

Update

Databases hosted on the impacted node have been restarted, our operators are running post-incident checks.

Update

The node has been restored and databases are being restarted. We will keep you updated.

Investigating

One of our database hosting node is misbehaving. Our team has been alerted.

[osc-fr1] A database hosting node is misbehaving

» View Event Details | Created Tue, 22 Oct 2024 15:15:00 +0000

Resolved

The situation has improved. The incident is considered as over on our side. If you are still facing issues, please contact our support.

Investigating

One of our database hosting node is misbehaving. Our team has been alerted.

[osc-fr1] Unexpected 502 errors when reaching apps

» View Event Details | Created Wed, 23 Oct 2024 12:25:00 +0000

Post-Mortem

The incident was triggered by a bug in a caching mechanism within a component upstream of our platform. This component, which is critical for directing HTTP requests to your applications, was not being properly supplied with the metadata needed to route the HTTP traffic to the appropriate containers. As a result, traffic may have been routed to containers that were no longer operational, either due to a redeployment or a modification in the topology of your applications, such as a scale down operation.

Although preexisting, this issue manifested as a side effect following an update to our infrastructure servers.

Our development teams have pinpointed the exact origin of the problem and a fix is currently being developed.

We have also initiated enhancements to the configuration of our monitoring probes to increase our ability to detect and respond more promptly to such incidents.

A comprehensive retrospective of this incident is also scheduled to explore additional potential improvements.

Resolved

The situation has been stable for 30 minutes. We are still investigating but the incident is considered closed.

If you are still experiencing issues, please contact our support.

Update

The 502 errors should have disappeared following our operators intervention.

Investigation is still ongoing in order to identify a root cause.

Investigating

We detected an unusual rate of 502 errors when reaching apps hosted on the osc-fr1 region.

Our operators are currently investigating the issue.

We'll keep you updated.