[osc-secnum-fr1] A database hosting node is misbehaving

» View Event Details | Created Thu, 06 Mar 2025 10:18:00 +0000

Resolved

The incident is now considered as over.

Posted: Thu, 06 Mar 2025 17:51:00 +0000

In Progress

We are still making operations on this server. Restart of databases is planned in a few minutes

Posted: Thu, 06 Mar 2025 17:30:00 +0000

Update

The node in question is still causing issues. We have identified the root cause and are currently working on it. This will require restarting the affected VM. Customers with a Business plan will not be impacted, while customers with a Starter plan will experience a few minutes of downtime

Posted: Thu, 06 Mar 2025 17:02:00 +0000

Resolved

All checks are OK. The incident is now considered as over on our side.

Posted: Thu, 06 Mar 2025 11:20:00 +0000

Update

All databases are up and running. Our operators are entering post incidents checks. If you still have issues with your database please contact us via the support.

Posted: Thu, 06 Mar 2025 11:16:00 +0000

In Progress

Databases are currently being restarted on the impacted database hosting node.

Posted: Thu, 06 Mar 2025 10:49:00 +0000

In Progress

The database hosting node has been restarted. Databases will restart soon on the impacted hosting node.

Posted: Thu, 06 Mar 2025 10:29:00 +0000

In Progress

We are restarting the impacted database hosting node.

Posted: Thu, 06 Mar 2025 10:22:00 +0000

Investigating

One of our database hosting node is misbehaving. Our team has been alerted.

Posted: Thu, 06 Mar 2025 10:18:00 +0000

[osc-fr1] Network access to the region is interrupted

» View Event Details | Created Tue, 11 Mar 2025 10:21:00 +0000

Post-Mortem

You can find the postmortem for this incident at the following link: https://doc.scalingo.com/security/postmortems/spm-2025-001 For any further questions, please don’t hesitate to contact our support team.

Posted: Mon, 31 Mar 2025 13:20:00 +0000

Update

The situation remains stable, and network access to the platform is at a nominal level. We take this event very seriously and would like to share additional information at this time.

The root cause has been identified as a DDoS attack on the osc-fr1 region. Our teams have implemented measures to mitigate the impact and improve detection for potential new occurrences. our operators continue to closely monitor the environment and stand ready to respond if another wave of attacks emerges.

Our primary focus remains maintaining platform stability. In the coming days, we will provide a more detailed communication that will retrace a chronology of the events and present the work to perform on short and long term in order to improve the platform experience for all of our customers.

In the meantime, our teams are fully mobilized to address these issues. If you have further questions, you can contact our support team.

Posted: Tue, 11 Mar 2025 17:41:00 +0000

Update

The situation is stable for 45 minutes.

Our operators are still working on it.

Posted: Tue, 11 Mar 2025 16:16:00 +0000

Update

A new wave of perturbation is ongoing impacting the network access,

our operators are still working on theses issues

Posted: Tue, 11 Mar 2025 15:32:00 +0000

Resolved

The situation has been stable for a while, We are closing the incident

Our operators are still monitoring the situation

Posted: Tue, 11 Mar 2025 14:23:00 +0000

Update

The situation improved,

Our operators are still monitoring the situation

Posted: Tue, 11 Mar 2025 13:33:00 +0000

Update

A new occurrence of the network interruption is currently ongoing.

Our operators are working on it.

Posted: Tue, 11 Mar 2025 13:23:00 +0000

Resolved

The situation is now stable for more than 30 minutes. The incident is considered as over on our side

Our operators are still analyzing what happened. We will update this event with relevant information once we know more about it.

If you are still facing issues, please contact our support.

Posted: Tue, 11 Mar 2025 11:29:00 +0000

Update

The traffic is back up but our operators keep monitoring the situation.

We will keep you updated.

Posted: Tue, 11 Mar 2025 10:59:00 +0000

Update

The network is now degraded again and there is a new interruption ongoing.

Our operators are fully mobilized on the investigation.

Posted: Tue, 11 Mar 2025 10:40:00 +0000

Update

The traffic towards the region has been restablished.

Our operators are still investigating what happened. We also are in contact with our infrastructure provider to get more details about this situation.

Posted: Tue, 11 Mar 2025 10:34:00 +0000

Investigating

A critical incident is currently occurring. The network access to the platform is interrupted.

Our operators are currently investigating the situation. You'll be notified when we know more about the situation.

Posted: Tue, 11 Mar 2025 10:21:00 +0000

[osc-secnum-fr1] Deployment and scale up operation issue

» View Event Details | Created Fri, 14 Mar 2025 20:10:00 +0000

Resolved

The situation is stable for more than 10 minutes. We are closing the incident on our side. If you are still facing issues, please contact our support.

Posted: Fri, 14 Mar 2025 20:44:00 +0000

Investigating

Scale-up and deployment operations are potentially slower than usual. We are currently investigating to understand the exact origin of the problem.

Posted: Fri, 14 Mar 2025 20:10:00 +0000

[osc-secnum-fr1] Deployment and scale up operation issue

» View Event Details | Created Fri, 14 Mar 2025 23:31:00 +0000

Resolved

The incident has been fixed by our object storage provider. The situation is back to nominal state for a few hours now. We are closing the incident on our side. Please reach our support if you are still facing issues.

Posted: Sat, 15 Mar 2025 11:34:00 +0000

Update

The issue has been identified and acknowledged by our object storage provider, and their teams are currently working on a fix.

Posted: Sat, 15 Mar 2025 00:09:00 +0000

Investigating

Scale-up and deployment operations are slower than usual. We are currently investigating to understand the exact origin of the problem.

Posted: Fri, 14 Mar 2025 23:31:00 +0000

[osc-fr1] Issue with deployments

» View Event Details | Created Wed, 19 Mar 2025 09:28:00 +0000

Resolved

We are back to normal on deployments. We are closing the incident. If you still have issue with deployment please reach us through the support

Posted: Wed, 19 Mar 2025 11:23:00 +0000

In Progress

Deployments are again being scheduled properly. It will take some time to catch up with the backlog in the deployment queue

Posted: Wed, 19 Mar 2025 10:47:00 +0000

In Progress

We still experiencing some issues with deployment scheduling, we are on it

Posted: Wed, 19 Mar 2025 10:15:00 +0000

In Progress

We released a fix. Deployments are being progressively processed

Posted: Wed, 19 Mar 2025 09:55:00 +0000

In Progress

We have identified the root cause and are working on a fix

Posted: Wed, 19 Mar 2025 09:48:00 +0000

Update

It also impact our API on osc-fr1. We are still investigating

Posted: Wed, 19 Mar 2025 09:36:00 +0000

Investigating

Our team has been notified of an issue with deployments, we are investigating

Posted: Wed, 19 Mar 2025 09:28:00 +0000

[osc-fr1] Issue with logs ingestion

» View Event Details | Created Wed, 19 Mar 2025 12:45:00 +0000

Resolved

Logs ingestion is now back on track. Closing the incident. Please reach our support if you are still facing an issue with logs

Posted: Wed, 19 Mar 2025 13:36:00 +0000

In Progress

Logs ingestion is actually working again. We're handling the queue on logs ingestion

Posted: Wed, 19 Mar 2025 13:15:00 +0000

Investigating

Some applications may be seeing a degradation in logs on osc-fr1, our team is working on the issue

Posted: Wed, 19 Mar 2025 12:45:00 +0000

[osc-fr1] Issue with deployments

» View Event Details | Created Wed, 19 Mar 2025 15:17:00 +0000

Resolved

The situation is now stable, we are closing this incident. If you still face issues, please contact our support.

Posted: Wed, 19 Mar 2025 16:11:00 +0000

Update

Deployments are correctly executed but can face some delays. We are working with our infrastructure provider to improve the situation

Posted: Wed, 19 Mar 2025 15:54:00 +0000

Investigating

Our team has been notified of an issue with some deployments, we are investigating

Posted: Wed, 19 Mar 2025 15:17:00 +0000

[osc-fr1] Short Network Disruption - Resolved

» View Event Details | Created Wed, 19 Mar 2025 15:55:00 +0000

Update

This outage was caused by maintenance performed ahead of schedule by our infrastructure provider. We are in contact with them to understand what measures will be implemented to prevent a similar incident from happening again.

Posted: Fri, 21 Mar 2025 15:45:00 +0000

Resolved

We experienced a brief network disruption that lasted approximately 2 minutes, from 16:54 to 16:56 (CET). Services were temporarily impacted during this period. We have opened a ticket with our infrastructure provider to investigate the root cause. If you are still experiencing issues, please contact our support team.

Posted: Wed, 19 Mar 2025 15:55:00 +0000

[osc-fr1] Application Node misbehaving

» View Event Details | Created Sat, 22 Mar 2025 21:40:00 +0000

Resolved

All application containers have been restarted

Posted: Sat, 22 Mar 2025 21:47:00 +0000

Investigating

A server hosting application has been misbehaving, some application may have encountered a few connections errors, applications are being restarted.

Posted: Sat, 22 Mar 2025 21:40:00 +0000