» View Event Details | Created
Resolved
A fix has been applied and new database operations should succeed.
if you're still facing any issue, please contact our customer support
Investigating
An issue preventing addon provisionning and plan changes from succeeding on the osc-fr1 region.
A workaround is being deployed by our operators that are aware of the issue.
» View Event Details | Created
Resolved
The performances are back to the nominal state. We are closing this incident.
Update
Our infrastructure provider deployed a fix. We are currently monitoring the situation.
Investigating
Deployments are currently taking longer than usual to complete due to increased latency affecting a service from our infrastructure provider: https://status.outscale.com/incidents/c6mdlqtkwsj0. We are monitoring the situation and will provide updates as they become available.
» View Event Details | Created
Resolved
All affected databases have been restarted. If you are still facing issue, please contact our support team.
Update
Most databases are restarted. The remaining ones are being handled.
Update
Affected databases are currently restarting.
Investigating
A database node is currently mis-behaving. We are proceeding with a restart in order to restore service. Starter plans hosted on this node are temporarily unreachable. Customers with a Business plan are not impacted by this incident. We will keep you informed as the situation evolves.
» View Event Details | Created
Post-Mortem
During the evening of June 24, starting at approximately 21:30, Scalingo experienced significant disruptions affecting both the osc-secnum-fr1 and osc-fr1 regions.
The incident occurred during an extreme heatwave affecting the area where one of our infrastructure provider's datacenters is located. Under these conditions, cooling systems at the datacenter began to fail, resulting in severe overheating and impacting some of the provider's infrastructure.
Although Scalingo's infrastructure is not directly hosted in the affected datacenter, the incident had cascading effects on some services we rely on:
osc-secnum-fr1: A critical network component operated by our infrastructure provider was hosted in the affected datacenter, resulting in network unavailability towards the region. Connectivity was restored at approximately 23:10, after which applications and databases in the region became reachable again. We observed no additional impact resulting from the incident after connectivity was restored.
osc-fr1: Although object storage data is replicated across multiple sites, we identified that our provider's API used to access the service was hosted exclusively in the affected zone. This resulted in minor disruptions to operations relying on object storage, including some deployment operations.
Resolved
Situation is stable again, closing the incident. Please contact our support for any question.
Update
Access to the osc-secnum-fr1 region is available again, we're monitoring the situation
Update
The following information has been communicated to us: only network interconnections are impacted in the region osc-secnum-fr1, data storage and workloads are unaffected.
We are waiting for the network incident resolution to get back the access to the applications.
Update
The incident is still ongoing, the problem appears to be wider and coming from Equinix PA4.
We don't have an ETA at the moment, waiting from information from the chain of providers.
Update
The incident is caused by a network disruption of our infrastructure provider, an incident is opened on their side, we are in contact with them.
Investigating
Internet access to the region osc-secnum-fr1 is currently unavailable. Applications hosted on the region are not reachable.
Our team is investigating.
» View Event Details | Created
Resolved
A workaround has been set up and the region is back to nominal performances.
The incident is considered as closed on our side.
Update
Our operators are still monitoring the situation. We observed an improvement on the network access to the region.
The deployments and container operations are still in a degraded mode but should be working as expected soon after the deployment of a workaround on our infrastructure provider side.
Update
The situation is improving. Databases are accessible again and most containers operations are working again. Our team is monitoring the situation.
Update
App containers operations (restart, scaling, deployments, etc) are currently unavailable due to an unavailability on the storage backend. Our operators are currently investigating and monitoring the situation.
Investigating
Access to Scalingo hosted databases from inside and outside the Scalingo network can be disrupted. Connections may fail with a No route to host error. Our team is investigating.
» View Event Details | Created
Resolved
The situation is back to the nominal state. Deployments are no longer delayed, and execution times have returned to their standard levels. The disruption was caused by increased latency affecting the object storage service provided by our infrastructure provider.
Update
The situation is improving, delayed deployments are being processed, we're still monitoring the situation
Investigating
Deployments are currently taking longer than usual to complete. Our team is investigating the issue
» View Event Details | Created
Post-Mortem
The disruption was caused by increased load on the object storage service of our infrastructure provider. They have implemented mitigation measures to help prevent similar incidents from recurring.
Resolved
Deployments and application operations are working again since 20 minutes. We are considering this issue as over. If you are still facing issues, please contact our Support team.
Update
The situation has improved since 21:56 CEST. We are still investigating the issue.
Investigating
We have identified an issue with a component handling part of the operations related to applications. You may experience issues with deployments and scaling operations. Our team is working on a fix.
» View Event Details | Created
Resolved
The situation is back to the nominal state. Deployments are no longer delayed, and execution times have returned to their standard levels. The disruption was caused by increased latency affecting the object storage service provided by our infrastructure provider.
In Progress
A workaround has been set. If your deployment has been aborted, please retry a deployment.
Investigating
Deployments are currently taking longer than usual to complete. Our team is investigating the issue.
