[global] Deployments with GitHub are failing

» View Event Details | Created Tue, 09 Jan 2024 13:22:00 +0000

Resolved

Deployments with GitHub now work properly again.

Posted: Tue, 09 Jan 2024 14:31:00 +0000

Update

Some deployments are currently proceeding correctly, but the situation is still unstable. GitHub incident still ongoing

Posted: Tue, 09 Jan 2024 13:50:00 +0000

In Progress

Deployments through GitHub integration are currently failing. An incident is on-going on the GitHub side: https://www.githubstatus.com/incidents/pxg3dz4yg7lp

If you need to deploy urgently, you still could do through the git push method: https://doc.scalingo.com/platform/deployment/deploy-with-git

Posted: Tue, 09 Jan 2024 13:22:00 +0000

[osc-fr1] Creation and updating of databases is not possible

» View Event Details | Created Fri, 12 Jan 2024 13:38:00 +0000

Resolved

All impacted databases have been processed. Incident is now considered as over, if you are still encountering issues please contact our support.

Posted: Fri, 12 Jan 2024 16:43:00 +0000

Update

The root cause has been identified, a temporary mitigation has been implemented. The impacted databases are now being processed.

Posted: Fri, 12 Jan 2024 15:56:00 +0000

Update

Our operators are still investigating the issue. We will keep you updated

Posted: Fri, 12 Jan 2024 15:34:00 +0000

Investigating

An incident is currently in progress on the osc-fr1 region, making it impossible to create, update or change plans on databases. All types of database are impacted. Our operators are currently investigating the root cause.

Posted: Fri, 12 Jan 2024 13:38:00 +0000

[osc-fr1] Deployments are slow

» View Event Details | Created Mon, 15 Jan 2024 10:30:00 +0000

Resolved

The deployments are working for more than one hour

The incident is now considered as over

Posted: Mon, 15 Jan 2024 13:22:00 +0000

Update

A workaround has been applied on this region. New deployments are working as expected. You can now trigger new deployments and the previous ones will be interrupted.

Our operators are still working on the root cause

Posted: Mon, 15 Jan 2024 11:40:00 +0000

Investigating

Our team is investigating an issue with Deployments on osc-fr1. Deployments are currently slow on this region.

Posted: Mon, 15 Jan 2024 10:30:00 +0000

[osc-fr1] Problems reaching google API

» View Event Details | Created Mon, 15 Jan 2024 16:58:00 +0000

Update

Our operators have setup a workaround for this issue. The situation should be back to normal.

If you still have issues contacting Google Cloud Platform Services, please contact us on the support.

Posted: Tue, 16 Jan 2024 15:50:00 +0000

Update

The issue has been acknowledged on Google side and their teams are investigating. We will keep you informed of any improvement.

Posted: Tue, 16 Jan 2024 08:33:00 +0000

Investigating

Some applications using the Google services API may experience HTTP 403 issues. We have opened a ticket on the Google support to report that requests from the osc-fr1 region are affected by this problem. We have not yet received any response. We are currently considering temporary workarounds to address the problem.

In the meantime, we also invite you to contact Google Support to report the problem on the service that is impacting you. The public IP address for outgoing traffic for the osc-fr1 region is 171.33.105.206

Posted: Mon, 15 Jan 2024 16:58:00 +0000

[osc-fr1] Access to applications degraded

» View Event Details | Created Tue, 16 Jan 2024 20:22:00 +0000

Resolved

IP has been added back to our Ingress IPs pool. Situation is nominal again.

Root cause was related to an hardware crash from our server provider.

Posted: Tue, 16 Jan 2024 20:43:00 +0000

Update

IP is available again, situation should get stable again, we're monitoring the situation.

Posted: Tue, 16 Jan 2024 20:42:00 +0000

Investigating

One of our ingress IP "5.104.101.30" seems to be unreachable. HTTP/HTTPS requests are timing out.

Access to applications may be degraded. The IP has been removed from our IP pools, all custom domains configured with a CNAME should work as expected.

Our team is working on it.

Posted: Tue, 16 Jan 2024 20:22:00 +0000

[osc-fr1] Network access is unstable

» View Event Details | Created Thu, 18 Jan 2024 11:10:00 +0000

Post-Mortem

A postmortem on this incident is available. It explains the reasons behind the network issues on the osc-fr1 region. If you have any questions, don’t hesitate to contact our support team. https://doc.scalingo.com/security/postmortems/spm-2024-001

Posted: Fri, 02 Feb 2024 14:37:00 +0000

Resolved

The situation is stable.

If you still have issues please contact us via the support.

Posted: Thu, 18 Jan 2024 16:17:00 +0000

Update

The situation is mostly stable. We are continuing to monitor the situation.

Posted: Thu, 18 Jan 2024 15:14:00 +0000

Update

The situation is continuing to improve. The platform is close to its nominal state. The platform is now mostly behaving correctly.

We are continuing to monitor the situation.

Posted: Thu, 18 Jan 2024 14:43:00 +0000

Update

A first fix has been deployed on our network provider side. Our monitoring confirm that the situation is indeed improving.

Posted: Thu, 18 Jan 2024 14:26:00 +0000

Update

The incident is now also available on our network provider status page: https://status.outscale.com/incidents/vc6t6y79jhdh (French)

Posted: Thu, 18 Jan 2024 13:42:00 +0000

Update

Investigation is still in progress. We will keep you updated

Posted: Thu, 18 Jan 2024 13:10:00 +0000

Update

> We received an update from our network provider. Multiple customers are impacted on their side. This issue is their top priority.

Posted: Thu, 18 Jan 2024 12:43:00 +0000

Update

The issue is being escalated on our network provider side. Our team is continuing to investigate.

Posted: Thu, 18 Jan 2024 12:30:00 +0000

Update

Investigation is still ongoing at the moment. We will keep you updated

Posted: Thu, 18 Jan 2024 12:08:00 +0000

Update

We contacted our infrastructure provider, their engineers are currently working on identifying and fixing the issue

Posted: Thu, 18 Jan 2024 11:54:00 +0000

Update

Our operators are still investigating the network degradation issue, we'll continue to share updates as more information is gathered

Posted: Thu, 18 Jan 2024 11:33:00 +0000

Investigating

Our team is investigating an issue with the network access to hosted applications. We will update this ticket as soon as we have more information.

Posted: Thu, 18 Jan 2024 11:10:00 +0000

[osc-fr1] Database node is mis-behaving

» View Event Details | Created Thu, 25 Jan 2024 15:45:00 +0000

Resolved

All databases are back up and running. If you still have issues please contact our support.

Posted: Thu, 25 Jan 2024 18:51:00 +0000

Update

The node has restarted and the databases are being rebooted.

Posted: Thu, 25 Jan 2024 17:38:00 +0000

Update

The first operation wasn't successful and we need to restart the node again.

Posted: Thu, 25 Jan 2024 17:36:00 +0000

Resolved

All databases are back up and running. If you still have issues please contact our support.

Posted: Thu, 25 Jan 2024 16:27:00 +0000

Update

Databases are restarting on the impacted node.

Posted: Thu, 25 Jan 2024 16:03:00 +0000

Investigating

One of our database host is mis-behaving. Our team has been alerted and the host is being restarted.

Posted: Thu, 25 Jan 2024 15:45:00 +0000

[osc-fr1] Databases not accessible

» View Event Details | Created Mon, 29 Jan 2024 19:15:00 +0000

Post-Mortem

A postmortem on this incident is available. It explains why the databases were unavailable on the osc-fr1 region. If you have any questions, don’t hesitate to contact our support team. https://doc.scalingo.com/security/postmortems/spm-2024-002

Posted: Wed, 31 Jan 2024 13:20:00 +0000

Update

Our engineers are currently preparing a Postmortem.

Posted: Tue, 30 Jan 2024 14:23:00 +0000

Resolved

We now consider the incident as resolved, we are still monitoring the situation to make sure the platform is stable.

Posted: Tue, 30 Jan 2024 00:47:00 +0000

Update

If you are still experiencing problems accessing your databases, please contact our support team.

Posted: Mon, 29 Jan 2024 23:50:00 +0000

Update

Most databases are back online. We are in the process of fixing the last databases that could cause problems.

Posted: Mon, 29 Jan 2024 23:35:00 +0000

Update

We have successfully tested the solution on a small subset of databases. We are now running the procedure for all the databases affected by the incident.

Posted: Mon, 29 Jan 2024 23:02:00 +0000

Update

A solution is currently being worked out by our engineers.

Posted: Mon, 29 Jan 2024 22:34:00 +0000

Update

We have identified a network configuration issue, given the incident's scope, Starter and Business plans are affected.

Posted: Mon, 29 Jan 2024 21:34:00 +0000

Update

The underlying issue has been identified, our team is actively working to restore normal operations.

Posted: Mon, 29 Jan 2024 20:39:00 +0000

Update

The incident is still in progress, and access to your databases may still be affected.

Posted: Mon, 29 Jan 2024 20:00:00 +0000

Update

Databases are restarting on the impacted node.

Posted: Mon, 29 Jan 2024 19:25:00 +0000

Investigating

One of our database host is mis-behaving. Our team has been alerted and the host is being restarted.

Posted: Mon, 29 Jan 2024 19:15:00 +0000

[osc-fr1] Network degradation

» View Event Details | Created Tue, 30 Jan 2024 08:42:00 +0000

Post-Mortem

A postmortem on this incident is available. It explains the reasons behind the network issues on the osc-fr1 region. If you have any questions, don’t hesitate to contact our support team. https://doc.scalingo.com/security/postmortems/spm-2024-002

Posted: Wed, 31 Jan 2024 12:21:00 +0000

Resolved

The situation is stable again. We consider the incident closed on our side, if you are still facing issues, we invite you to contact our support.

Posted: Tue, 30 Jan 2024 09:39:00 +0000

Update

The situation continues to improve, our operators are still monitoring the issue.

Posted: Tue, 30 Jan 2024 09:08:00 +0000

Update

The situation is improving, we are still monitoring and our operators are working on fixing the issue.

Posted: Tue, 30 Jan 2024 09:01:00 +0000

Update

Our operators are still investigating, we will keep you updated on any improvement

Posted: Tue, 30 Jan 2024 08:54:00 +0000

Investigating

Our team noticed high latency and packet loss on the osc-fr1 region. Our operators have been called.

Posted: Tue, 30 Jan 2024 08:42:00 +0000