[osc-fr1] Some queries to the API time out

» View Event Details | Created Thu, 02 Mar 2023 11:01:00 +0000

Resolved

Our team consider this incident over. Sorry for the inconvenience. If you still experience timeouts, please contact our support team.

Posted: Thu, 02 Mar 2023 11:53:00 +0000

Update

We stabilized the internal component. We are still closely monitoring the situation. If you still experience timeouts, please contact our support team.

Posted: Thu, 02 Mar 2023 11:28:00 +0000

Investigating

We are currently investigating an issue with an internal component. The impact for our customers is that some queries targeting the API time out. Operations such as loading the dashboard, restarting applications, etc may also timeout.

Our team is working on restoring the situation.

Posted: Thu, 02 Mar 2023 11:01:00 +0000

[osc-fr1] A database node mis-behaving

» View Event Details | Created Fri, 03 Mar 2023 00:20:00 +0000

Resolved

All database are back up and running. If you're still facing issues. Please contact us on the support.

Posted: Fri, 03 Mar 2023 01:07:00 +0000

Update

The impacted node looks now healthy. We are checking that every databases looks healthy before closing this incident

Posted: Fri, 03 Mar 2023 00:34:00 +0000

Investigating

One database node is misbehaving. We are taking a look at it.

Posted: Fri, 03 Mar 2023 00:20:00 +0000

[osc-fr1] Network degradation

» View Event Details | Created Fri, 03 Mar 2023 08:53:00 +0000

Resolved

Incident is closed please come to our support if you have any question.

Posted: Fri, 03 Mar 2023 11:47:00 +0000

Update

Network probes are indicating nominal traffic performance since 10:30, situation is getting more stable

Posted: Fri, 03 Mar 2023 10:21:00 +0000

Update

Situation is improving, deployments are now going through

Posted: Fri, 03 Mar 2023 09:47:00 +0000

Update

The ticket has been escalated to our network provider.

Posted: Fri, 03 Mar 2023 09:13:00 +0000

Update

We are identifying packet loss between our infrastructure and Internet impacting application access and access to the Internet from the applications.

Posted: Fri, 03 Mar 2023 09:00:00 +0000

Update

Deployments seem to be impacted and my hang or fail.

Posted: Fri, 03 Mar 2023 09:00:00 +0000

Investigating

Our probes have detected networking slowdown, confirmed by some users in the 'osc-fr1' region infrastructure, our team is looking at it.

Posted: Fri, 03 Mar 2023 08:53:00 +0000

[osc-secnum-fr1] Network degradation

» View Event Details | Created Tue, 07 Mar 2023 15:16:00 +0000

Resolved

The incident has been closed on our side. An investigation is still in progress with our network provider.

Posted: Tue, 07 Mar 2023 17:35:00 +0000

Update

Access to the network has been stable for more than 1 hour now. We don't have the root cause yet but we are still monitoring and investigating these network issues.

Posted: Tue, 07 Mar 2023 16:21:00 +0000

Update

Access to the network has been stable for 30 minutes now. We are still investigating these network issues with our infrastructure provider.

Posted: Tue, 07 Mar 2023 15:49:00 +0000

Update

We have noticed multiple packet loss between our infrastructure and the Internet, which impacts access to applications and access to the Internet from applications. Here are the timestamps of each short down times we noticed:

• 15h48 -> 15h49
• 15h56 -> 15h57
• 16h09 -> 16h10
• 16h16 -> 16h17

We are currently in contact with our Infrastructure provider and the issue was reported to them.

Posted: Tue, 07 Mar 2023 15:31:00 +0000

Investigating

Our probes have detected networking slowdown, confirmed by some users on the 'osc-secnum-fr1' region infrastructure, our team is looking at it.

Posted: Tue, 07 Mar 2023 15:16:00 +0000

[osc-fr1] A database node mis-behaving

» View Event Details | Created Sun, 12 Mar 2023 00:26:00 +0000

Resolved

All databases has been restarted. If you still have an issue, please contact us on the support.

Posted: Sun, 12 Mar 2023 01:03:00 +0000

Update

The impacted node is now healthy again and databases are restarting.

Posted: Sun, 12 Mar 2023 00:40:00 +0000

Investigating

One database node is misbehaving. We are taking a look at it.

Posted: Sun, 12 Mar 2023 00:26:00 +0000

[osc-fr1] Maintenance on database nodes

» View Event Details | Created Mon, 13 Mar 2023 19:00:00 +0000

Resolved

The maintenance is now finished.

If your database is still unavailable please contact us via the support.

Posted: Mon, 13 Mar 2023 19:30:00 +0000

Update

The second batch has been completed. We are starting the post maintenance checks.

If your database is still unavailable please contact us via the support.

Posted: Mon, 13 Mar 2023 19:26:00 +0000

Update

The second node has been restarted. Database hosted on that node are currently rebooting.

Posted: Mon, 13 Mar 2023 19:19:00 +0000

Update

The first batch has been completed. Starting the second batch.

Posted: Mon, 13 Mar 2023 19:11:00 +0000

Update

The first node has been restarted. Database hosted on that node are currently rebooting.

Posted: Mon, 13 Mar 2023 19:06:00 +0000

In Progress

The maintenance is starting. The first batch of databases will be restarted in the next minutes.

Posted: Mon, 13 Mar 2023 19:01:00 +0000

Scheduled

A planned maintenance will take place on the osc-fr1 region. On the 13th of March at 20:00 CET.

Impact:

• No direct impact on application containers
• Databases using a business plan will experience an automatic failover, your applications should reconnect automatically after a couple of seconds
• Databases using a starter or sandbox plan will be unavailable for 30 to 60 minutes, until it is restarted

How do I know which of my applications are impacted:

We sent an email to each application owner whose database is impacted by this maintenance. If you did not receive any email, you’re not impacted.

If you have any question, do not hesitate to reach us via the in-app support.

Posted: Mon, 13 Mar 2023 19:00:00 +0000

[osc-fr1] A database node mis-behaving

» View Event Details | Created Thu, 23 Mar 2023 15:20:00 +0000

Resolved

All checks are green. The all databases are back up and running.

If you're still having issue with your database please contact the support.

Posted: Thu, 23 Mar 2023 16:10:00 +0000

Update

All databases have been restarted. We are currently running post incident checks.

Posted: Thu, 23 Mar 2023 15:56:00 +0000

Update

Database on the impacted node are being restarted.

Posted: Thu, 23 Mar 2023 15:40:00 +0000

Investigating

One of our database node is mis-behaving. We are currently looking into it.

Posted: Thu, 23 Mar 2023 15:20:00 +0000

[osc-fr1] A database node is mis-behaving

» View Event Details | Created Tue, 28 Mar 2023 11:14:00 +0000

Resolved

All checks are green. The all databases are back up and running.

If you're still having issue with your database please contact the support.

Posted: Tue, 28 Mar 2023 11:49:00 +0000

Update

All databases have been restarted, we are entering post incident checks.

Posted: Tue, 28 Mar 2023 11:49:00 +0000

Update

Database are restarting on the impacted node.

Posted: Tue, 28 Mar 2023 11:27:00 +0000

Investigating

One of our database node is mis-behaving. We are currently looking into it.

Posted: Tue, 28 Mar 2023 11:14:00 +0000

[osc-fr1] Elevated error rate on our API

» View Event Details | Created Thu, 30 Mar 2023 15:08:00 +0000

Resolved

The situation is back to normal.

The issue was traced back to the component for storing application metrics this component kept accepting request but never answered to those request.

This quickly filled up the HTTP slots of our API which stopped answering in its turn.

Posted: Thu, 30 Mar 2023 15:30:00 +0000

Update

Deployments are working again. The API is back up and running. However performances are not perfect. The Metrics part of our API is still down.

Posted: Thu, 30 Mar 2023 15:13:00 +0000

Investigating

We observe an elevated error rate on our API. Our team is currently working on it.

This impacts all API calls on osc-fr1 and deployments.

Applications and databases are still up and running.

Posted: Thu, 30 Mar 2023 15:08:00 +0000