Storage issues on cloudscale LPG

The volume attachment issues stemming from the upstream emergency OS update have been completely resolved.

In close coordination with our partner, Cloudscale, all affected virtual servers have been successfully remediated via live migrations and targeted power-cycles.

Our monitoring phase has concluded with all system performance, application connectivity, and infrastructure metrics showing a sustained return to normal operational baselines.

We are in the final stages of resolving the volume attachment issues for APPUiO.

We will update this page as soon as the system is fully verified and normal service has been resumed.

The active bridge call with Cloudscale continues.

While an increasing number of customer systems have successfully recovered, remediation is not yet fully complete.

We are continuing to work through the remaining affected servers and will provide our next update at 14:45 CEST.

Partner Status Link: https://www.cloudscale-status.net/incidents/247957

Our active bridge call with Cloudscale continues, and we are seeing a steady increase in successfully recovered customer systems.

Remediation is not fully complete yet, and we are continuing to process the remaining affected servers.

We remain on the line with our partner and will provide our next update at 14:15 CEST.

Partner Status Link: https://www.cloudscale-status.net/incidents/247957

Our active bridge call with Cloudscale remains ongoing.

We are seeing more and more customer systems recovering as the case-by-case remediation steps continue to prove effective.

We are not fully done yet and continue to systematically apply these fixes to the remaining affected servers.

We will provide our next heartbeat update at 13:45 CEST.

Partner Status Link: https://www.cloudscale-status.net/incidents/247957

Our active bridge call with Cloudscale remains ongoing, and the case-by-case remediation steps are proving effective.

We are seeing a steady return to normal operations on the individual systems we have processed so far.

We are continuing to systematically apply these fixes to the remaining affected servers.

We remain fully engaged on the line with our partner to see this through to completion.

Our next heartbeat update will be at 13:15 CEST.

Partner Status Link: https://www.cloudscale-status.net/incidents/247957

Our active bridge call with Cloudscale remains ongoing.

Remediation efforts are continuing in real time as we systematically work through the affected systems.

We remain fully engaged with our partner on the line and will provide our next update at 12:45 CEST.

Partner Status Link: https://www.cloudscale-status.net/incidents/247957

We remain on an active bridge call with Cloudscale tracking the infrastructure remediation. Together we are introducing targeted power-cycles (power-off/power-on) to safely clear any remaining volume attachment errors.
As individual servers are processed, systems are recovering automatically. However, you may experience brief, temporary application downtime lasting a few minutes if your specific host requires a power-cycle.
No customer action is required. We are tracking the progress in real time and will post our next heartbeat update at 12:15 CEST.

Partner Status Link: https://www.cloudscale-status.net/incidents/247957

Cloudscale is currently evaluating live migrations for all “Flex” and “Plus” virtual servers to relocate them to stable hypervisors.

This process keeps your systems online, but you may experience temporary performance degradation for a few minutes and brief network packet loss for a few seconds as the migration completes.

We are tracking the efforts in real time and will post our next update here at 11:45.

Partner Status Link: https://www.cloudscale-status.net/incidents/247957

Cloudscale’s work to reattach the remaining storage volumes is still underway. Affected servers are gradually recovering as they progress, but the incident remains active for outstanding nodes.
We remain in communication with their team and will post our next update here at 11:15.

Partner Status Link: https://www.cloudscale-status.net/incidents/247957

Cloudscale, together with our support, identified that the issue is storage volumes failing to attach to compute nodes. Cloudscale is now working through reattaching the affected volumes, which has already reduced the number of impacted servers.
We continue to work with Cloudscale to resolve the remaining ones and are waiting on their official root cause analysis.

Cloudscale Status: https://www.cloudscale-status.net/incidents/247957

We’re currently monitoring an issue with cloudscale storage in the LPG region which affects some customers on APPUiO LPG 2.

See https://www.cloudscale-status.net/incidents/247957 for the cloudscale status.