Instance 1-4

[Resolved] 2024-03-14 13:08
Cluster state is completely restored, all service operational.

[In progress] 2024-03-14 09:40
Unfortunately, the previous actions did not help, PXC is still malfunctioning. We initiate a disaster recovery plan. ETA 4 hours. All sites in "Instance 1-X" clusters will be unavailable during this period to preserve data integrity.

[Acknowledged] 2024-03-14 08:48
Probably the planned upgrade of Percona Xtradb Cluster to the latest version (from 1.13 to 1.14) caused a significant decrease in the performance of the DB cluster. We will try to disable all side services and scale cluster to one node and plan to rollback on off-peak period.

[Triggered] 2024-03-14 08:35

[Resolved] 2024-01-29 17:05
Cluster state is completely restored, all service operational.

[Acknowledged]  2024-01-29 13:20
We found a issues with k8s cluster. Will be migrate to a brand new cluster, ETA 1-2 hours.

[Triggered] 2024-01-29 13:10

[Resolved] 2023-07-03 20:05
Cluster state is completely restored, all service operational.

[Acknowledged] 2023-07-03 17:50
We found a split brain issue in PXC cluster, will restore cluster state from one of clsuers node. ETA 1-2 hours.

[Triggered] 2023-07-03 17:30

[Resolved] 2022-07-13 15:20
Backup completely restored, all service operational.

[Acknowledged] 2022-07-13 07:10
We decide to restore database information from daily backup to a brand new PXC cluster (version 8). ETA 4-5 hours.

[Triggered] 2022-07-13 05:15
During shceduled maintatnce (migration cluster from PXC 5.6 to PXC 8 with support Encryption at rest), we discovered a compatibility issue with the full recovery of the cluster when migrating from version 5.6 to 8 version.