Percona Xtradb Cluster (PXC) outage [Resolved]
[Resolved] 2024-03-14 13:08
Cluster state is completely restored, all service operational.
[In progress] 2024-03-14 09:40
Unfortunately, the previous actions did not help, PXC is still malfunctioning. We initiate a disaster recovery plan. ETA 4 hours. All sites in “Instance 1-X” clusters will be unavailable during this period to preserve data integrity.
[Acknowledged] 2024-03-14 08:48
Probably the planned upgrade of Percona Xtradb Cluster to the latest version (from 1.13 to 1.14) caused a significant decrease in the performance of the DB cluster. We will try to disable all side services and scale cluster to one node and plan to rollback on off-peak period.
[Triggered] 2024-03-14 08:35