The incident was caused by elevated I/O latency in GCE Persistent Disk within a single region, which impacted our database performance and led to increased application response times.
We mitigated the issue by relocating our database instances out of the affected zone. Our team is continuing to work with Google to ensure the underlying issue is fully resolved, but this is no longer impacting Ordergroove services.
The incident began at 2:20 PM EST, mitigation was completed by 2:59 PM EST, and full recovery was achieved at 3:18 PM EST.
Posted Oct 29, 2025 - 15:55 EDT
Update
We’ve identified and resolved the root cause of the issue. Latency levels are steadily improving, and we’re closely monitoring system performance to ensure continued stability. We’ll provide another update by 4 PM EST.
Posted Oct 29, 2025 - 15:22 EDT
Investigating
We're investigating the issue affecting multiple services. We're seeing increased latencies and some requests fail to complete.
We'll provide an update by 3:15PM EST.
Posted Oct 29, 2025 - 14:47 EDT
This incident affected: Ordergroove Platform (e-commerce platform agnostic) (Subscription Offers, Subscription Creation, Order Placement, APIs, Admin Dashboard (previously RC3)).