Incidents reported on status page for idocus https://status.idocus.com/ https://d1lppblt9t2x15.cloudfront.net/logos/d6727a8226973cc30f860d9865a8cef5.png https://status.idocus.com/ en https://status.idocus.com/incident/948940 Thu, 09 Jul 2026 20:48:10 -0000 https://status.idocus.com/incident/948940#9c041cdb087e30848357560a87558f8e94be7c0259aed6409a97e901d1e1f30a my.idocus.com recovered. https://status.idocus.com/incident/948940 Thu, 09 Jul 2026 20:41:56 -0000 https://status.idocus.com/incident/948940#cd4ae33669436a67647e5893421b7e1fa3a9cadbf6d90ace7f056e1bf4c0e125 my.idocus.com went down. https://status.idocus.com/incident/945485 Tue, 07 Jul 2026 01:15:02 -0000 https://status.idocus.com/incident/945485#4fdc4709a4114e888a25716408a307a283f78b6e48132acbbf22809ab814a99e my.idocus.com recovered. https://status.idocus.com/incident/945485 Tue, 07 Jul 2026 01:09:19 -0000 https://status.idocus.com/incident/945485#115a1f62b3687bcce5caeb00aaf48392643f49ee4fc6375c4456a704155314c9 my.idocus.com went down. https://status.idocus.com/incident/937642 Sun, 28 Jun 2026 12:03:07 -0000 https://status.idocus.com/incident/937642#0fcf45ae97ab5871de8fad9f97ecec396021d55494977521bc9501531652885e bpo.idocus.com and my.idocus.com recovered. https://status.idocus.com/incident/937642 Sun, 28 Jun 2026 11:38:52 -0000 https://status.idocus.com/incident/937642#330e44e00f3ab7cd8bce7c9549ed32fb9336088a11addb787e34d7a44bbf70fa bpo.idocus.com and my.idocus.com went down. https://status.idocus.com/incident/932512 Tue, 23 Jun 2026 20:05:47 -0000 https://status.idocus.com/incident/932512#ab21be0cbd6e406564ed48ae71630676d3806f4ae3eac3ef354449b5462ca0ea bpo.idocus.com recovered. https://status.idocus.com/incident/932512 Tue, 23 Jun 2026 17:30:56 -0000 https://status.idocus.com/incident/932512#3f0051edbc3cd54e470485cbce711c231fb5d2f861b031363fc828e93499f186 bpo.idocus.com went down. https://status.idocus.com/incident/931377 Mon, 22 Jun 2026 18:39:05 -0000 https://status.idocus.com/incident/931377#7a88b4b878e3ded8ca10929f96e1503ae1a08bbc00e85e6c47919ec019f15e97 my.idocus.com recovered. https://status.idocus.com/incident/931377 Mon, 22 Jun 2026 17:59:54 -0000 https://status.idocus.com/incident/931377#f357ffccedd1b40ef646216a626d42c889737eeb05564fea5c2b90f125a76f40 my.idocus.com went down. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 14:03:07 -0000 https://status.idocus.com/incident/919527#2b68d8766f46da77d9e68c18ae99126631c5ab71e860c5a01d1c593fd16c0737 bpo.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 13:57:47 -0000 https://status.idocus.com/incident/919527#f7e83ccfb3317228856ea1eebcef59770455876d5cc58727db2af8e4fab12dfd bpo.idocus.com went down. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 13:36:04 -0000 https://status.idocus.com/incident/919527#0d7834245ebaf0ac4bc7c4634a7093a7911869c134ebb295379e9ae7a6a64522 bpo.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 13:21:47 -0000 https://status.idocus.com/incident/919527#688676b42a5f4bdac04ab56096cebc003ba88ad1e019e43c0842b7f03b44ce0f bpo.idocus.com went down. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 06:48:11 -0000 https://status.idocus.com/incident/919527#23cfa7846b971cc6a274c9b88a4b2f8d5bb79b90d8d78a002772641888f06920 my.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 06:32:53 -0000 https://status.idocus.com/incident/919527#96255d2a772cebeacae7e0c0bc57b05c745880f5c25bef29483db5d7dbd5e6f6 my.idocus.com went down. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 05:12:18 -0000 https://status.idocus.com/incident/919527#d517b24155c47c5fb3f5b25cf0f6b9b03645db14fa8d6151ba30f87e05765930 my.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 05:11:53 -0000 https://status.idocus.com/incident/919527#9d6a7e60f44962880795e99f26496b6d0e8698cf08f0982eecd2069c0e5184ad bpo.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 04:45:39 -0000 https://status.idocus.com/incident/919527#4de5c119cd16151ecadb9b166999c47b6c7aaea0e71a8696aefdab8bb85656ec bpo.idocus.com and my.idocus.com went down. https://status.idocus.com/incident/918592 Tue, 09 Jun 2026 06:54:20 -0000 https://status.idocus.com/incident/918592#247d2cf7c1d199b19ec2b91358f830287a390bd6c5b4efb99c31185147c21b9d my.idocus.com recovered. https://status.idocus.com/incident/918592 Tue, 09 Jun 2026 06:16:06 -0000 https://status.idocus.com/incident/918592#17ce81081b3c65e51da64d6de9837336e74faf3ce377793488e0e9d441274fa4 my.idocus.com went down. https://status.idocus.com/incident/911742 Wed, 03 Jun 2026 06:57:00 -0000 https://status.idocus.com/incident/911742#30145d15a278560c6391a3f56a951b18de96ba48627050969c1583e9da70fe88 my.idocus.com recovered. https://status.idocus.com/incident/911742 Wed, 03 Jun 2026 06:26:57 -0000 https://status.idocus.com/incident/911742#a6f8ec9067cb3a218b4bd89df40d79ac23d2e026b5604184723d74edcc57fa5b my.idocus.com went down. https://status.idocus.com/incident/907042 Thu, 28 May 2026 07:42:10 -0000 https://status.idocus.com/incident/907042#bd00b56877cececfabd21827717050bfe74cfc4eddb11e694044718c3aeaa10f my.idocus.com recovered. https://status.idocus.com/incident/907042 Thu, 28 May 2026 06:14:51 -0000 https://status.idocus.com/incident/907042#7ae26037141c1a9d9161fec4d9aba58d2de8de37c4882c2774cb45be2c2c5f5f my.idocus.com went down. https://status.idocus.com/incident/878792 Fri, 24 Apr 2026 09:53:58 -0000 https://status.idocus.com/incident/878792#79928ca41df734c291cfdcbeba23c9d5985fb7dc637759f51f6dd52976d8fc84 my.idocus.com recovered. https://status.idocus.com/incident/878792 Fri, 24 Apr 2026 09:48:00 -0000 https://status.idocus.com/incident/878792#146a64aa3434c08983093dd0b466d5eb422f5e8ea0b9e52d3ed964520b18df16 my.idocus.com went down. https://status.idocus.com/incident/878792 Fri, 24 Apr 2026 08:47:50 -0000 https://status.idocus.com/incident/878792#d9af09b1e4caa2882bfb061841be5e3c8b68343b6a5428e5c68bcac1486636d9 bpo.idocus.com recovered. https://status.idocus.com/incident/878792 Fri, 24 Apr 2026 08:36:50 -0000 https://status.idocus.com/incident/878792#641028b54d7d7b14d81ade2c3a6319f0e01a589d49167983608788f051ff7a22 bpo.idocus.com went down. https://status.idocus.com/incident/857010 Wed, 25 Mar 2026 05:50:00 -0000 https://status.idocus.com/incident/857010#88503ce272e51d6ff1f31fcaf7d16ac34b5a43c4cb526adc7e41cab455289773 Incident is closed https://status.idocus.com/incident/857010 Wed, 25 Mar 2026 05:46:00 -0000 https://status.idocus.com/incident/857010#5d0bd8c2cabd71ebf56bd633395c50afdb0712e587992a0efcbd3655c94bab2f Incident Post-Mortem — March 25, 2026 Incident Title: Network failure causing load balancer split brain and virtual IP flapping Severity: Critical Date: March 25, 2026 Duration: 17 minutes (06:18 – 06:35 UTC) Status: Resolved Summary On March 25, 2026 at 06:18 UTC, a network disruption caused a split-brain condition on the load balancer cluster, resulting in virtual IP (VIP) flapping between nodes. This led to service unavailability as traffic was being routed inconsistently or just stopped being routed. The issue was detected via monitoring alerts, investigated, and resolved within 17 minutes. Timeline 06:18 — Incident detected. Monitoring alerts fired on VIP flapping and health check failures across the load balancer cluster. 06:25 — Incident acknowledged. On-call engineer picked up the alert and began initial triage. 06:32 — Investigation in progress. Root cause identified as a network partition between load balancer nodes, causing both nodes to claim VIP ownership simultaneously (split-brain). VRRP/Keepalived heartbeats were failing due to upstream network instability. 06:35 — Incident resolved. Network connectivity was restored, the VRRP election reconverged, and a single master was re-established. Traffic resumed normal routing. Health checks confirmed full service recovery. Root Cause A transient network failure disrupted communication between the HAProxy/Keepalived nodes in the load balancer cluster. With VRRP heartbeats no longer reaching the backup node, both nodes promoted themselves to master and began announcing the same virtual IP. This split-brain condition caused VIP flapping, leading to inconsistent traffic routing and intermittent client-facing errors. Impact Intermittent HTTP 502/503 errors for end users during the 17-minute window then service totally unavailable. What Went Well Fast detection: alerting triggered within seconds of the first VIP flap. Quick acknowledgment and focused triage — root cause was identified in under 10 minutes. Total time to resolution was 17 minutes. What Could Be Improved The cluster had no split-brain fencing mechanism in place (e.g., no unicast VRRP fallback, no quorum-based arbitration). No automated remediation kicked in to force a single-master state during the flapping window. Action Items Implement a fencing/quorum mechanism to prevent dual-master scenarios (e.g., add a third lightweight witness node or configure unicast VRRP peers as fallback). Add automated remediation: if VIP flapping is detected for more than N seconds, force-demote one node. Add a runbook entry for split-brain recovery procedures.