Incidents reported on status page for idocus https://status.idocus.com/ https://d1lppblt9t2x15.cloudfront.net/logos/d6727a8226973cc30f860d9865a8cef5.png https://status.idocus.com/ en https://status.idocus.com/incident/1046366 Wed, 02 Sep 2026 06:57:41 -0000 https://status.idocus.com/incident/1046366#887a7985058c8e3c6de6c893babb05dd9959352e134dbbb6ce47a4d62b07d132 my.idocus.com recovered. https://status.idocus.com/incident/1046366 Wed, 02 Sep 2026 06:51:10 -0000 https://status.idocus.com/incident/1046366#b4c7f3df8083df866dfe6b68e03490314738dfb552d01e1de57bcf7aa860409a my.idocus.com went down. https://status.idocus.com/incident/1046366 Wed, 02 Sep 2026 06:27:20 -0000 https://status.idocus.com/incident/1046366#876111cdd61e87d3835bb50fb50c53a7deced7e3413418252d9341164c5468b2 my.idocus.com recovered. https://status.idocus.com/incident/1046366 Wed, 02 Sep 2026 06:20:57 -0000 https://status.idocus.com/incident/1046366#3b7a9af66167f57b1ecd8f4ae49fe2882a9e5542aa8cc86eab44927be37a2c39 my.idocus.com went down. https://status.idocus.com/incident/1046366 Wed, 02 Sep 2026 06:15:16 -0000 https://status.idocus.com/incident/1046366#1a699922a38cada57987dde8b2a5fbf4b6546528725708e7cc23be71b99974bc my.idocus.com recovered. https://status.idocus.com/incident/1046366 Wed, 02 Sep 2026 06:09:04 -0000 https://status.idocus.com/incident/1046366#8888a3c6784d1d22c0e75cab20af4197f9c36b7ada07507bc13f3faa42bb5115 my.idocus.com went down. https://status.idocus.com/incident/1044452 Tue, 01 Sep 2026 07:18:25 -0000 https://status.idocus.com/incident/1044452#682686b1aed75d3d1d08e856860f726e280073ca5fcc4761948d6264c5a01f6a my.idocus.com recovered. https://status.idocus.com/incident/1044452 Tue, 01 Sep 2026 06:11:51 -0000 https://status.idocus.com/incident/1044452#fae29c9bc95f0439e225193ddd7e742ca63e8b174679c3885804e27ddbe341ca my.idocus.com went down. https://status.idocus.com/incident/1033775 Wed, 26 Aug 2026 06:44:56 -0000 https://status.idocus.com/incident/1033775#6c6a90c71a62ecc8dec0e888ca796080073394dc7481ccaa9d8936040c0cdac2 my.idocus.com recovered. https://status.idocus.com/incident/1033775 Wed, 26 Aug 2026 06:35:49 -0000 https://status.idocus.com/incident/1033775#02169678b2804d010d5492ab93e0b6e2974f0996032013a913756abf3215bb71 my.idocus.com went down. https://status.idocus.com/incident/1008070 Tue, 11 Aug 2026 07:27:11 -0000 https://status.idocus.com/incident/1008070#da970132362daa3d8b637a3f340d99fbb20e853777f4a64b2096174e59afb3ee my.idocus.com recovered. https://status.idocus.com/incident/1008070 Tue, 11 Aug 2026 06:29:51 -0000 https://status.idocus.com/incident/1008070#18765884d75d7512467e4ca52dd39090539787e19daf430d0c23e7ddd6fa24c1 my.idocus.com went down. https://status.idocus.com/incident/1008070 Tue, 11 Aug 2026 05:26:56 -0000 https://status.idocus.com/incident/1008070#fd5a65be314cfe311caeac3e82b21a5b3aa44a9fe34bab6cdd9c99b5ab9f1f62 bpo.idocus.com and my.idocus.com recovered. https://status.idocus.com/incident/1008070 Tue, 11 Aug 2026 04:40:01 -0000 https://status.idocus.com/incident/1008070#56b50faf481f1a83582e6dafbc52b46f205beb4ac4d7a8508317f1db4c470750 my.idocus.com went down. https://status.idocus.com/incident/1008070 Tue, 11 Aug 2026 04:39:42 -0000 https://status.idocus.com/incident/1008070#b95ac1910d4930d6011033a5ff16c0afb09fa7c06e7bcd320ea438ea87a827e2 bpo.idocus.com went down. https://status.idocus.com/incident/995465 Tue, 04 Aug 2026 07:33:12 -0000 https://status.idocus.com/incident/995465#05309a1fa86c366acdbacad3ae4fcca80b73f1e729ef96aec2153e89a5d8a7b7 my.idocus.com recovered. https://status.idocus.com/incident/995465 Tue, 04 Aug 2026 07:26:50 -0000 https://status.idocus.com/incident/995465#7995c1194f0a29cfc6791d530a6deece6ac8262315fdfb6aba2176420135c497 my.idocus.com went down. https://status.idocus.com/incident/987984 Fri, 31 Jul 2026 07:06:11 -0000 https://status.idocus.com/incident/987984#b7d217737b6173a6ca49f3418f5c70c34583bd963fcd9baca6ee68c8ea9bcd53 my.idocus.com recovered. https://status.idocus.com/incident/987984 Fri, 31 Jul 2026 06:48:21 -0000 https://status.idocus.com/incident/987984#71fe0539770cb9c03d4cdbc48967b90077c738e3fdaa1e6cc91d80b1d821e703 my.idocus.com went down. https://status.idocus.com/incident/982173 Tue, 28 Jul 2026 07:12:16 -0000 https://status.idocus.com/incident/982173#2d20f96e8dabb7385faf33251b0ea383baf1ed41a4affff13323fa4a7dbb6ead my.idocus.com recovered. https://status.idocus.com/incident/982173 Tue, 28 Jul 2026 06:59:55 -0000 https://status.idocus.com/incident/982173#ec651628a14b6c3d268af84de3b6021494999fbd1f3dd7f90091e1706fb2cf85 my.idocus.com went down. https://status.idocus.com/incident/965692 Thu, 16 Jul 2026 06:57:07 -0000 https://status.idocus.com/incident/965692#49ab933494b9ef34691033b49ce6e60022c52247ae1f673cd9426323cfa406d4 my.idocus.com recovered. https://status.idocus.com/incident/965692 Thu, 16 Jul 2026 06:50:54 -0000 https://status.idocus.com/incident/965692#04a9bf6aad55a3647ef019b1070b94839a69b4412c0c3514f1cc3444f308c888 my.idocus.com went down. https://status.idocus.com/incident/964727 Wed, 15 Jul 2026 12:00:06 -0000 https://status.idocus.com/incident/964727#0e3284ee3f8fa84637d4995d47cca3b942612d53af77d05f4a616b39bfcc9f91 my.idocus.com recovered. https://status.idocus.com/incident/964727 Wed, 15 Jul 2026 11:29:57 -0000 https://status.idocus.com/incident/964727#f852041965e4c9c3de809bc4424567a3b56186deda3af7fa75fe780a994ec266 my.idocus.com went down. https://status.idocus.com/incident/948940 Fri, 10 Jul 2026 04:11:53 -0000 https://status.idocus.com/incident/948940#9dbfc8520b905f5629eb8781784facac3920cfd84fee61797c84cbc0431b702f bpo.idocus.com and my.idocus.com recovered. https://status.idocus.com/incident/948940 Thu, 09 Jul 2026 23:48:37 -0000 https://status.idocus.com/incident/948940#c4bfee3e682b290c77eb8ea20e6f43924c7f66c746f62eb300ac8cd6ca313cc3 bpo.idocus.com and my.idocus.com went down. https://status.idocus.com/incident/948940 Thu, 09 Jul 2026 20:48:10 -0000 https://status.idocus.com/incident/948940#9c041cdb087e30848357560a87558f8e94be7c0259aed6409a97e901d1e1f30a my.idocus.com recovered. https://status.idocus.com/incident/948940 Thu, 09 Jul 2026 20:41:56 -0000 https://status.idocus.com/incident/948940#cd4ae33669436a67647e5893421b7e1fa3a9cadbf6d90ace7f056e1bf4c0e125 my.idocus.com went down. https://status.idocus.com/incident/945485 Tue, 07 Jul 2026 01:15:02 -0000 https://status.idocus.com/incident/945485#4fdc4709a4114e888a25716408a307a283f78b6e48132acbbf22809ab814a99e my.idocus.com recovered. https://status.idocus.com/incident/945485 Tue, 07 Jul 2026 01:09:19 -0000 https://status.idocus.com/incident/945485#115a1f62b3687bcce5caeb00aaf48392643f49ee4fc6375c4456a704155314c9 my.idocus.com went down. https://status.idocus.com/incident/937642 Sun, 28 Jun 2026 12:03:07 -0000 https://status.idocus.com/incident/937642#0fcf45ae97ab5871de8fad9f97ecec396021d55494977521bc9501531652885e bpo.idocus.com and my.idocus.com recovered. https://status.idocus.com/incident/937642 Sun, 28 Jun 2026 11:38:52 -0000 https://status.idocus.com/incident/937642#330e44e00f3ab7cd8bce7c9549ed32fb9336088a11addb787e34d7a44bbf70fa bpo.idocus.com and my.idocus.com went down. https://status.idocus.com/incident/932512 Tue, 23 Jun 2026 20:05:47 -0000 https://status.idocus.com/incident/932512#ab21be0cbd6e406564ed48ae71630676d3806f4ae3eac3ef354449b5462ca0ea bpo.idocus.com recovered. https://status.idocus.com/incident/932512 Tue, 23 Jun 2026 17:30:56 -0000 https://status.idocus.com/incident/932512#3f0051edbc3cd54e470485cbce711c231fb5d2f861b031363fc828e93499f186 bpo.idocus.com went down. https://status.idocus.com/incident/931377 Mon, 22 Jun 2026 18:39:05 -0000 https://status.idocus.com/incident/931377#7a88b4b878e3ded8ca10929f96e1503ae1a08bbc00e85e6c47919ec019f15e97 my.idocus.com recovered. https://status.idocus.com/incident/931377 Mon, 22 Jun 2026 17:59:54 -0000 https://status.idocus.com/incident/931377#f357ffccedd1b40ef646216a626d42c889737eeb05564fea5c2b90f125a76f40 my.idocus.com went down. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 14:03:07 -0000 https://status.idocus.com/incident/919527#2b68d8766f46da77d9e68c18ae99126631c5ab71e860c5a01d1c593fd16c0737 bpo.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 13:57:47 -0000 https://status.idocus.com/incident/919527#f7e83ccfb3317228856ea1eebcef59770455876d5cc58727db2af8e4fab12dfd bpo.idocus.com went down. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 13:36:04 -0000 https://status.idocus.com/incident/919527#0d7834245ebaf0ac4bc7c4634a7093a7911869c134ebb295379e9ae7a6a64522 bpo.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 13:21:47 -0000 https://status.idocus.com/incident/919527#688676b42a5f4bdac04ab56096cebc003ba88ad1e019e43c0842b7f03b44ce0f bpo.idocus.com went down. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 06:48:11 -0000 https://status.idocus.com/incident/919527#23cfa7846b971cc6a274c9b88a4b2f8d5bb79b90d8d78a002772641888f06920 my.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 06:32:53 -0000 https://status.idocus.com/incident/919527#96255d2a772cebeacae7e0c0bc57b05c745880f5c25bef29483db5d7dbd5e6f6 my.idocus.com went down. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 05:12:18 -0000 https://status.idocus.com/incident/919527#d517b24155c47c5fb3f5b25cf0f6b9b03645db14fa8d6151ba30f87e05765930 my.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 05:11:53 -0000 https://status.idocus.com/incident/919527#9d6a7e60f44962880795e99f26496b6d0e8698cf08f0982eecd2069c0e5184ad bpo.idocus.com recovered. https://status.idocus.com/incident/919527 Wed, 10 Jun 2026 04:45:39 -0000 https://status.idocus.com/incident/919527#4de5c119cd16151ecadb9b166999c47b6c7aaea0e71a8696aefdab8bb85656ec bpo.idocus.com and my.idocus.com went down. https://status.idocus.com/incident/918592 Tue, 09 Jun 2026 06:54:20 -0000 https://status.idocus.com/incident/918592#247d2cf7c1d199b19ec2b91358f830287a390bd6c5b4efb99c31185147c21b9d my.idocus.com recovered. https://status.idocus.com/incident/918592 Tue, 09 Jun 2026 06:16:06 -0000 https://status.idocus.com/incident/918592#17ce81081b3c65e51da64d6de9837336e74faf3ce377793488e0e9d441274fa4 my.idocus.com went down. https://status.idocus.com/incident/911742 Wed, 03 Jun 2026 06:57:00 -0000 https://status.idocus.com/incident/911742#30145d15a278560c6391a3f56a951b18de96ba48627050969c1583e9da70fe88 my.idocus.com recovered. https://status.idocus.com/incident/911742 Wed, 03 Jun 2026 06:26:57 -0000 https://status.idocus.com/incident/911742#a6f8ec9067cb3a218b4bd89df40d79ac23d2e026b5604184723d74edcc57fa5b my.idocus.com went down. https://status.idocus.com/incident/907042 Thu, 28 May 2026 07:42:10 -0000 https://status.idocus.com/incident/907042#bd00b56877cececfabd21827717050bfe74cfc4eddb11e694044718c3aeaa10f my.idocus.com recovered. https://status.idocus.com/incident/907042 Thu, 28 May 2026 06:14:51 -0000 https://status.idocus.com/incident/907042#7ae26037141c1a9d9161fec4d9aba58d2de8de37c4882c2774cb45be2c2c5f5f my.idocus.com went down. https://status.idocus.com/incident/878792 Fri, 24 Apr 2026 09:53:58 -0000 https://status.idocus.com/incident/878792#79928ca41df734c291cfdcbeba23c9d5985fb7dc637759f51f6dd52976d8fc84 my.idocus.com recovered. https://status.idocus.com/incident/878792 Fri, 24 Apr 2026 09:48:00 -0000 https://status.idocus.com/incident/878792#146a64aa3434c08983093dd0b466d5eb422f5e8ea0b9e52d3ed964520b18df16 my.idocus.com went down. https://status.idocus.com/incident/878792 Fri, 24 Apr 2026 08:47:50 -0000 https://status.idocus.com/incident/878792#d9af09b1e4caa2882bfb061841be5e3c8b68343b6a5428e5c68bcac1486636d9 bpo.idocus.com recovered. https://status.idocus.com/incident/878792 Fri, 24 Apr 2026 08:36:50 -0000 https://status.idocus.com/incident/878792#641028b54d7d7b14d81ade2c3a6319f0e01a589d49167983608788f051ff7a22 bpo.idocus.com went down. https://status.idocus.com/incident/857010 Wed, 25 Mar 2026 05:50:00 -0000 https://status.idocus.com/incident/857010#88503ce272e51d6ff1f31fcaf7d16ac34b5a43c4cb526adc7e41cab455289773 Incident is closed https://status.idocus.com/incident/857010 Wed, 25 Mar 2026 05:46:00 -0000 https://status.idocus.com/incident/857010#5d0bd8c2cabd71ebf56bd633395c50afdb0712e587992a0efcbd3655c94bab2f Incident Post-Mortem — March 25, 2026 Incident Title: Network failure causing load balancer split brain and virtual IP flapping Severity: Critical Date: March 25, 2026 Duration: 17 minutes (06:18 – 06:35 UTC) Status: Resolved Summary On March 25, 2026 at 06:18 UTC, a network disruption caused a split-brain condition on the load balancer cluster, resulting in virtual IP (VIP) flapping between nodes. This led to service unavailability as traffic was being routed inconsistently or just stopped being routed. The issue was detected via monitoring alerts, investigated, and resolved within 17 minutes. Timeline 06:18 — Incident detected. Monitoring alerts fired on VIP flapping and health check failures across the load balancer cluster. 06:25 — Incident acknowledged. On-call engineer picked up the alert and began initial triage. 06:32 — Investigation in progress. Root cause identified as a network partition between load balancer nodes, causing both nodes to claim VIP ownership simultaneously (split-brain). VRRP/Keepalived heartbeats were failing due to upstream network instability. 06:35 — Incident resolved. Network connectivity was restored, the VRRP election reconverged, and a single master was re-established. Traffic resumed normal routing. Health checks confirmed full service recovery. Root Cause A transient network failure disrupted communication between the HAProxy/Keepalived nodes in the load balancer cluster. With VRRP heartbeats no longer reaching the backup node, both nodes promoted themselves to master and began announcing the same virtual IP. This split-brain condition caused VIP flapping, leading to inconsistent traffic routing and intermittent client-facing errors. Impact Intermittent HTTP 502/503 errors for end users during the 17-minute window then service totally unavailable. What Went Well Fast detection: alerting triggered within seconds of the first VIP flap. Quick acknowledgment and focused triage — root cause was identified in under 10 minutes. Total time to resolution was 17 minutes. What Could Be Improved The cluster had no split-brain fencing mechanism in place (e.g., no unicast VRRP fallback, no quorum-based arbitration). No automated remediation kicked in to force a single-master state during the flapping window. Action Items Implement a fencing/quorum mechanism to prevent dual-master scenarios (e.g., add a third lightweight witness node or configure unicast VRRP peers as fallback). Add automated remediation: if VIP flapping is detected for more than N seconds, force-demote one node. Add a runbook entry for split-brain recovery procedures.