Lakeside Software Cloud Services OUTAGE - Germany
Incident Report for SysTrack Cloud Edition
Postmortem

What was the issue?

Lakeside Germany website was not loading for all customers.

What was the impact?

All customers with Lakeside Cloud Tenant on Lakeside Germany domains were not able to access website and its modules.

Root cause:

Microsoft Networking outage (Tracking ID VSG1-B90) between 07:05 UTC and 12:43 UTC on 25 January 2023.

Prevention Strategy

Implement external https checks for the website that will also trigger alerts for faster customer notifications.

Posted Feb 02, 2023 - 19:32 UTC

Resolved
This incident has been resolved. After monitoring the situation for the past 1.5 hours, we are confident that Microsoft's Mitigation has worked. If you have any future issues, please contact Lakeside Support at support@lakesidesoftware.com.
Posted Jan 25, 2023 - 11:10 UTC
Update
We are continuing to monitor for any further issues.
Posted Jan 25, 2023 - 09:50 UTC
Monitoring
Azure Teams has remediated the issue as per the latest update.

Lakeside Services on SysTrack Cloud Edition - Germany are back to online. We are continuing to actively monitor the situation.

Tracking ID - VSG1-B90
https://app.azure.com/h/VSG1-B90/5287ef
Posted Jan 25, 2023 - 09:43 UTC
Identified
The Incident is related to Azure Global outage on Networking Services. We are monitoring the status and we are in constant touch with Azure. We will share the next update as soon possible.
Posted Jan 25, 2023 - 09:17 UTC
Investigating
We are currently investigating this issue.
Posted Jan 25, 2023 - 09:13 UTC
Identified
We are currently investigating this issue.
Posted Jan 25, 2023 - 09:13 UTC
Update
The Incident is related to Azure Global outage on Networking Services. We are monitoring the status and we are in constant touch with Azure. We will share the next update as soon possible.
Posted Jan 25, 2023 - 09:03 UTC
Investigating
Lakeside Software is currently experiencing an outage related to SysTrack Cloud Edition impacting the below regions. The outage is currently impacting core cloud services. During this outage, users will not be able to use the SysTrack cloud UI, make API calls, or use third-party integrations that depend on the API. We're actively working on identifying the root cause and will resolve the problem as soon as possible. We apologize for the inconvenience and will provide an update as soon as all services are operational.
Posted Jan 25, 2023 - 08:52 UTC
This incident affected: SysTrack Cloud Edition - Germany (SysTrack API/UI, SysTrack Endpoint Connections).