Sensor Processing

Incident Report for SysTrack Cloud Edition

Postmortem

What was the issue?

Access to the SysTrack CE America’s cloud was slow due to sensor processing.

What was the root cause?

Preliminary root cause, which is still being verified, was an Azure networking issue causing the utilization of our caching service to respond slowly to requests. This slowness caused cascading slowness to other areas of the platform.

What is the Prevention Strategy?

  1. Running a large cache until other steps have been taken. (Completed)
  2. Migrations to a different networking path to access the cache (Sept 2025)
  3. Updates to how the system gets the cache information for recovery quicker (Oct 2025)
  4. Updating the Sensor agent upload mechanism to implement resends or retries. (TBD)
Posted Jul 15, 2025 - 15:57 UTC

Resolved

This incident has been resolved. If you need further support, please open a ticket at the customer portal (https://customers.lakesidesoftware.com/).
Posted Jul 08, 2025 - 20:13 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 08, 2025 - 18:48 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Jul 08, 2025 - 16:27 UTC

Investigating

We are currently investigating this issue.
Posted Jul 08, 2025 - 15:39 UTC
This incident affected: SysTrack Cloud Edition - Americas (SysTrack API/UI, SysTrack Endpoint Connections).