Degraded
Elevated errors during document processing
Started 24 Jun at 06:35pm UTC, resolved 25 Jun at 01:45am UTC.
Resolved
We have implemented and validated a fix for the problematic edge case. All systems are back to normal.
Updated
We have identified the root cause as being a newly surfaced edge case scenario in requests being submitted for processing. These particular requests consume an unusually high level of compute resources, which has downstream impact on other requests processed by the same worker processes.
We are currently working on a solution to handle these edge cases gracefully.
Updated
We have reduced error rates further by over provisioning compute resources and spreading requests across more worker nodes. We are continuing work on identifying the root cause of the issue.
Created
We are experiencing an increased error rate in document processing requests. A small percentage of requests are returning a 504 error response. We are investigating the issue.