Resolved

Incident Details:
Date: 2023-07-04
Start Time: 16:37:00 UTC
End Time: 16:49:00 UTC
Impact: Veryfi API gateways are unavailable

Incident Summary:
Our regular monitoring systems picked up signs of the Veryfi API gateways not being accessible at 16:37:00 UTC on the 4th of July, 2023. This happened due to a significant update of our internal Veryfi resources which unfortunately led to a period of instability in our public services. We acknowledge that this issue may have caused some irregularities in the usage of our services.

Important Note:
For users who have set up a retry mechanism for times when Veryfi API fails, it's probable that most requests were successfully processed on subsequent attempts following the brief period of service degradation.

Resolution:
Upon identifying the issue, our engineering team immediately started investigating the root cause. At 16:49:00 UTC, we were able to resolve the issue by finishing the update of the internal Veryfi resources which stopped the network instability and made the Veryfi API gateway available again.

Preventive Measures:
To ensure that such an incident does not happen again, we have taken the following steps:
1. Fully separated our internal resources from our production resources in terms of networking to make them totally independent.
2. Reviewed and enhanced our infrastructure update process to ensure a proper update plan before changes were applied.
3. Conducted a thorough post-mortem analysis to identify any additional areas of improvement in our infrastructure and processes.