API is down
Started 4 Jul at 04:38pm UTC, resolved 4 Jul at 04:49pm UTC.
Start Time: 16:37:00 UTC
End Time: 16:49:00 UTC
Impact: Veryfi API gateways are unavailable
Our regular monitoring systems picked up signs of the Veryfi API gateways not being accessible at 16:37:00 UTC on the 4th of July, 2023. This happened due to a significant update of our internal Veryfi resources which unfortunately led to a period of instability in our public services. We acknowledge that this issue may have caused some irregularities in the usage of our services.
For users who have set up a retry mechanism for times when Veryfi API fails, it's probable that most requests were successfully processed on subsequent attempts following the brief period of service degradation.
Upon identifying the issue, our engineering team immediately started investigating the root cause. At 16:49:00 UTC, we were able to resolve the issue by finishing the update of the internal Veryfi resources which stopped the network instability and made the Veryfi API gateway available again.
To ensure that such an incident does not happen again, we have taken the following steps:
1. Fully separated our internal resources from our production resources in terms of networking to make them totally independent.
2. Reviewed and enhanced our infrastructure update process to ensure a proper update plan before changes were applied.
3. Conducted a thorough post-mortem analysis to identify any additional areas of improvement in our infrastructure and processes.
At 16:37:00 UTC on the 4th of July, 2023 our regular monitoring systems picked up signs of the Veryfi API gateways not being accessible.