Incidents reported on status page for Veryfi, Inc https://status.veryfi.com/ https://d1lppblt9t2x15.cloudfront.net/logos/23cc4ad3258f07c4eee1a60ee9f18af7.png https://status.veryfi.com/ en https://status.veryfi.com/incident/608517 Wed, 25 Jun 2025 01:45:00 -0000 https://status.veryfi.com/incident/608517#a43a71ff46cd685350901e3c18a535cbf8f2bc99ec5d5157356d1374dfe40155 We have implemented and validated a fix for the problematic edge case. All systems are back to normal. https://status.veryfi.com/incident/608517 Wed, 25 Jun 2025 00:36:00 -0000 https://status.veryfi.com/incident/608517#77a185ab4bdcf6b167b1dc128d7b90dba26a1df23eb8a7489dcbc14cec7077ae We have identified the root cause as being a newly surfaced edge case scenario in requests being submitted for processing. These particular requests consume an unusually high level of compute resources, which has downstream impact on other requests processed by the same worker processes. We are currently working on a solution to handle these edge cases gracefully. https://status.veryfi.com/incident/608517 Tue, 24 Jun 2025 21:37:00 -0000 https://status.veryfi.com/incident/608517#6cefe3c9744c3b5e4b2f3e641ba4d1b8cd9c616b7de5a9eda986806eea57fe68 We have reduced error rates further by over provisioning compute resources and spreading requests across more worker nodes. We are continuing work on identifying the root cause of the issue. https://status.veryfi.com/incident/608517 Tue, 24 Jun 2025 18:35:00 -0000 https://status.veryfi.com/incident/608517#b54b463e2a693d6895f9e3ce763caf0779107bfa9593872f21abede96242297e We are experiencing an increased error rate in document processing requests. A small percentage of requests are returning a 504 error response. We are investigating the issue. https://status.veryfi.com/incident/596514 Tue, 03 Jun 2025 22:11:00 -0000 https://status.veryfi.com/incident/596514#9bb8ad32591457092be4a96b1cb37aa1920a6be35bd12c0cb55e566cc6d461ef Elevated error rates have now stopped without our intervention. We are continuing to investigate the incident, however all services are now back to normal. https://status.veryfi.com/incident/596514 Tue, 03 Jun 2025 21:08:00 -0000 https://status.veryfi.com/incident/596514#663d60fd7d4fe2c2bf4dd7d86c1abe6717697f958e8565706bc691e5b12b506f We are observing an elevated error count related to processing of documents in files that are submitted in pre-signed S3 URLs. Other files appear to be processing as expected. We are investigating the issue. https://status.veryfi.com/incident/567007 Fri, 16 May 2025 06:01:35 -0000 https://status.veryfi.com/incident/567007#10bc1f410ce8fec6e4519331efb9b56d7a7b403c042e1c46321f453cc0b5c29c Lens SDK Repository recovered. https://status.veryfi.com/incident/567007 Fri, 16 May 2025 05:58:30 -0000 https://status.veryfi.com/incident/567007#dbd181b651259b773c1e729c8e202fb3fff747c430d2c02262ef402d8a398e10 Lens SDK Repository went down. https://status.veryfi.com/incident/556860 Mon, 05 May 2025 13:55:00 -0000 https://status.veryfi.com/incident/556860#1876187264ba6113b0e00efaa6ccedea3fa30451809f4639cbfe4dca3e8f5a9c Email collector is back online https://status.veryfi.com/incident/550588 Thu, 24 Apr 2025 19:51:00 -0000 https://status.veryfi.com/incident/550588#0333966ff7d433a5bf1a2e1936b7361ed0f6afbd0befd304025347fcdd1ef3ea The email collector service is online. The email queue will be fully processed shortly. https://status.veryfi.com/incident/550588 Thu, 24 Apr 2025 15:40:00 -0000 https://status.veryfi.com/incident/550588#79dbda8c41a1b2097ad7c3042c2b112662650981f6969e6634e903555271e18f Due to new Google security requirements, Veryfi email collector is under maintenance. We are working to make it available as soon as possible. https://status.veryfi.com/incident/547136 Fri, 18 Apr 2025 09:49:00 -0000 https://status.veryfi.com/incident/547136#673f21d94202a0b53585be46c6b0305e8336d3245ab4ebddb87933286f507074 We have completed recovering the Nexus repository. While all artifacts have been recovered, some authentication keys were not recoverable. If you experience authentication errors while connecting to Lens SDK repositories, please reset your keys at https://app.veryfi.com/api/settings/keys/ For further help, please contact support@veryfi.com https://status.veryfi.com/incident/547136 Fri, 18 Apr 2025 08:14:00 -0000 https://status.veryfi.com/incident/547136#1725718391e4013f230724b3ebbbb4226db58ba3f3888f91d058cbd894d59215 We are currently addressing an issue with our Nexus repository caused by data corruption. Our team is actively working to recover all artifacts and restore full functionality as quickly as possible. To prevent future issues, we are also migrating to improved infrastructure for greater reliability and performance. We apologize for any inconvenience and appreciate your patience. Updates will be provided as we make progress. For urgent concerns, please contact support@veryfi.com https://status.veryfi.com/incident/547136 Fri, 18 Apr 2025 02:11:02 -0000 https://status.veryfi.com/incident/547136#f28887ed53b5ed206e7c7191d0577bdfa6ef1782c0f3e6a0fb12c81de15fd381 Lens SDK Repository went down. https://status.veryfi.com/incident/541977 Tue, 08 Apr 2025 16:23:39 -0000 https://status.veryfi.com/incident/541977#c2470dadeba88a444316619e669e640205799e4858ae9d83676d5295524386ee Lens SDK Repository recovered. https://status.veryfi.com/incident/541977 Tue, 08 Apr 2025 16:17:30 -0000 https://status.veryfi.com/incident/541977#cee412095b5cdeda44de8d9b5d7ae6f5e3acf4ebda6cf42a0e8b36407b7bcdb4 Lens SDK Repository went down. https://status.veryfi.com/incident/527119 Wed, 12 Mar 2025 14:33:00 -0000 https://status.veryfi.com/incident/527119#810540dc2c5dd2774477bdab6650eb27e96d1b1188bd08495d1c3ec6e7c825b6 We have identified the root cause of the issue as a recently updated Web Application Firewall (WAF) rule. The impact was confined to POST requests containing multipart/form-data payloads with specific data patterns that triggered the WAF rule. Given the nature of the rule, the specific data required to activate it, and the low volume of affected requests, this false positive scenario was not detected during our testing phase. The change has been reverted, and the issue is now fully resolved. https://status.veryfi.com/incident/527119 Wed, 12 Mar 2025 12:59:00 -0000 https://status.veryfi.com/incident/527119#6613fd438622e9ebb7e7b996b22b4dbbe406c1642e6b113d901df120b8b94136 Some customers are experiencing 403 errors affecting some specific document processing requests. This is impacting a low volume of traffic. https://status.veryfi.com/incident/508730 Fri, 07 Feb 2025 02:35:40 -0000 https://status.veryfi.com/incident/508730#4d1f81a355a0c91d07f6425c594cbcd8c20decc805b80f078c217bac51c01794 Lens SDK Repository recovered. https://status.veryfi.com/incident/508730 Fri, 07 Feb 2025 02:25:51 -0000 https://status.veryfi.com/incident/508730#977fe98a5597ca01d35488cd62f1f9364d0e91b027c1b01bc5fb3c383d6bf1cf Lens SDK Repository went down. https://status.veryfi.com/incident/497539 Fri, 17 Jan 2025 21:14:00 -0000 https://status.veryfi.com/incident/497539#a12fba90f4853ba0281af2dd095381050b5afe00182d9ce7608f57263467c43f Start Time: 2025-01-17 12:55:00 UTC Resolution Time: 2025-01-17 21:15:00 UTC Current Status: All issues with our API services have been resolved. The following services are now fully operational: Documents Inbox and Search API: Documents uploaded via the API are now visible in the inbox and Search API is fully operational. Document Updates Webhooks: Webhook notifications for document updates are functioning correctly. If updates happened during the incident, webhooks will arrive shortly. Duplicates Similarity Search: The similarity search for duplicates is working. If the document was uploaded during the incident, it wasn't checked against data uploaded during that time. We appreciate your patience and apologize for any inconvenience this may have caused. Next Update: No further updates are scheduled unless new issues arise. Thank you for your understanding. https://status.veryfi.com/incident/497539 Fri, 17 Jan 2025 21:14:00 -0000 https://status.veryfi.com/incident/497539#a12fba90f4853ba0281af2dd095381050b5afe00182d9ce7608f57263467c43f Start Time: 2025-01-17 12:55:00 UTC Resolution Time: 2025-01-17 21:15:00 UTC Current Status: All issues with our API services have been resolved. The following services are now fully operational: Documents Inbox and Search API: Documents uploaded via the API are now visible in the inbox and Search API is fully operational. Document Updates Webhooks: Webhook notifications for document updates are functioning correctly. If updates happened during the incident, webhooks will arrive shortly. Duplicates Similarity Search: The similarity search for duplicates is working. If the document was uploaded during the incident, it wasn't checked against data uploaded during that time. We appreciate your patience and apologize for any inconvenience this may have caused. Next Update: No further updates are scheduled unless new issues arise. Thank you for your understanding. https://status.veryfi.com/incident/497539 Fri, 17 Jan 2025 12:55:00 -0000 https://status.veryfi.com/incident/497539#7c481af43c9938385cf179df3640fcb77aafe8983d8646ef532917cb8e7b887c Start Time: 2025-01-17 12:55:00 UTC We are currently experiencing an issue with some of our API services. The following services are affected: Documents Inbox and Search API: Documents uploaded via the API are temporarily not visible in the inbox. However, you can still query these documents using their ID. Document Updates Webhooks: Webhook notifications for document updates are not functioning as expected. Duplicates Similarity Search: The similarity search for duplicates is not working for data uploaded after the incident start time. Ongoing Efforts: Our team is actively working to diagnose and resolve these issues. We apologize for any inconvenience caused and appreciate your patience as we work to restore full service. Next Update: We will provide another update within the next hour or once we have significant progress to report. Thank you for your understanding. https://status.veryfi.com/incident/497539 Fri, 17 Jan 2025 12:55:00 -0000 https://status.veryfi.com/incident/497539#7c481af43c9938385cf179df3640fcb77aafe8983d8646ef532917cb8e7b887c Start Time: 2025-01-17 12:55:00 UTC We are currently experiencing an issue with some of our API services. The following services are affected: Documents Inbox and Search API: Documents uploaded via the API are temporarily not visible in the inbox. However, you can still query these documents using their ID. Document Updates Webhooks: Webhook notifications for document updates are not functioning as expected. Duplicates Similarity Search: The similarity search for duplicates is not working for data uploaded after the incident start time. Ongoing Efforts: Our team is actively working to diagnose and resolve these issues. We apologize for any inconvenience caused and appreciate your patience as we work to restore full service. Next Update: We will provide another update within the next hour or once we have significant progress to report. Thank you for your understanding. https://status.veryfi.com/incident/473027 Thu, 05 Dec 2024 16:41:00 -0000 https://status.veryfi.com/incident/473027#78a61c3d2b2fd54ec7be8d37b604a81c3450601840f33dc9c5e02e9c1dfc82cd Start Time: 09:48:00 UTC Failure Period: - High failure rate: 09:45:00-10:10:00 UTC - Medium failure rate: 10:10:00-13:10:00 UTC Current Status: Problem fully mitigated Recovery Actions: - Utilize the retry mechanism as the retried calls would return the correct response - Veryfi can assist with reprocessing requests with lost OCR text, please request We experienced a significant degradation in our document extraction service, specifically impacting the Optical Character Recognition (OCR) text extraction process. This failure prevented text retrieval from a substantial portion of processed documents. Our team is actively investigating the cause of the issues. https://status.veryfi.com/incident/442531 Thu, 10 Oct 2024 18:32:11 -0000 https://status.veryfi.com/incident/442531#cb4fff90493a393715737fe362f3698a2cbcd0bda5b699c39af20bef98813621 Lens SDK Repository recovered. https://status.veryfi.com/incident/442531 Thu, 10 Oct 2024 18:00:23 -0000 https://status.veryfi.com/incident/442531#e437d607ca83284faeb00608a4f155ce1352b3049f91b804d47d5c3f383abe76 Lens SDK Repository went down. https://status.veryfi.com/incident/438823 Thu, 03 Oct 2024 17:19:00 -0000 https://status.veryfi.com/incident/438823#1df4201c100fbc6aac78b349b84a614fe8e3d67e8c3eb15bec90d29dbcbd8e9c All services are currently back to normal. The root cause of this issue is related to underlying, intermittent network latencies in AWS and is a recurrence of an earlier issue that we are currently still investigating with AWS. While investigations continue, we have restructured our infrastructure workloads such that fewer customers are impacted during these network latencies. We truly value your ongoing trust and appreciate your patience and understanding as we continue to work on this issue. https://status.veryfi.com/incident/438823 Thu, 03 Oct 2024 16:58:00 -0000 https://status.veryfi.com/incident/438823#49cbd0a5d2c22663aa3ef0cc69b00f8d9e598c244f8ae54271493075f62b6809 All services have been operating at improved levels for the last 50 mins, however we do still observe somewhat higher than normal processing times for some customers. We are continuing to work on this issue. https://status.veryfi.com/incident/438823 Thu, 03 Oct 2024 15:06:00 -0000 https://status.veryfi.com/incident/438823#9e230ed5e1bbe3e39917cb4260a097dea2e390aad8c4782d42ed11a0b1ec0787 We are currently experiencing degradation in processing time, which may impact a subset of clients. Start time: 13:15:00 UTC. Our team is actively working to resolve the issue and restore optimal performance. We’re reviewing the root cause of the incident. Thank you for your patience and understanding . https://status.veryfi.com/incident/437067 Mon, 30 Sep 2024 18:25:00 -0000 https://status.veryfi.com/incident/437067#f51a5ba9a6af4f9873fc01d0a1885e3c4cfcabb269fe3c75979792e60933761f All services are currently back to normal. The root cause of this issue is related to underlying latencies in AWS and is a recurrence of an earlier issue that we believed we had resolved. While this issue is currently not in effect, we are continuing to work with AWS to identify and remediate the root issue to prevent its recurrence. https://status.veryfi.com/incident/437067 Mon, 30 Sep 2024 14:00:00 -0000 https://status.veryfi.com/incident/437067#c144e269f7f75e6681b8b628bc2ff15fe7441c57005252bf18a14f9b3cd2c043 We are experiencing a general degradation in processing time since 13:37:00 UTC. We're reviewing the root cause of the incident. https://status.veryfi.com/incident/433675 Mon, 23 Sep 2024 19:54:00 -0000 https://status.veryfi.com/incident/433675#0026afd0159d932a3481ca80165883aaf8828467db95f5bd235b298fb7311d44 All services are currently back to normal. We have identified the root cause being related to network latencies in AWS. We are continuing to work with AWS to better understand the problem and mitigation strategies going forward. https://status.veryfi.com/incident/433675 Mon, 23 Sep 2024 18:35:00 -0000 https://status.veryfi.com/incident/433675#e704926f1ce4f61ba953e6ec7b45c55ccdcbc7ac9063fcf7c0a1cc0fd44b6e76 We are experiencing a general degradation in processing time since 16:08:00 UTC. We're reviewing the root cause of the incident. https://status.veryfi.com/incident/407716 Sat, 17 Aug 2024 06:21:52 +0000 https://status.veryfi.com/incident/407716#0b64c135855bc3f96cecfa3217925b62a274be891f064115ed6791d51b02bcf8 Maintenance completed https://status.veryfi.com/incident/407716 Sat, 17 Aug 2024 06:21:52 -0000 https://status.veryfi.com/incident/407716#5a4230f1d6fa598e898a15f1aa243eb76ea47fd260363b0af51df0089ee1839c The Veryfi OCR API will be unavailable for a short time on Saturday, 17 August, 2024. Planned maintenance is scheduled to begin at 06:00 UTC and is designed to take less than 15 minutes. All OCR API calls during this maintenance will receive a server 500 message. The API Portal will be simultaneously affected during this time. For your planning, the full maintenance window is 17 August, 2024 from 06:00 to 07:00 UTC with an impact on system availability for up to 15 minutes. Historically, such maintenance windows have resulted in less than 5 minutes of actual service impact. If you have any questions or concerns, please contact us support@veryfi.com https://status.veryfi.com/incident/407716 Sat, 17 Aug 2024 06:21:52 +0000 https://status.veryfi.com/incident/407716#0b64c135855bc3f96cecfa3217925b62a274be891f064115ed6791d51b02bcf8 Maintenance completed https://status.veryfi.com/incident/407716 Sat, 17 Aug 2024 06:21:52 -0000 https://status.veryfi.com/incident/407716#5a4230f1d6fa598e898a15f1aa243eb76ea47fd260363b0af51df0089ee1839c The Veryfi OCR API will be unavailable for a short time on Saturday, 17 August, 2024. Planned maintenance is scheduled to begin at 06:00 UTC and is designed to take less than 15 minutes. All OCR API calls during this maintenance will receive a server 500 message. The API Portal will be simultaneously affected during this time. For your planning, the full maintenance window is 17 August, 2024 from 06:00 to 07:00 UTC with an impact on system availability for up to 15 minutes. Historically, such maintenance windows have resulted in less than 5 minutes of actual service impact. If you have any questions or concerns, please contact us support@veryfi.com https://status.veryfi.com/incident/407716 Sat, 17 Aug 2024 06:00:00 -0000 https://status.veryfi.com/incident/407716#5a4230f1d6fa598e898a15f1aa243eb76ea47fd260363b0af51df0089ee1839c The Veryfi OCR API will be unavailable for a short time on Saturday, 17 August, 2024. Planned maintenance is scheduled to begin at 06:00 UTC and is designed to take less than 15 minutes. All OCR API calls during this maintenance will receive a server 500 message. The API Portal will be simultaneously affected during this time. For your planning, the full maintenance window is 17 August, 2024 from 06:00 to 07:00 UTC with an impact on system availability for up to 15 minutes. Historically, such maintenance windows have resulted in less than 5 minutes of actual service impact. If you have any questions or concerns, please contact us support@veryfi.com https://status.veryfi.com/incident/407716 Sat, 17 Aug 2024 06:00:00 -0000 https://status.veryfi.com/incident/407716#5a4230f1d6fa598e898a15f1aa243eb76ea47fd260363b0af51df0089ee1839c The Veryfi OCR API will be unavailable for a short time on Saturday, 17 August, 2024. Planned maintenance is scheduled to begin at 06:00 UTC and is designed to take less than 15 minutes. All OCR API calls during this maintenance will receive a server 500 message. The API Portal will be simultaneously affected during this time. For your planning, the full maintenance window is 17 August, 2024 from 06:00 to 07:00 UTC with an impact on system availability for up to 15 minutes. Historically, such maintenance windows have resulted in less than 5 minutes of actual service impact. If you have any questions or concerns, please contact us support@veryfi.com https://status.veryfi.com/incident/409951 Tue, 06 Aug 2024 22:22:24 -0000 https://status.veryfi.com/incident/409951#3dfffd67d45127be71604ce8d65f627007afc660f8ce384b07632a141984554c Lens SDK Repository recovered. https://status.veryfi.com/incident/409951 Tue, 06 Aug 2024 22:17:57 -0000 https://status.veryfi.com/incident/409951#41fa287bb5e7afb5bfb9ba6c435dbed59cf5dc9eda465193f364ed681e722e64 Lens SDK Repository went down. https://status.veryfi.com/incident/409951 Tue, 06 Aug 2024 22:04:19 -0000 https://status.veryfi.com/incident/409951#cb10cb068d1a20e1d7c1b738cef8062bc92a0c76c90d4a6ec740ff45c779b71c Lens SDK Repository recovered. https://status.veryfi.com/incident/409951 Tue, 06 Aug 2024 22:02:28 -0000 https://status.veryfi.com/incident/409951#06777f2cfa4469bc0a8fb82c8525a7cc0f3cdeab04bb8eebb4b6c931cf85133c Lens SDK Repository went down. https://status.veryfi.com/incident/388766 Mon, 24 Jun 2024 17:50:00 -0000 https://status.veryfi.com/incident/388766#e67db7a6982d4ae690fd1fb8deb05926c9854c3e3d5326284d0cae8cb859f51a All services are stable and operating at normal levels. We identified a component in the Veryfi API that scaled insufficiently under a particularly high load. This issue was resolved and services began to operate once again at stable levels. https://status.veryfi.com/incident/388766 Mon, 24 Jun 2024 17:20:00 -0000 https://status.veryfi.com/incident/388766#66bf7ca980fd1dbabf3b331dfc31b1416cdce12afe13d7cdc52daf8813ca44bc We are currently experiencing cloud networking issues. The Veryfi API performance is degraded. Some requests' latency has increased. We're doing our best in order to found out the root cause of these networking issues and fix it. https://status.veryfi.com/incident/384506 Fri, 14 Jun 2024 22:23:33 -0000 https://status.veryfi.com/incident/384506#9d65884cece19c5ce4b647bf6b3877868feeb45c3afcc5f657eece738eae12d0 Lens SDK Repository recovered. https://status.veryfi.com/incident/384506 Fri, 14 Jun 2024 22:22:58 -0000 https://status.veryfi.com/incident/384506#6ec6b4c837187d3f5f7aa3f70c2024f68ec930260718f3ff546a1bd72727afe6 Lens SDK Repository went down. https://status.veryfi.com/incident/378839 Mon, 03 Jun 2024 16:30:00 -0000 https://status.veryfi.com/incident/378839#2d95d3a8ac62d7b242d1ceef0fdb1b938b4577d2ac80c3c5fc55e1048ff358e5 From June 1 0:00 UTC until June 3 15:00 UTC processing performance has been affected. At the moment, all services are operating normally. https://status.veryfi.com/incident/372618 Tue, 21 May 2024 15:33:53 -0000 https://status.veryfi.com/incident/372618#00178a39fb67aa497c1eeee545b15366655f6f85314d664b667a9b3bfccdb811 Lens SDK Repository recovered. https://status.veryfi.com/incident/372618 Tue, 21 May 2024 15:30:00 -0000 https://status.veryfi.com/incident/372618#685bdd2415e180cd4b4fe4d3ecc9a49965b9ceaf40bcd79362539b7a85cc2452 Lens SDK Repository went down. https://status.veryfi.com/incident/372618 Tue, 21 May 2024 15:26:42 -0000 https://status.veryfi.com/incident/372618#33b607a2238b70e7762985a958162cf3ee7f4ab498a2d3df7b3be4e65d7f0415 Lens SDK Repository recovered. https://status.veryfi.com/incident/372618 Tue, 21 May 2024 15:22:06 -0000 https://status.veryfi.com/incident/372618#684424b26517f41c3b7de36e52689dc47dbeb6dce0e29522ff3d120bf9204760 Lens SDK Repository went down. https://status.veryfi.com/incident/372618 Tue, 21 May 2024 14:18:10 -0000 https://status.veryfi.com/incident/372618#8dd38f363be3f8a8322e90a8352170a91b7553bbf43ea99bcbb46e1b1805eb46 Lens SDK Repository recovered. https://status.veryfi.com/incident/372618 Tue, 21 May 2024 14:16:38 -0000 https://status.veryfi.com/incident/372618#3e1fa90c984603e1c441b37d038607c94b60861597eb25e978614568a669ba85 Lens SDK Repository went down. https://status.veryfi.com/incident/364457 Fri, 03 May 2024 02:20:00 -0000 https://status.veryfi.com/incident/364457#b18df682f09d5c04afd7fb2b79a5d52bb05cc0a92bc67d40449dcd18b8f52d75 We have identified the root cause being silent failures in a recent deployment to some servers. The issue has been resolved and services are operating normally again. https://status.veryfi.com/incident/364457 Fri, 03 May 2024 02:07:00 -0000 https://status.veryfi.com/incident/364457#a7518e4d00f3ba51e5c6b0fdaa31497d9741f75773e564039126d480476f1e0b We have identified an increased error rate as of 01:17 UTC and are investigating the underlying cause. https://status.veryfi.com/incident/356327 Tue, 16 Apr 2024 15:27:00 -0000 https://status.veryfi.com/incident/356327#86e997fa02725bf7cccd3650f8ca228c9225f1b7dce9e792a7d5da17bbb365f6 The processing time has returned to its normal value. https://status.veryfi.com/incident/356327 Tue, 16 Apr 2024 15:18:00 -0000 https://status.veryfi.com/incident/356327#a90d6762a33a466b4d0c2d3ab6e5b10a7dc6ec8892e697921925ff70cf4d6f7b We recently made a change to our infrastructure that temporarily increased processing times (14:00 - 15:05 UTC). We've resolved the issue, and everything is now operating as normal. https://status.veryfi.com/incident/291778 Fri, 24 Nov 2023 18:20:00 -0000 https://status.veryfi.com/incident/291778#3e903c4f2bffe1966b14cd105de6bfe28196e11f2ad26460a8045838b71f9532 Services continue to run at normal performance levels and have been stable for the last 15mins. The issue has been resolved. https://status.veryfi.com/incident/291778 Fri, 24 Nov 2023 18:20:00 -0000 https://status.veryfi.com/incident/291778#3e903c4f2bffe1966b14cd105de6bfe28196e11f2ad26460a8045838b71f9532 Services continue to run at normal performance levels and have been stable for the last 15mins. The issue has been resolved. https://status.veryfi.com/incident/291778 Fri, 24 Nov 2023 18:08:00 -0000 https://status.veryfi.com/incident/291778#60d93a8d11c71fe6242172b7460bb22ab64b51ec416d5bda0ceb84d012c71eae We have identified the underlying cause as a long running database query that has been triggered a high number of times throughout our cluster. The query has been updated and we are already seeing signs of improvement in API performance. We are continuing to monitor. https://status.veryfi.com/incident/291778 Fri, 24 Nov 2023 18:08:00 -0000 https://status.veryfi.com/incident/291778#60d93a8d11c71fe6242172b7460bb22ab64b51ec416d5bda0ceb84d012c71eae We have identified the underlying cause as a long running database query that has been triggered a high number of times throughout our cluster. The query has been updated and we are already seeing signs of improvement in API performance. We are continuing to monitor. https://status.veryfi.com/incident/291778 Fri, 24 Nov 2023 17:40:00 -0000 https://status.veryfi.com/incident/291778#51b70d003a0b735c46be15ec0ca597d47736acba792bc28a4fe1681199cc46eb We are currently experiencing some service performance degredation affecting the Veryfi API https://status.veryfi.com/incident/291778 Fri, 24 Nov 2023 17:40:00 -0000 https://status.veryfi.com/incident/291778#51b70d003a0b735c46be15ec0ca597d47736acba792bc28a4fe1681199cc46eb We are currently experiencing some service performance degredation affecting the Veryfi API https://status.veryfi.com/incident/261013 Tue, 19 Sep 2023 04:45:00 -0000 https://status.veryfi.com/incident/261013#17b41716ca757c480c12deb58937f444ad43cb59d66f5a8daac79953a088775b Veryfi's systems have been operating consistently at normal levels since 2:25am UTC. AWS have confirmed that all services have been returned to normal operating levels and stable. https://status.veryfi.com/incident/261013 Tue, 19 Sep 2023 04:45:00 -0000 https://status.veryfi.com/incident/261013#17b41716ca757c480c12deb58937f444ad43cb59d66f5a8daac79953a088775b Veryfi's systems have been operating consistently at normal levels since 2:25am UTC. AWS have confirmed that all services have been returned to normal operating levels and stable. https://status.veryfi.com/incident/261013 Tue, 19 Sep 2023 01:23:00 -0000 https://status.veryfi.com/incident/261013#9c1ac0bcee7f082d99943d69b78418912cba1dd60b8762739aa275fa49dd9772 Our cloud service provider continues to work on the underlying network latencies and has issued updates that some stability was attained in the last hour or so, with operations yet to return completely to normal. We also see improvements in Veryfi's overall performance over the last hour, though not yet back to normal levels. https://status.veryfi.com/incident/261013 Tue, 19 Sep 2023 01:23:00 -0000 https://status.veryfi.com/incident/261013#9c1ac0bcee7f082d99943d69b78418912cba1dd60b8762739aa275fa49dd9772 Our cloud service provider continues to work on the underlying network latencies and has issued updates that some stability was attained in the last hour or so, with operations yet to return completely to normal. We also see improvements in Veryfi's overall performance over the last hour, though not yet back to normal levels. https://status.veryfi.com/incident/261013 Mon, 18 Sep 2023 20:43:00 -0000 https://status.veryfi.com/incident/261013#ef2c121c2eedc3ebd4fb064cfedb6e7163433cbb4ba8e47016f1f7426d67f2f1 Our cloud service provider (AWS) is currently experiencing network latencies that are impacting our services. Some requests are currently experiencing performance degradation. The AWS team have identified the root cause and are progressing towards a full resolution. https://status.veryfi.com/incident/261013 Mon, 18 Sep 2023 20:43:00 -0000 https://status.veryfi.com/incident/261013#ef2c121c2eedc3ebd4fb064cfedb6e7163433cbb4ba8e47016f1f7426d67f2f1 Our cloud service provider (AWS) is currently experiencing network latencies that are impacting our services. Some requests are currently experiencing performance degradation. The AWS team have identified the root cause and are progressing towards a full resolution. https://status.veryfi.com/incident/228635 Tue, 04 Jul 2023 16:49:00 -0000 https://status.veryfi.com/incident/228635#c68931647236778fae9538b68f940b3b51e4be7d70138b3b76fe37305827fd07 Incident Details: Date: 2023-07-04 Start Time: 16:37:00 UTC End Time: 16:49:00 UTC Impact: Veryfi API gateways are unavailable Incident Summary: Our regular monitoring systems picked up signs of the Veryfi API gateways not being accessible at 16:37:00 UTC on the 4th of July, 2023. This happened due to a significant update of our internal Veryfi resources which unfortunately led to a period of instability in our public services. We acknowledge that this issue may have caused some irregularities in the usage of our services. Important Note: For users who have set up a retry mechanism for times when Veryfi API fails, it's probable that most requests were successfully processed on subsequent attempts following the brief period of service degradation. Resolution: Upon identifying the issue, our engineering team immediately started investigating the root cause. At 16:49:00 UTC, we were able to resolve the issue by finishing the update of the internal Veryfi resources which stopped the network instability and made the Veryfi API gateway available again. Preventive Measures: To ensure that such an incident does not happen again, we have taken the following steps: 1. Fully separated our internal resources from our production resources in terms of networking to make them totally independent. 2. Reviewed and enhanced our infrastructure update process to ensure a proper update plan before changes were applied. 3. Conducted a thorough post-mortem analysis to identify any additional areas of improvement in our infrastructure and processes. https://status.veryfi.com/incident/228635 Tue, 04 Jul 2023 16:38:00 -0000 https://status.veryfi.com/incident/228635#5904ef4b674890f04504020fdfd6dbd1c2b6ac59169b6ed02434d9969b067d13 At 16:37:00 UTC on the 4th of July, 2023 our regular monitoring systems picked up signs of the Veryfi API gateways not being accessible. https://status.veryfi.com/incident/206770 Fri, 12 May 2023 22:24:27 -0000 https://status.veryfi.com/incident/206770#7773c9b23978d15e0be7dd79869f6c8747972f4ab8075162a4cc074d3bc2be1a API Portal recovered. https://status.veryfi.com/incident/206770 Fri, 12 May 2023 22:21:03 -0000 https://status.veryfi.com/incident/206770#56c1606622c1760ace06ffd92ac13d401fbf6407ec1362a3353203bf99c1bbf0 API Portal went down. https://status.veryfi.com/incident/198180 Wed, 19 Apr 2023 14:53:00 -0000 https://status.veryfi.com/incident/198180#9e7fa4cb4c3fc2123ab69388563c879886a983dcf71878a409ddc335b3bb3801 At 14:53:00 UTC, the underlying issues have now been resolved and all services are back to normal. Incident Details: Date: 2023-04-19 Start Time: 14:13:00 UTC End Time: 14:53:00 UTC Impact: Performance degradation and processing of documents performed by a non-current version of ML model Incident Summary: At 14:13 UTC, a component of our primary processing cluster was restarted as part of a routine automated maintenance process. This process failed due to a version mismatch recently introduced during component upgrades. From that point in time, all traffic began being routed to a fallback cluster, which is running a non-current version of the machine learning model. During the impact window, some results may have been different than expected due to a different machine learning model being in use. The fallback cluster began scaling out and reached full production levels within approximately 20 mins and the request processing time returned to normal. At 14:53 the underlying issue in the primary cluster was resolved and traffic was redirected back to it. Resolution: The version mismatch in our primary cluster was resolved and all services returned to full operation. Preventive Measures: To ensure that such an incident does not happen again, we have taken the following steps: We have revised our processes for version control and patching to ensure this mismatch does not happen in the future We have revised our processes around fallback clusters to ensure they are also running production versions of code and models https://status.veryfi.com/incident/198180 Wed, 19 Apr 2023 14:13:00 -0000 https://status.veryfi.com/incident/198180#f87b52ec073f5ad13bc5059398441f5e2ef80d327204c82ef89cc907a90c831f At 14:13 UTC, we discovered a performance degradation and processing of documents performed by a non-current version of ML model https://status.veryfi.com/incident/186472 Sat, 18 Mar 2023 16:02:00 -0000 https://status.veryfi.com/incident/186472#a1763e5b6a207c414a14439802262fb989242ea42af3535cd4fd8875c99226fc At 16:02:00 UTC, we were able to resolve the issue by killing all malfunctioning instances and implementing updated configurations to prevent similar errors from occurring in the future. https://status.veryfi.com/incident/186472 Sat, 18 Mar 2023 12:51:00 -0000 https://status.veryfi.com/incident/186472#9d36b417b57d403fa6f1f3c834904c57cb62eb9e179fc8c9967f5e736bb4c60b Incident Details: - Date: 2023-03-18 - Start Time: 12:50:00 UTC - End Time: 16:02:00 UTC - Impact: Random HTTP 500 server responses Incident Summary: On March 18, 2023, at 12:50:00 UTC, our monitoring systems detected some newly started Veryfi instances that were malfunctioning. This resulted in random HTTP 500 server responses being served to our API clients. We understand that this may have caused disruptions in the normal use of our services. Important Note: If you have implemented a retry mechanism when Veryfi API fails, it is likely that most of your requests were successfully processed on the second attempt during this period of service degradation. Resolution: Upon identifying the issue, our engineering team immediately started investigating the root cause and working on a fix. At 16:02:00 UTC, we were able to resolve the issue by killing all malfunctioning instances and implementing updated configurations to prevent similar errors from occurring in the future. Preventive Measures: - To ensure that such an incident does not happen again, we have taken the following steps: - Implemented a more stringent monitoring system to detect and address issues in instances more promptly. - Reviewed and enhanced our instance deployment process to ensure proper configuration before instances go live. - Conducted a thorough post-mortem analysis to identify any additional areas of improvement in our infrastructure and processes. https://status.veryfi.com/incident/143305 Thu, 08 Dec 2022 21:39:18 -0000 https://status.veryfi.com/incident/143305#b3cca14b56aebe89c1bb6047ba81d655aa0c009d2fc514531e76607d90033145 API Portal and Website recovered. https://status.veryfi.com/incident/143305 Thu, 08 Dec 2022 21:36:59 -0000 https://status.veryfi.com/incident/143305#27658055ad1fc81d8799c2038565f4fcd97be5c10988cdf1bb357ac5a0779fa6 API Portal and Website went down. https://status.veryfi.com/incident/131865 Wed, 02 Nov 2022 03:58:39 -0000 https://status.veryfi.com/incident/131865#4dd03fd934c7f88193e5cb2dbe2cd92958e5a54772ee619fb098d8c80a6177a6 API recovered. https://status.veryfi.com/incident/131865 Wed, 02 Nov 2022 03:51:30 -0000 https://status.veryfi.com/incident/131865#054357fce237e6c58b1454feca412b68aae3d1404445057317f9e5e8a8d3332d API went down. https://status.veryfi.com/incident/126175 Thu, 13 Oct 2022 16:25:13 -0000 https://status.veryfi.com/incident/126175#57232ee7f8e4fc82ec88aaa830983a7bda54223d3ee6085009c57ac684f512d6 API recovered. https://status.veryfi.com/incident/126175 Thu, 13 Oct 2022 16:22:00 -0000 https://status.veryfi.com/incident/126175#d8a29969316caa004b17bbf774832af83956ba055c02affd22388bf1e518f08a API went down. https://status.veryfi.com/incident/125373 Mon, 10 Oct 2022 23:47:51 -0000 https://status.veryfi.com/incident/125373#e4027f0c00113679007cd419a7d6b332ecc73e029900ed295f6bfe4875d37697 API recovered. https://status.veryfi.com/incident/125373 Mon, 10 Oct 2022 23:45:34 -0000 https://status.veryfi.com/incident/125373#7baa722138e0b393deb94a4a1029698b3709fab6de3fc959964d278ebcbc905e API went down. https://status.veryfi.com/incident/123736 Wed, 05 Oct 2022 19:30:10 -0000 https://status.veryfi.com/incident/123736#72f99624e1a8f2a1679176a327074c0b1dcffa869bd2dc30d7521baef9ce238d API recovered. https://status.veryfi.com/incident/123736 Wed, 05 Oct 2022 19:22:14 -0000 https://status.veryfi.com/incident/123736#d710113f2615b85731e1b9f24b49e3161d546dfdd5b44ce12a58e71da92e9f11 API went down. https://status.veryfi.com/incident/118626 Sat, 01 Oct 2022 06:44:10 +0000 https://status.veryfi.com/incident/118626#32f9d0b6f9897e07a144c464b31c23e4f40adb5c003154e2515adad5a95d0b3f Maintenance completed https://status.veryfi.com/incident/118626 Sat, 01 Oct 2022 06:44:10 -0000 https://status.veryfi.com/incident/118626#4955b803076a8ebc4f76d796f1cf910e497855bd12c1beb81ed3cd3f2ddc5aac We're conducting system maintenance and anticipate up to 15 minutes of downtime on October 1, 2022 between 06:00 and 07:00 UTC. https://status.veryfi.com/incident/118626 Sat, 01 Oct 2022 06:44:10 +0000 https://status.veryfi.com/incident/118626#32f9d0b6f9897e07a144c464b31c23e4f40adb5c003154e2515adad5a95d0b3f Maintenance completed https://status.veryfi.com/incident/118626 Sat, 01 Oct 2022 06:44:10 -0000 https://status.veryfi.com/incident/118626#4955b803076a8ebc4f76d796f1cf910e497855bd12c1beb81ed3cd3f2ddc5aac We're conducting system maintenance and anticipate up to 15 minutes of downtime on October 1, 2022 between 06:00 and 07:00 UTC. https://status.veryfi.com/incident/118626 Sat, 01 Oct 2022 06:27:31 -0000 https://status.veryfi.com/incident/118626#4955b803076a8ebc4f76d796f1cf910e497855bd12c1beb81ed3cd3f2ddc5aac We're conducting system maintenance and anticipate up to 15 minutes of downtime on October 1, 2022 between 06:00 and 07:00 UTC. https://status.veryfi.com/incident/118626 Sat, 01 Oct 2022 06:27:31 -0000 https://status.veryfi.com/incident/118626#4955b803076a8ebc4f76d796f1cf910e497855bd12c1beb81ed3cd3f2ddc5aac We're conducting system maintenance and anticipate up to 15 minutes of downtime on October 1, 2022 between 06:00 and 07:00 UTC. https://status.veryfi.com/incident/118626 Sat, 01 Oct 2022 06:00:00 -0000 https://status.veryfi.com/incident/118626#4955b803076a8ebc4f76d796f1cf910e497855bd12c1beb81ed3cd3f2ddc5aac We're conducting system maintenance and anticipate up to 15 minutes of downtime on October 1, 2022 between 06:00 and 07:00 UTC. https://status.veryfi.com/incident/118626 Sat, 01 Oct 2022 06:00:00 -0000 https://status.veryfi.com/incident/118626#4955b803076a8ebc4f76d796f1cf910e497855bd12c1beb81ed3cd3f2ddc5aac We're conducting system maintenance and anticipate up to 15 minutes of downtime on October 1, 2022 between 06:00 and 07:00 UTC. https://status.veryfi.com/incident/121418 Tue, 27 Sep 2022 04:31:06 -0000 https://status.veryfi.com/incident/121418#2da0233b35d48f6e854a2899509b2d539b39676e104688c9030e5a087f4705cd API Portal recovered. https://status.veryfi.com/incident/121418 Tue, 27 Sep 2022 04:28:28 -0000 https://status.veryfi.com/incident/121418#c9a8d287a6db4467205c7e8f9def1c46902791cf1f88c7aff9b0a347e1a237bc API Portal and Website went down. https://status.veryfi.com/incident/103808 Thu, 21 Jul 2022 22:37:00 -0000 https://status.veryfi.com/incident/103808#9fceb6ff006d06f433f8bc67c4103b3d68d1407741b358e083fe364fdcabb036 We have resolved the session issue in the API Portal. https://status.veryfi.com/incident/103808 Thu, 21 Jul 2022 22:28:00 -0000 https://status.veryfi.com/incident/103808#8ab57860069d2a0bbd027f0aede19ba36657f5d77b661f36d3806509987a15ab We have identified the problem and expect to have a resolution in 10 - 20 mins https://status.veryfi.com/incident/103808 Thu, 21 Jul 2022 22:08:00 -0000 https://status.veryfi.com/incident/103808#84a1b3928b0f69e51c781a940b3cea282f2968ef7e23edcca2b6c4fe8460d42d Sessions in the API Portal are currently timing out prematurely and users are being logged out as a result. This is not affecting API processing. We are working on a solution. https://status.veryfi.com/incident/87730 Thu, 12 May 2022 13:05:00 -0000 https://status.veryfi.com/incident/87730#e72e59a20ee8c37829bce73ed8c5d9d7a2e794d8ea816dd152627ca37ff3acdd API health checks failed however all services remain stable. No downtime has been suffered. https://status.veryfi.com/incident/87730 Thu, 12 May 2022 13:03:00 -0000 https://status.veryfi.com/incident/87730#27cedc315b67b38699bd2ca402fa534a6cb1f60a015c52324ffe3c8a37a8c51d API health checks have failed https://status.veryfi.com/incident/73500 Sat, 12 Mar 2022 06:07:33 +0000 https://status.veryfi.com/incident/73500#a9dc6c68d1cff431a76b7dfea2cf94210c47877a8215e52c42fe1531db04b089 Maintenance completed https://status.veryfi.com/incident/73500 Sat, 12 Mar 2022 06:07:33 -0000 https://status.veryfi.com/incident/73500#45ac1f86b445db5d59ddf28e67fc05f66a4594dd1f961af0f1163dcc7ff9391f We will be performing database maintenance and expect a short period of system interruption of up to 10 minutes during the maintenance window. During this time, requests to the API will return HTTP status 500 errors. https://status.veryfi.com/incident/73500 Sat, 12 Mar 2022 06:07:33 +0000 https://status.veryfi.com/incident/73500#a9dc6c68d1cff431a76b7dfea2cf94210c47877a8215e52c42fe1531db04b089 Maintenance completed https://status.veryfi.com/incident/73500 Sat, 12 Mar 2022 06:07:33 -0000 https://status.veryfi.com/incident/73500#45ac1f86b445db5d59ddf28e67fc05f66a4594dd1f961af0f1163dcc7ff9391f We will be performing database maintenance and expect a short period of system interruption of up to 10 minutes during the maintenance window. During this time, requests to the API will return HTTP status 500 errors. https://status.veryfi.com/incident/73500 Sat, 12 Mar 2022 06:00:00 -0000 https://status.veryfi.com/incident/73500#45ac1f86b445db5d59ddf28e67fc05f66a4594dd1f961af0f1163dcc7ff9391f We will be performing database maintenance and expect a short period of system interruption of up to 10 minutes during the maintenance window. During this time, requests to the API will return HTTP status 500 errors. https://status.veryfi.com/incident/73500 Sat, 12 Mar 2022 06:00:00 -0000 https://status.veryfi.com/incident/73500#45ac1f86b445db5d59ddf28e67fc05f66a4594dd1f961af0f1163dcc7ff9391f We will be performing database maintenance and expect a short period of system interruption of up to 10 minutes during the maintenance window. During this time, requests to the API will return HTTP status 500 errors. https://status.veryfi.com/incident/69176 Thu, 03 Feb 2022 15:15:00 -0000 https://status.veryfi.com/incident/69176#3e3fbca8d9891e73376a068f2732a9e7702fb3c47f2cfb4760d26d7bf7454447 We have identified and resolved the root cause. Services are now back up to normal speeds. Veryfi's OCR API system performance was degraded for approximately four (4) hours. Processing time increased by an average of ~15 seconds per document beginning at 11:23 UTC. Significant improvements were made by 14:15 UTC and full service was restored at 15:14 UTC. The root cause was identified and resolved to prevent a future recurrence. https://status.veryfi.com/incident/69176 Thu, 03 Feb 2022 14:16:00 -0000 https://status.veryfi.com/incident/69176#b73904f48705fd6bbff3c1c9d489177890d5a6aad5e22613bac0661e22bc6b3e We have identified the root cause of the issue and are working towards a prompt resolution. Significant improvements have already been made with full restoration expected in the next hour. https://status.veryfi.com/incident/69176 Thu, 03 Feb 2022 11:24:00 -0000 https://status.veryfi.com/incident/69176#9dfe187745d87bec0a6db3ccdbc95fa136c8882468fd77d08c1c4c90b5c3bf40 We are experiencing network issues that are causing a slow down in some document processing requests. https://status.veryfi.com/incident/62552 Wed, 15 Dec 2021 16:00:57 -0000 https://status.veryfi.com/incident/62552#669ec765a76704b8b129012b5afc8854594405bafed741a68cc4de683702e597 API and API Portal recovered. https://status.veryfi.com/incident/62552 Wed, 15 Dec 2021 16:00:57 -0000 https://status.veryfi.com/incident/62552#669ec765a76704b8b129012b5afc8854594405bafed741a68cc4de683702e597 API and API Portal recovered. https://status.veryfi.com/incident/62552 Wed, 15 Dec 2021 15:28:00 -0000 https://status.veryfi.com/incident/62552#67595608aaaa414dcb798341e5e62eafa0192e82409d236279c6864900f49ea9 API, API Portal, and Website are experiencing degraded performance. We are currently investigating. https://status.veryfi.com/incident/62552 Wed, 15 Dec 2021 15:28:00 -0000 https://status.veryfi.com/incident/62552#67595608aaaa414dcb798341e5e62eafa0192e82409d236279c6864900f49ea9 API, API Portal, and Website are experiencing degraded performance. We are currently investigating.