Resolved
Updated

At approximately 7:40 am EDT on Thursday, July 6, 2023, we experienced a primary and secondary system failure that had the following impacts:

  • Customers who utilize our call recording service were unable to place outbound calls
  • Our Dynamic Caller ID (DCID) service became unavailable for outbound calls
  • Customers could no longer access My.Vaspian

To resolve the inability to make outbound calls, as customers gave us permission, we modified the way their calls route through our system. This resolved the outbound calling issue, but eliminated the ability for calls to be recorded through our customer accessible call recording service. Any calls that were recorded prior to this failure were unaffected.

The following day, both outbound and inbound phone calls were terminating as expected. Our engineering team continued to work around the clock to restore access to our call recording service. This effort took an extended amount of time due to the enormous amount of data that we record and store.

As of Sunday morning, July 9th, all services were restored.

In the process of restoring services, we migrated the impacted database to a more resilient environment that will drastically reduce the likelihood of this type of incident occurring in the future. We've also created a disaster recovery plan that includes an off site instance of this database that will be available to us in the unlikely event that we do have a failure.

We truly appreciate the patience and support that our customers showed us throughout this event.

Avatar for Denise Carlson
Denise Carlson
Resolved

We completed all planned work over the weekend. All services were restored as of Sunday morning, July 9th.

We will be issuing a formal Reason For Outage (RFO) on Thursday, July 13th.

It is important that you let us know if you are having any issues. Please contact our support team at support@vaspian.com or by calling 716-961-2120 option 3.

Again, we value you as our customer and appreciate your support.

Avatar for Denise Carlson
Denise Carlson
Recovering

All planned work for this weekend has been completed. As of Sunday morning, July 9th, all services have been restored.

We will have a formal Reason For Outage (RFO) available on Thursday, July 13th.

If you are experiencing any issues, please let us know by contacting our support team at support@vaspian.com or by calling 716-961-2120 option 3.

We value you as a customer and thank you for your support.

Avatar for Denise Carlson
Denise Carlson
Identified

We have an important update regarding our call recording service.

  • Yesterday morning, we suffered a major failure with our call recording service.

  • Service has improved from yesterday and both outbound and inbound phone calls are terminating as expected, however, our ability to record calls is still impaired and customer access to our call recording service remains down.

  • This work is taking an extended amount of time to resolve, due to the enormous amount of data that needs to be preserved. All historical call recording is safe and unaffected.

  • Our Dynamic Caller ID (DCID) service and access to My.Vaspian are still unavailable.

  • Our team is working around the clock to restore these services for Monday.

If you are currently having issues with outbound or inbound phone calls, please contact us at support@vaspian.com or by calling our support team at 716-961-2120 option 3.

We understand how important these services are to your business and appreciate your patience.

Avatar for Denise Carlson
Denise Carlson
Updated

At approximately 7:40 am EDT on Thursday, July 6, 2023, we experienced a primary and secondary system failure that had the following impacts:

  • Customers who utilize our call recording service were unable to place outbound calls
  • Our Dynamic Caller ID (DCID) service became unavailable for outbound calls
  • Customers could no longer access My.Vaspian

To resolve the inability to make outbound calls, as customers gave us permission, we modified the way their calls route through our system. This resolved the outbound calling issue, but eliminated the ability for calls to be recorded through our customer accessible call recording service. Any calls that were recorded prior to this failure have been preserved.

Our Engineering and Leadership teams are on site and fully engaged in working to resolve these issues. We will continue to provide updates as we have them.

We recognize the disruption that this caused for our impacted customers. We appreciate your patience as we continue to focus on implementing the solution.

Avatar for Denise Carlson
Denise Carlson
Investigating

Some people are experiencing problems with our voice service right now. We're so sorry about that. Please standby for further updates, and we'll be in touch as soon as we know more.

Avatar for
Began at:

Affected components
  • Voice Services