How to create an IVR experience test using TestIVR

In this post we are going to develop an IVR experience test using TestIVR

In the previous article I discussed how you can implement an IVR feature test using TestIVR API. In this post I am going to discuss IVR Experience Testing and how you can implement it using TestIVR.

Experience testing is testing the Customer Experience of the IVR system 24/7. Basically test calls should be made at regular intervals to make sure everything is working as expected with the IVR system all the time. In short we monitor the availability, functionality, and performance of the IVR system using experience testing. The IVR system should be tested on regular basis (i.e. daily/hourly/every X minutes) with alerts set for issues that being identified. So let's discuss how we can implement an IVR experience test using TestIVR.

After signing up the TestIVR web portal will provide you with an "API key" which can be used to call the TestIVR RESTful API. Note that at this moment you can only communicate with TestIVR feature testing tool through our API. Using the API key you can call the TestIVR experince testing tool via different tools such as "curl", python libraries such as "requests" or tools such as postman .

The TestIVR API end point is https://api.testivr.com/. Keep in mind that in all your API calls to TestIVR API endpoint, you should add your "API key" in a Header called "Authorization" and you need to add "Api-Key " at its begining. So for instance in "curl" you will have something like (replace XXXXX with your "API key"):

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Api-Key XXXXX" \
  -d '{
    "description":"sample path"
  }' https://api.testivr.com/path/
                

Or in python:

import requests

headers = {
    'Content-Type': 'application/json',
    'Authorization': 'Api-Key XXXXX',
}

json_data = {
    'description': 'sample path',
}

response = requests.post('https://api.testivr.com/path/', headers=headers, json=json_data)
                

To develop an IVR experience test using TestIVR, similar to implementing a feature test you should start with creating a "path". As discussed in the previous article, a "path" is basically a route for your test to follow. Later you will add "action"s (such as pressing a button or saying something to the IVR) to this path. To create a "path" you simply need to submit a POST request to /path/ endpoint and pass a description as part of the payload. For example:

curl --request POST \
  -k \
  --url https://api.testivr.com/path/ \
  --header 'Authorization: Api-Key XXXXX' \
  --header 'Content-Type: application/json' \
  --data '{
  "description": "path to sales"
}'
                

The API will response with something like this:

{
  "status": "OK",
  "data": {
    "id": 3,
    "created_at": "2022-10-24T23:15:41.262451Z",
    "description": "path to sales",
    "user": 1
  }
}
              

What is important from above is "id": 6 part which basically tells us what is the id of the new path we just created. You can read more about the API request for adding "path" in the Add Path section of our document.

The next step, similar to implementing a feature test, is adding "action"s to our new "path". As discussed in the previous article, actions are the communications units we define between the TestIVR and the IVR system we are testing. For instance after TestIVR calling the IVR system, if we need TestIVR to press a button that is an "action" that we can define on the test "path". Or if the IVR is going to tell something to the caller and we want the TestIVR to evaluate what it said, that is another "action" we can add to the "path". To "create" an "action" you need to submit a POST request to /action/ endpoint and pass path, order, type, value and description as part of the payload.

In our example we want TestIVR to first listen to the IVR system and evaluate the intro:

curl --request POST \
  -k \
  --url https://api.testivr.com/action/ \
  --header 'Authorization: Api-Key XXXXX' \
  --header 'Content-Type: application/json' \
  --data '{
  "description": "listening to the ivr intro",
  "path": 3,
  "order": 0,
  "type": "listen",
  "value": "Hello, how can we direct your call? Press 1 for sales, or say sales. To reach support, press 2 or say support.",
  "asr_threshold": 0.9,
  "pause_before": 0.0,
  "pause_after": 1.0
}'
                

In this call, path should be set to the id of the "path" we want to add this action to. In our case we want to add it to the same path we created in our previous call to the API, which was "id": 3, so we set "path": 6. We also have "order": 0 in above meaning this is the first action that TestIVR should run on this path. So TestIVR runs the actions through the IVR system one by one based on their order.

We have set the action type to listen meaning we want TestIVR to listen to the IVR system. TestIVR is using an ASR (Automated Speech Recognition) model to convert the IVR responses to text, and then compares the result with text provided in the value field above. TestIVR assumes the outcome is valid if the match ratio between the ASR result and the provided text in value is greater than the threshold defined in asr_threshold field.

We also have other types that you can define for an action:

listen: listentning to the IVR response, transcribing it and compare the transcription with provided text defined in the value field.
say: saying the expression defined in the value field to the IVR
press: pressing the number provided in the value field
command: running the command provide in the value field

At this moment the only available command is terminate, which ends the call.

The pause_before and pause_after are the amount of time (in seconds) tha TestIVR should wait before triggering the next action. By default these values are set to zero, but if you have a special case that needs some delay between each trigger you can set that up using these fields. In our case we are setting "pause_after": 1.0 meaning we want TestIVR to wait for a second before running the next action.

Finally description is a field for providing a brief description of the action, which is provided only for documenting purposes.

Next we want TestIVR to "press 1" on the keypad (to redirect to sales). So we submit the following request this time:

curl --request POST \
  -k \
  --url https://api.testivr.com/action/ \
  --header 'Authorization: Api-Key XXXXX' \
  --header 'Content-Type: application/json' \
  --data '{
  "description": "press 1 for sales",
  "path": 3,
  "order": 1,
  "type": "press",
  "value": "1"
}'
              

In this action we set the "order": 1 meaning this one should run after the previous action that had "order": 0. Also the type is "press" this time and its value is set to "1", meaning we are asking TestIVR to press 1 on the keypad. Next we need to have add another action to evaluate the IVR response:

curl --request POST \
  -k \
  --url https://api.testivr.com/action/ \
  --header 'Authorization: XXXXX' \
  --header 'Content-Type: application/json' \
  --data '{
  "description": "listening to the prompt",
  "path": 3,
  "order": 2,
  "type": "listen",
  "value": "You asked for sales, please hold",
  "asr_threshold": 0.9,
  "pause_before": 0,
  "pause_after": 1.0
}'
              

This time we set the order to "2", type to "listen" and value to the prompt we expect from the IVR. We have also set the pause_after to "1.0", meaning that TestIVR will wait for a second after this action and before running the next action. Finally, we are going to add the final actoin which is going to terminate the call.

curl --request POST \
  -k \
  --url https://api.testivr.com/action/ \
  --header 'Authorization: Api-Key W9tSy60v.5bcKaoutBcU8XwMRGTYKmvgTIbM9GtuB' \
  --header 'Content-Type: application/json' \
  --data '{
  "description": "terminating the call",
  "path": 3,
  "order": 3,
  "type": "command",
  "value": "terminate"
}'
              

For this last action we set the order to "3", the type to "command" and value to "terminate".

The next step is to setup the experience test using the defined path. You can do that by submitting a POST request to /experience/test/ endpoint. You need to pass:

path: the path you want to use for your test.
phone: the phone number of the IVR you want to run the test on
frequency: the frequency of running the test, this can be set to: daily for daily run, hourly for hourly run and thirty_minutes for running it every 30 minutes
notify_email: your email for getting notification in case the test fails.

For example:

curl -X POST \
  -k \
  --url https://api.testivr.com/experience/test/ \
  --header "Content-Type: application/json" \
  --header "Authorization: Api-Key XXXXX" \
  --data '{
      "name": "sales path experience test ",
      "description":"experience test for sales path",
      "path": 3,
      "frequency": "daily",
      "phone": "16506839455",
      "notify_email": "your.email@domain.com"
  }'
                

And the result will be like this:

{
  "status": "OK",
  "data": {
    "id": 2,
    "created_at": "2022-10-25T00:03:16.881647Z",
    "description": "experience test for sales path",
    "name": "sales path experience test",
    "phone": "16506839455",
    "frequency": "daily",
    "notify_email": "your.email@domain.com",
    "user": 1,
    "path": 3
  }
}
                

From above you need to keep in mind what is the id for your new test, which in this case is 2.

After setting up the experience test, it will run automatically based on the frequency you have set. Each run will have a separate outcome, a run can fail due to various reason such as IVR system declining the call, or the promopt doesn't match the expected values. If a run fails, you will get notified by email through provided notify_email.

If you need to pull the outcomes you need to first pull the test id, to do that you can submit a GET request to /experience/run/ endpoint:

curl -X GET \
 -k \
 --url https://api.testivr.com/experience/run/ \
 --header "Content-Type: application/json" \
 --header "Authorization: Api-Key XXXXX"
                

The output goning to be like this:

{
  "count":6,
  "next":null,
  "previous":null,
  "results":[
    {
      "id":8,
      "created_at":"2022-10-25T00:56:25.693036Z",
      "user":1,
      "experience_test":2,
      "test":9
    },
    {
      "id":7,
      "created_at":"2022-10-25T00:54:18.468378Z",
      "user":1,
      "experience_test":2,
      "test":8
    },
    ...
  ]
}
              

You see for the experience_test with id "2" we have two results, meaning that code ran twice until now. To see the outcomes of the last run, that has test id of "9" for instance, you can submit the following request:

curl -X GET \
  -k \
  --url https://api.testivr.com/outcome/?test=9 \
  --header "Content-Type: application/json" \
  --header "Authorization: Api-Key XXXXX"
              

Note that we are filtering the outcomes based on the test id by adding ?test=9 to the request. The result will be like:

{
  "count":2,
  "next":null,
  "previous":null,
  "results":[
    {
      "id":8,
      "created_at":"2022-10-25T00:56:39.495090Z",
      "outcome":"match",
      "value":"hello how can we direct your call press one for sales or say sales to reach support press two or say support",
      "expected_value":"Hello, how can we direct your call? Press 1 for sales, or say sales. To reach support, press 2 or say support.",
      "match_ratio":0.908256880733945,
      "user":1,
      "test":9,
      "action":3
    },
    {
      "id":9,
      "created_at":"2022-10-25T00:56:49.767187Z",
      "outcome":"match",
      "value":"you asked for sales please hold",
      "expected_value":"You asked for sales, please hold",
      "match_ratio":0.9523809523809523,
      "user":1,
      "test":9,
      "action":5
    }
    ]
  }
              

In the response, each items of the resuls shows the outcome of one of the "actions" we had on the "path". We can see the "action" id in the action field of the response. The field outcome is important when your action had a "type" of "listen" and the outcome shows whether the response of the IVR is a "match" to the "value" you provided as the expected response.

The field match_ratio is also closely related to the outcome. Basically it shows the closeness of the (transcribed) IVR response, which is provided above as value to the expected result, provide as expected_value. If the calcuated match_ratio is more than the asr_threshold we have provided for the "action", the outcome is going to be "match".

You can listen to the audio file of the example we provided in this article below:

This article shows how you can create a simple experince test for IVR using TestIVR. Please let me know if you have any question through our email: support@testivr.com

We also provide a good document on our API which provides more detailed information on all the calls you can make to TestIVR.