config.json
Overview​
config.json defines how to perform the test execution. It is common to have multiple configuration files when performing tests.
See the information below to manage the configuration file successfully.
Required Fields​
urls​
See documentation for more info.
{
"oauth": "https://auth.crt.nuance.com/oauth2/token",
"asr": "asr.api.nuance.com:443",
"nlu": "nlu.api.nuance.com:443"
},
credentials​
Default value for OAuth is enabled. set to false for On Premise testing or outside of Mix
Scopes required: asr and nlu.
The client ID includes the appID: prefix.
See documentation for more info.
{
"oauth_enabled": true,
"oauth_scopes": "asr nlu",
"client_id": "appID:NMDPTRIAL_x_nuance_com_20200224T203206802515",
"client_secret": "...-..."
}
Optional Fields​
log_level​
Options: DEBUG, INFO, WARN, and ERROR.
requests_concurrent​
Default of 1, maxmium of 5.
asr​
If null, ASR will not be executed.
nlu​
If null, NLU will not be executed.
Sample config.json​
{
"description": "Test Set #1, Execution Run #42",
"log_level": "INFO",
"language": "eng-USA",
"requests_concurrent": 2,
"urls": {
"oauth": "https://auth.crt.nuance.com/oauth2/token",
"asr": "asr.api.nuance.com:443",
"nlu": "nlu.api.nuance.com:443"
},
"credentials": {
"oauth_scopes": "asr nlu",
"client_id": "appID:NMDPTRIAL_x_nuance_com_20200224T203206802515",
"client_secret": "...-..."
},
"asr": {
"topic": "GEN",
"auto_punctuate": false,
"dlm_uri": "urn:nuance-mix:tag:model/.../mix.asr?=language=eng-USA",
"dlm_weight": 0.7,
"wordset_path": null,
"wordset_uri": null,
"inline_wordset_weight": 0.1,
"compiled_wordset_weight": 0.1,
"builtins": null,
"builtins_weight": 0.25,
"timeout": 5,
"retry_count": 2,
"retry_sleep_s": 5,
"discard_speaker_adaptation": false,
"asr_settings_uri": null,
"utterance_detection_mode": "DISABLED",
"speech_detection_sensitivity": 0.5,
"utterance_end_silence_ms": 0,
"formatting_options": {},
"formatting_scheme": "date",
"wakeup_word": "Hey Dragon",
"filter_wakeup_word": true,
"use_speaker_profile": true,
"use_character_error_rate": false,
"score_against_mft": false
},
"nlu": {
"use_asr_results": true,
"model_uri": "urn:nuance-mix:tag:model/.../mix.nlu?=language=eng-USA",
"wordset_path": null,
"wordset_uri": null,
"timeout": 15,
"input_logging_mode": "SUPPRESSED",
"retry_count": 2,
"retry_sleep_s": 5,
"nBest": false,
"predicted_entities_to_ignore": {},
"format_calendar_canonicals": false,
"acceptance_confidence_threshold": null,
"confirmation_confidence_threshold": null,
"output_full_entity_hierarchy": null,
"flattened_predefined_output": false,
"useEntityF1": false,
"subtotal_intents": {
"ID": [],
"OOD": []
}
}
}
Properties​
| Property | Type | Description |
|---|---|---|
description | String | Describe the test run, so that there is some ability to trace-back to your execution |
log_level | String | For execution output |
language | String | 6-character code used in the execution |
requests_concurrent | Integer | Number of parallel runs to split the test set up into for faster execution time |
urls | Object | oauth for auth server, asr and nlu for services |
credentials | Object | oauth_scopes are used to define access to services; client_id and client_secret are obtained from the Mix dashboard |
asr | Object | Properties for ASR execution, see below. If null, does not perform ASR. |
nlu | Object | Properties for NLU execution, see below. If null, does not perform NLU. |
ASR Properties​
| Property | Type | Description |
|---|---|---|
topic | String | Language topic to use for execution, ie. GEN, DTV, .. |
auto_punctuate | Boolean | Whether or not to use auto punctuation |
dlm_uri | String or List | Reference DLM(s) to exercise; if empty, will not use DLM |
dlm_weight | Number or List | Weight to apply to DLM(s) |
wordset_path | String | Reference to an inline wordset containing dynamic entities (for personalization) |
wordset_uri | String or List | Reference to one or more compiled wordset resource containing dynamic entities (for personalization) |
inline_wordset_weight | Number | Weight to apply to the inline wordset if specified. Default 0.1. |
compiled_wordset_weight | Number or List | Weight to apply to the compiled wordset(s) if specified. Default 0.1. |
builtins | String or List | Builtin recognition object(s) to use |
builtins_weight | Number or List | Weight(s) to use for ASR builtins if specified. Default 0.25. |
timeout | Number | gRPC timeout for execution |
suppress_call_recording | Boolean | Whether to disable call recording (i.e. logging); defaults to true. |
retry_count | Number | Number of retry attempts. Default 2. Max 4. |
retry_sleep_s | Number | Number of seconds to sleep before retry-attempt. Default 5. Max 30. |
utterance_detection_mode | Enum | How the end of the utterance in the test case audio is determined ; default is DISABLED . See EnumUtteranceDetectionMode for more information |
asr_settings_uri | String | Optional reference to the ASR settings for a Mix project; used to set the ASR data pack version explicitly. See ResourceReference in the ASRaaS gRPC API documentation |
discard_speaker_adaptation | Boolean | Whether or not to adapt the speaker profile(s) based on the audio from the test cases; default is false. See RecognitionFlags in the ASRaaS gRPC API documentation for additional details |
speech_detection_sensitivity | Number | A balance between detecting speech and noise (breathing, etc.), 0 to 1. 0 means ignore all noise, 1 means interpret all noise as speech; default is 0.5. See RecognitionParameters in the ASRaaS gRPC API documentation for additional details |
utterance_end_silence_ms | Number | Minimum silence, in milliseconds, that determines the end of a sentence. Default (0) means server default, usually 500ms or half a second. See RecognitionParameters in the ASRaaS gRPC API documentation for additional details |
formatting_scheme | String | How to preferetially interpret ambiguous numbers; default depends on the data pack. See FormattingScheme in the ASRaaS gRPC API documentation for more information |
formatting_options | Dictionary | A dictionary of key/value pairs that specify how to display words and numbers in the formattedText result field; these supplement (and can override) parameters that belong to the formatting_scheme described above. See FormattingOptions in the ASRaaS gRPC API documentation for additional information |
wakeup_word | String or List | A phrase or list of phrases to be treated as wakeup words. See Wakeup Words in the ASRaaS gRPC API documentation for additional information |
filter_wakeup_word | Boolean | Specifies whether or not any wakeup word phrases should be filtered out from the ASR transcription. See "Filter Wakeup Words" under the Wakeup Words section in the ASRaaS gRPC API documentation for additional information |
use_speaker_profile | Boolean | Defaults to true, in which case speaker profiles are created and/or updated according to the speaker IDs for each test case given in the truth file. By setting this parameter to false, speaker profiles will not be created and/or updated (see speaker profiles for more information) |
use_character_error_rate | Boolean | Defaults to false. If true, the ASR hypothesis will be compared to the reference transcription in the truth file character by character instead of word by word. See Character Error Rate for more information. |
score_against_mft | Boolean | Defaults to false. If true, the minimally formatted text from the ASR hypothesis will be compared to the reference transcription in the truth file instead of the formatted text. See the Minimally Formatted Text section for more detail. |
NLU Properties​
| Property | Type | Description |
|---|---|---|
use_asr_results | Boolean | Leverages the ASR Result vs the Text Input |
model_uri | String | Reference the NLU model |
wordset_path | String | Reference to an inline wordset containing any dynamic entities (personalization) |
wordset_uri | String or List | Reference to one or more compiled wordset resource(s) containing (personalization) |
timeout | Number | gRPC timeout for execution |
input_logging_mode | String | Logging mode, either PLAINTEXT Or SUPPRESSED; defaults to SUPPRESSED. |
retry_count | Number | Number of retry attempts. Default 2. Max 4. |
retry_sleep_s | Number | Number of seconds to sleep before retry-attempt. Default 5. Max 30. |
nBest | Boolean | When set to true, if any of the NLU intepretation results match the reference in the truth value, that test case will be counted as correct for NLU scoring purposes. Default value is false. See the nBest advanced NLU guide for more details |
predicted_entities_to_ignore | Dictionary | Regular expression rules that can be used to ignore specific entities during scoring. See ignoring predicted entities for details |
format_calendar_canonicals | Boolean | Defaults to false. If true, the tool will format nuance_CALENDARX entity output for supported values and ranges. See the CALENDARX section of the entity evaluation NLU advanced guide |
acceptance_confidence_threshold | Number | A number between 0.0 and 1.0. See the Correct Acceptance and Correct Confirmation section for more info. If specified, the confirmation_confidence_threshold must also be given |
confirmation_confidence_threshold | Number | A number between 0.0 and 1.0. See the Correct Acceptance and Correct Confirmation section for more info. If specified, the acceptance_confidence_threshold must also be given |
output_full_entity_hierarchy | Boolean | Determines how hierarchical entities are formatted in the output. See the Hierarchical Entity Support section for more info. Default value is false. |
flattened_predefined_output | Boolean | If false, predefined entity output is not flattened and the nested dictionary structure returned by NLUaaS is preserved in the output. See entity evaluation for more info. Default value is true. |
useEntityF1 | Boolean | If true, precision, recall, and F1 scores for entities will be included in the .html report ourput. Default value is false. |
subtotal_intents | Dictionary | This can be used to specify intents that belong to in-domain and out-of-domain groups. Use the ID key for in-domain intents, and the OOD key for out-of-domain intents. The value for each key should be a list with the individual intents given as list elements. If in-domain and out-of-domain intents are specified, additional output will be provided as described in Confidence and Intent Subtotaling. |