config.json

Overview

config.json defines how to perform the test execution. It is common to have multiple configuration files when performing tests.

See the information below to manage the configuration file successfully.

Required Fields

urls

See documentation for more info.

{
    "oauth": "https://auth.crt.nuance.com/oauth2/token",
    "asr": "asr.api.nuance.com:443",
    "nlu": "nlu.api.nuance.com:443"
},

credentials

Default value for OAuth is enabled. set to false for On Premise testing or outside of Mix

Scopes required: asr and nlu.

The client ID includes the appID: prefix.

See documentation for more info.

{
    "oauth_enabled": true,
    "oauth_scopes": "asr nlu",
    "client_id": "appID:NMDPTRIAL_x_nuance_com_20200224T203206802515",
    "client_secret": "...-..."
}

Optional Fields

log_level

Options: DEBUG, INFO, WARN, and ERROR.

requests_concurrent

Default of 1, maxmium of 5.

asr

If null, ASR will not be executed.

nlu

If null, NLU will not be executed.

Sample config.json

{
    "description": "Test Set #1, Execution Run #42",
    "log_level": "INFO",
    "language": "eng-USA",
    "requests_concurrent": 2,
    "urls": {
        "oauth": "https://auth.crt.nuance.com/oauth2/token",
        "asr": "asr.api.nuance.com:443",
        "nlu": "nlu.api.nuance.com:443"
    },
    "credentials": {
        "oauth_scopes": "asr nlu",
        "client_id": "appID:NMDPTRIAL_x_nuance_com_20200224T203206802515",
        "client_secret": "...-..."
    },
    "asr": {
        "topic": "GEN",
        "auto_punctuate": false,
        "dlm_uri": "urn:nuance-mix:tag:model/.../mix.asr?=language=eng-USA",
        "dlm_weight": 0.7,
        "wordset_path": null,
        "wordset_uri": null,
		"inline_wordset_weight": 0.1,
		"compiled_wordset_weight": 0.1,
		"builtins": null,
		"builtins_weight": 0.25,
        "timeout": 5,
        "retry_count": 2,
        "retry_sleep_s": 5,
        "discard_speaker_adaptation": false,
        "asr_settings_uri": null,
        "utterance_detection_mode": "DISABLED",
        "speech_detection_sensitivity": 0.5,
        "utterance_end_silence_ms": 0,
        "formatting_options": {},
        "formatting_scheme": "date",
        "wakeup_word": "Hey Dragon",
        "filter_wakeup_word": true,
        "use_speaker_profile": true,
		"use_character_error_rate": false,
		"score_against_mft": false
    },
    "nlu": {
        "use_asr_results": true,
        "model_uri": "urn:nuance-mix:tag:model/.../mix.nlu?=language=eng-USA",
        "wordset_path": null,
        "wordset_uri": null,
        "timeout": 15,
        "input_logging_mode": "SUPPRESSED",
        "retry_count": 2,
        "retry_sleep_s": 5,
        "nBest": false,
        "predicted_entities_to_ignore": {},
        "format_calendar_canonicals": false,
        "acceptance_confidence_threshold": null,
        "confirmation_confidence_threshold": null,
        "output_full_entity_hierarchy": null,
        "flattened_predefined_output": false,
        "useEntityF1": false,
        "subtotal_intents": {
            "ID": [],
            "OOD": []
        }
    }
}

Properties

Property	Type	Description
description	String	Describe the test run, so that there is some ability to trace-back to your execution
log_level	String	For execution output
language	String	6-character code used in the execution
requests_concurrent	Integer	Number of parallel runs to split the test set up into for faster execution time
urls	Object	oauth for auth server, asr and nlu for services
credentials	Object	oauth_scopes are used to define access to services; client_id and client_secret are obtained from the Mix dashboard
asr	Object	Properties for ASR execution, see below. If null, does not perform ASR.
nlu	Object	Properties for NLU execution, see below. If null, does not perform NLU.

ASR Properties

Property	Type	Description
topic	String	Language topic to use for execution, ie. GEN, DTV, ..
auto_punctuate	Boolean	Whether or not to use auto punctuation
dlm_uri	String or List	Reference DLM(s) to exercise; if empty, will not use DLM
dlm_weight	Number or List	Weight to apply to DLM(s)
wordset_path	String	Reference to an inline wordset containing dynamic entities (for personalization)
wordset_uri	String or List	Reference to one or more compiled wordset resource containing dynamic entities (for personalization)
inline_wordset_weight	Number	Weight to apply to the inline wordset if specified. Default 0.1.
compiled_wordset_weight	Number or List	Weight to apply to the compiled wordset(s) if specified. Default 0.1.
builtins	String or List	Builtin recognition object(s) to use
builtins_weight	Number or List	Weight(s) to use for ASR builtins if specified. Default 0.25.
timeout	Number	gRPC timeout for execution
suppress_call_recording	Boolean	Whether to disable call recording (i.e. logging); defaults to true.
retry_count	Number	Number of retry attempts. Default 2. Max 4.
retry_sleep_s	Number	Number of seconds to sleep before retry-attempt. Default 5. Max 30.
utterance_detection_mode	Enum	How the end of the utterance in the test case audio is determined ; default is DISABLED . See EnumUtteranceDetectionMode for more information
asr_settings_uri	String	Optional reference to the ASR settings for a Mix project; used to set the ASR data pack version explicitly. See ResourceReference in the ASRaaS gRPC API documentation
discard_speaker_adaptation	Boolean	Whether or not to adapt the speaker profile(s) based on the audio from the test cases; default is false. See RecognitionFlags in the ASRaaS gRPC API documentation for additional details
speech_detection_sensitivity	Number	A balance between detecting speech and noise (breathing, etc.), 0 to 1. 0 means ignore all noise, 1 means interpret all noise as speech; default is 0.5. See RecognitionParameters in the ASRaaS gRPC API documentation for additional details
utterance_end_silence_ms	Number	Minimum silence, in milliseconds, that determines the end of a sentence. Default (0) means server default, usually 500ms or half a second. See RecognitionParameters in the ASRaaS gRPC API documentation for additional details
formatting_scheme	String	How to preferetially interpret ambiguous numbers; default depends on the data pack. See FormattingScheme in the ASRaaS gRPC API documentation for more information
formatting_options	Dictionary	A dictionary of key/value pairs that specify how to display words and numbers in the formattedText result field; these supplement (and can override) parameters that belong to the formatting_scheme described above. See FormattingOptions in the ASRaaS gRPC API documentation for additional information
wakeup_word	String or List	A phrase or list of phrases to be treated as wakeup words. See Wakeup Words in the ASRaaS gRPC API documentation for additional information
filter_wakeup_word	Boolean	Specifies whether or not any wakeup word phrases should be filtered out from the ASR transcription. See "Filter Wakeup Words" under the Wakeup Words section in the ASRaaS gRPC API documentation for additional information
use_speaker_profile	Boolean	Defaults to true, in which case speaker profiles are created and/or updated according to the speaker IDs for each test case given in the truth file. By setting this parameter to false, speaker profiles will not be created and/or updated (see speaker profiles for more information)
use_character_error_rate	Boolean	Defaults to false. If true, the ASR hypothesis will be compared to the reference transcription in the truth file character by character instead of word by word. See Character Error Rate for more information.
score_against_mft	Boolean	Defaults to false. If true, the minimally formatted text from the ASR hypothesis will be compared to the reference transcription in the truth file instead of the formatted text. See the Minimally Formatted Text section for more detail.

NLU Properties

Property	Type	Description
use_asr_results	Boolean	Leverages the ASR Result vs the Text Input
model_uri	String	Reference the NLU model
wordset_path	String	Reference to an inline wordset containing any dynamic entities (personalization)
wordset_uri	String or List	Reference to one or more compiled wordset resource(s) containing (personalization)
timeout	Number	gRPC timeout for execution
input_logging_mode	String	Logging mode, either PLAINTEXT Or SUPPRESSED; defaults to SUPPRESSED.
retry_count	Number	Number of retry attempts. Default 2. Max 4.
retry_sleep_s	Number	Number of seconds to sleep before retry-attempt. Default 5. Max 30.
nBest	Boolean	When set to true, if any of the NLU intepretation results match the reference in the truth value, that test case will be counted as correct for NLU scoring purposes. Default value is false. See the nBest advanced NLU guide for more details
predicted_entities_to_ignore	Dictionary	Regular expression rules that can be used to ignore specific entities during scoring. See ignoring predicted entities for details
format_calendar_canonicals	Boolean	Defaults to false. If true, the tool will format nuance_CALENDARX entity output for supported values and ranges. See the CALENDARX section of the entity evaluation NLU advanced guide
acceptance_confidence_threshold	Number	A number between 0.0 and 1.0. See the Correct Acceptance and Correct Confirmation section for more info. If specified, the confirmation_confidence_threshold must also be given
confirmation_confidence_threshold	Number	A number between 0.0 and 1.0. See the Correct Acceptance and Correct Confirmation section for more info. If specified, the acceptance_confidence_threshold must also be given
output_full_entity_hierarchy	Boolean	Determines how hierarchical entities are formatted in the output. See the Hierarchical Entity Support section for more info. Default value is false.
flattened_predefined_output	Boolean	If false, predefined entity output is not flattened and the nested dictionary structure returned by NLUaaS is preserved in the output. See entity evaluation for more info. Default value is true.
useEntityF1	Boolean	If true, precision, recall, and F1 scores for entities will be included in the .html report ourput. Default value is false.
subtotal_intents	Dictionary	This can be used to specify intents that belong to in-domain and out-of-domain groups. Use the ID key for in-domain intents, and the OOD key for out-of-domain intents. The value for each key should be a list with the individual intents given as list elements. If in-domain and out-of-domain intents are specified, additional output will be provided as described in Confidence and Intent Subtotaling.