Enhancements​

• Added a rate limit system to prevent server from blocking requests
• Included a ROC curve in the report
• Added a run.py wrapper

ASRaaS functionality and params exposed​

• inline_wordset_weight
• compiled_wordset_weight
• builtins
• builtins_weight
• suppress_call_recording
• speech_detection_sensitivity
• utterance_end_silence_ms
• use_character_error_rate
• score_against_mft

NLUaaS functionality and params exposed​

• output_full_entity_hierarchy
• flattened_predefined_output
• useEntityF1
• subtotal_intents

Evaluation functionalities​

• Added support for TRSX input file
• Added a tool to convert TRSX to TSV (nuance.conversion)
  • The flag --excluded-utts-only can be used to convert samples possesing the attribute excluded set to True only

Thanks,
Nuance Mix Team

Enhancements​

• added oauth_enabled flag to config for diabling OAuth token generation for on premise deployments
• added request header 'x-nuance-client-id' to request headers for on-premise deployments

ASRaaS functionality and params exposed​

• inline_wordset_weight
• compiled_wordset_weight
• builtins
• builtins_weight
• suppress_call_recording
• speech_detection_sensitivity
• utterance_end_silence_ms
• use_character_error_rate
• score_against_mft

NLUaaS functionality and params exposed​

• output_full_entity_hierarchy
• flattened_predefined_output
• useEntityF1
• subtotal_intents

Evaluation functionalities​

• Added oauth_enabled flag to config for diabling OAuth token generation for on premise deployments
• Fixed header incorrectly displayed in the report
• Added support for Apple Silicon

Thanks,
Nuance Mix Team

Enhancements​

• Client Data provided with each request
  • used for troubleshooting, x-client-request-id
• Ability to provide simplified values in truth file testing prebuilt entities
  • only works for CALENDARX at this time
• New validation process up front for truth file
• Refreshed report design
• Improved error visibility
• Latest versions of ASR and NLU protos
• Compiled wordset support for ASR and NLU (reference only)

ASRaaS functionality and params exposed​

• utterance_detection_mode
• asr_settings_uri
• discard_speaker_adaptation
• formatting_scheme
• formatting_options
• wakeup_word + filter_wakeup_word
• use_speaker_profile

NLU parsing and eval functionality​

• use_entity_literal
• predicted_entities_to_ignore
• entity_evaluation
• nBest
• Confidence thresholds

See config.json for information on each of these.

Fixes and improvements​

• Fix issue where report output was not being properly sorted by error rate
• Fix issue with dlm_weight not being properly picked up from config.json
• Fix issue where adjudication map was not properly loaded
• Fix utterance detection mode (default: DISABLED)
• Fix missing logs per request in table
• Minor updates to Test Run Settings tab in .html report:
  • Include the system path to results file
  • Display the language if the default of eng-USA is used when absent from config.json
  • Pretty-print ASR & NLU settings

Thanks,
Nuance Mix Team

This update includes the following enhancements:

• Add retry mechanism
  • ASR and NLU properties now have: retry_count and retry_sleep_s
  • Ensures token is valid for retry
  • Note: retry_count max is 4, retry_sleep_s max is 30
  • See config.json for more
• Add NLU config support: input_logging_mode, default PLAINTEXT
• New errors.tsv file, compliments results.tsv for those rows for which errors occured
• Add default client_data for ASR and NLU requests
  • Includes: tool version, os/platform system in context of runner

Some fixes:

• Fix auto_punctuate evaluation of true/false
• Fix file handling and various newline issues for Windows users
• File previously results.postprocessed.{timestamp}.json is now out.json

Thanks,
Nuance Mix Team

This update includes the following fixes:

• Re-establish authentication with XaaS, if token is expired

Thanks,
Nuance Mix Team

Initial release, hello world.

This testing tool is available in two types of packages:

• Python Wheel
• Docker Image

See Downloads for more details.

The tool provides the ability to execute transactions against:

• Nuance Cloud ASR Service (powered by Krypton), and/or
• Nuance Cloud NLU Service (powered by NLE)

Provide the following:

• truth.tsv
• config.json
• mappings.adjmap
• audio/x.wav

Audio data must be mono, 16-bit linear PCM, either 8kHz or 16kHz.

When executing the tests, the following will be stored in the output directory as defined:

• log.txt - raw logs, based on the log_level in config.json
• overview.json - raw json for the test report
• report.html - rendered HTML report
• results.tsv - data pre-scoring
• results.nuanscored.tsv - scored results
• results.postprocessed.{date}.json - each row's transaction response pre-scoring tool

Check out Quick Start and get started!

Thanks,
Nuance Mix Team