Version: Next

Execution

Python

All-in-one with `run.sh`

note

Copy the below snippet into run.sh. Ensure executable: chmod +x run.sh.

Setup your Environment Variables (optional)

export CONFIG=config.json
export OUTPUT_DIR_BASE=./output
export TRUTH_TSV=./truth.tsv
export AUDIO_DIR=./audio
export SCORING_ADJMAP_FILE=./mappings.adjmap

Execute

./run.sh

`run.sh` Contents

#!/usr/bin/env bash

: '

ENV VARS:
    CONFIG
    OUTPUT_DIR_BASE
    TRUTH_TSV
    AUDIO_DIR
    SCORING_ADJMAP_FILE
    VALIDATION_LOG_FILE

./scripts/run.sh <config-file> <output> <truth-tsv> <audio-dir> <adjmap-file>

e.g. 
    ./scripts/run.sh config.json ./output ./samples/16khz_media/truth.tsv ./samples/16khz_media/audio/wav ./samples/16khz_media/mappings.adjmap

'

PYTHON=python

CONFIG=${CONFIG:-./config.json}
OUTPUT_DIR_BASE=${OUTPUT_DIR_BASE:-./output}
TRUTH_TSV=${TRUTH_TSV:-./truth.tsv}
AUDIO_DIR=${AUDIO_DIR:-./audio}
SCORING_ADJMAP_FILE=${SCORING_ADJMAP_FILE:-./mappings.adjmap}

if [ -n "$1" ];
then
    CONFIG=${1}
fi
if [ -n "$2" ];
then
    OUTPUT_DIR_BASE=${2}
fi
if [ -n "$3" ];
then
    TRUTH_TSV=${3}
fi
if [ -n "$4" ];
then
    AUDIO_DIR=${4}
fi
if [ -n "$5" ];
then
    SCORING_ADJMAP_FILE=${5}
fi

OUTPUT_DIR=${OUTPUT_DIR_BASE}/$(date +"%Y.%m.%d-%H.%M.%S")
RUNNER_LOG_FILE=${OUTPUT_DIR}/log.txt
VALIDATION_LOG_FILE=${OUTPUT_DIR}/test_set_validation.log
RAW_RESULTS_TSV=${OUTPUT_DIR}/results.tsv 
RESULTS_SCORED_TSV=${OUTPUT_DIR}/results.nuanscored.tsv

mkdir -p $OUTPUT_DIR

echo ""
echo "Running with"
echo " -> CONFIG ${CONFIG}"
echo " -> RUNNER_LOG_FILE ${RUNNER_LOG_FILE}"
echo " -> VALIDATION_LOG_FILE ${VALIDATION_LOG_FILE}"
echo " -> TRUTH_TSV ${TRUTH_TSV}"
echo " -> AUDIO_DIR ${AUDIO_DIR}"
echo " -> OUTPUT_DIR ${OUTPUT_DIR}"
echo " -> RAW_RESULTS_TSV ${RAW_RESULTS_TSV}"
echo " -> RESULTS_SCORED_TSV ${RESULTS_SCORED_TSV}"
echo " -> SCORING_ADJMAP_FILE ${SCORING_ADJMAP_FILE}"
echo ""

$PYTHON -m nuance.validation \
    --config $CONFIG \
    --truth_tsv $TRUTH_TSV \
    --audio_file_path $AUDIO_DIR \
    --log $VALIDATION_LOG_FILE

if [[ $? -ne 0 ]] ; then
    echo ""
    echo "Test set validation failed."
    echo "See $VALIDATION_LOG_FILE for details."
    echo ""
    echo "To re-run:
$PYTHON -m nuance.validation \
    --config $CONFIG \
    --truth_tsv $TRUTH_TSV \
    --audio_file_path $AUDIO_DIR \
    --log $VALIDATION_LOG_FILE"
    exit 1
fi

$PYTHON -m nuance.testaccuracy \
    -c $CONFIG \
    -t $TRUTH_TSV \
    -a $AUDIO_DIR \
    -o $OUTPUT_DIR \
    --log $RUNNER_LOG_FILE

if [[ $? -ne 0 ]] ; then
    echo ""
    echo "Tests failed."
    echo ""
    exit 1
else
    echo "To re-run:
$PYTHON -m nuance.testaccuracy \
    -c $CONFIG \
    -t $TRUTH_TSV \
    -a $AUDIO_DIR \
    -o $OUTPUT_DIR \
    --log $RUNNER_LOG_FILE"
fi

$PYTHON -m nuance.score \
    --csvref $TRUTH_TSV \
    --csvhyp $RAW_RESULTS_TSV \
    --map $SCORING_ADJMAP_FILE \
    --out $OUTPUT_DIR \
    --runtimeConfig $CONFIG

if [[ $? -ne 0 ]] ; then
    echo ""
    echo "Scoring failed."
    echo ""
    exit 1
else
    echo "To re-run:
$PYTHON -m nuance.score \
    --csvref $TRUTH_TSV \
    --csvhyp $RAW_RESULTS_TSV \
    --map $SCORING_ADJMAP_FILE \
    --out $OUTPUT_DIR \
    --runtimeConfig $CONFIG"
fi

$PYTHON -m nuance.report \
    $RESULTS_SCORED_TSV \
    $CONFIG \
    $OUTPUT_DIR

if [[ $? -ne 0 ]] ; then
    echo ""
    echo "Report generation failed."
    echo ""
    exit 1
else
    echo "To re-run:
$PYTHON -m nuance.report \
    $RESULTS_SCORED_TSV \
    $CONFIG \
    $OUTPUT_DIR"
fi

echo "Run complete."
echo ""
echo "open $OUTPUT_DIR/report.html"
echo ""

Ensure that the config.json and values in run.sh are up to date.

Then, when set up, simply:

./run.sh

All-in-one with `run.py`

An alternative wrapper process was introduced to add additional options as well as to allow execution natively with Windows via cmd and PowerShell without having to install and use a Linux environment such as Cygwin.

Some options made available by run.py are:

ability to skip truth file validation if desired
allows pointing to an existing output directory if results already exist for a particular test set and you want to re-run a specific step (e.g. you want to re-run scoring and report generation with different options but can use existing inference results)
supports Mix .trsx as an input format

See all available options and get additional help and information by running python scripts/run.py -h.

Separate Calls

An end to end run is not always desirable.

As such, each of the parts can be separately run.

Part 1. Validating Files

This is the first step and validates the truth file and required resources.

python -m nuance.validation \
    --config ./config.json \
    --truth_tsv ./truth.tsv \
    --audio_file_path ./audio \
    --log ./output/test_set_validation.log

Part 2. Execution & Results Gathering

After validating, execute transactions to Nuance XaaS using a truth file and audio.

Create a directory for the output of this test run:

mkdir ./output/latest

Start a test run:

python -m nuance.testaccuracy \
    -c ./config.json \
    -t ./truth.tsv \
    -a ./audio \
    -o ./output/latest \
    --log /tmp/test.log

This step creates a results.tsv file in the output directory.

Part 3. Scoring

With result data, it is possible to use the scoring library using a configuration and output destination.

python -m nuance.score \
    --csvref ./truth.tsv \
    --csvhyp ./output/latest/results.tsv \
    --map ./mappings.adjmap \
    --out ./output/latest \
    --runtimeConfig ./config.json

This step generates a results.nuanscored.tsv in the output directory.

Part 4. Report Generation

Once scored, a report can be generated using the data.

python -m nuance.report \
    ./output/latest/results.nuanscored.tsv \
    ./config.json \
    ./output/latest/ 

Launch the report in a browser:

open ./output/latest/report.html

Output

After a test has been run, the following files will be available in the output directory specified.

ls -l output/latest/

The package contents are as follows:

-rw-r--r--   1 user  group     157 Jun 19 23:47 errors.tsv
-rw-r--r--   1 user  group   11154 Jun 19 23:47 out.json
-rw-r--r--   1 user  group  129841 Jun 19 23:47 overview.json
-rw-r--r--@  1 user  group  556472 Jun 19 23:47 report.html
-rw-r--r--   1 user  group   42923 Jun 19 23:47 results.nuanscored.tsv
-rw-r--r--   1 user  group    7502 Jun 19 23:47 results.tsv
drwxr-xr-x  44 user  group    1408 Jun 19 23:47 xaas_logs

Python​

All-in-one with run.sh​

Setup your Environment Variables (optional)​

Execute​

run.sh Contents​

All-in-one with run.py​

Separate Calls​

Part 1. Validating Files​

Part 2. Execution & Results Gathering​

Part 3. Scoring​

Part 4. Report Generation​

Output​