truth.tsv
Overview
truth.tsv contains all the information to leverage when performing analysis.
⚠️ Ensure tabs are used as the separator.
Data
| Column | Type | Description |
|---|---|---|
| speaker | String | speaker profile identifer |
| codedWvnm | String | file name; paths are based on the input of the tool - see run.sh |
| transcription | String | used as input if NLU-only, or fallback if ASR fails |
| annotation | JSON string | entity mentions - key-value pairs; also supports literal if value is "" |
| intent | String | expected intent |
Sample truth.tsv
speaker codedWvnm transcription annotation intent
1 1-1.wav Book a flight to Atlanta {"eArrivalCity": "Atlanta"} iBookFlight
1 1-2.wav I'd like to go to Atlanta on May 4th {"eArrivalCity": "Atlanta", "eDepartureDate": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_DAY": 4, "nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_MONTH": 5}} iBookFlight
1 1-3.wav Is there a flight from New York to Phoenix? {"eDepartureCity": "New York City", "eArrivalCity": "Phoenix"} iBookFlight
1 1-4.wav Can I fly to San Francisco? {"eArrivalCity": "San Francisco"} iBookFlight
1 1-5.wav I'm going to New York tomorrow {"eArrivalCity": "New York City", "eDepartureDate": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_REL.nuance_STEP": "day", "nuance_CALENDAR.nuance_DATE.nuance_DATE_REL.nuance_INCREMENT": 1}} iBookFlight
1 1-6.wav Did I lose my luggage in New York? {} iFAQLostLuggage
1 1-7.wav Where's my luggage? {} iFAQLostLuggage
1 1-8.wav I'd like to travel from Phoenix to Atlanta on July 20th {"eArrivalCity": "Atlanta", "eDepartureCity": "Phoenix", "eDepartureDate": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_MONTH": 7, "nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_DAY": 20}} iBookFlight
1 1-9.wav Yeah, is my flight on time? {} iFlightStatus
1 1-10.wav When is my flight? {} iFlightStatus
1 1-11.wav What's my itinerary? {} iFlightStatus
1 1-12.wav What is my flight status? {} iFlightStatus
1 1-13.wav What's a good restaurant in New York? {} NO_INTENT
1 1-14.wav To New York {"eArrivalCity": "New York City"} NO_INTENT
1 1-15.wav From Phoenix {"eDepartureCity": "Phoenix"} NO_INTENT
1 1-16.wav Tomorrow {"nuance_CALENDARX": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_REL.nuance_STEP": "day", "nuance_CALENDAR.nuance_DATE.nuance_DATE_REL.nuance_INCREMENT": 1}} NO_INTENT
1 1-17.wav No actually I want to leave from Atlanta {"eDepartureCity": "Atlanta"} NO_INTENT
1 1-18.wav Change the destination from New York to Atlanta {"eDepartureCity": "New York City", "eArrivalCity": "Atlanta"} NO_INTENT
1 1-19.wav Make it on August 5th {"eDepartureDate": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_MONTH": 8, "nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_DAY": 5}} NO_INTENT
1 1-20.wav Can I go from Montreal to Phoenix on May 1st? {"eArrivalCity": "Phoenix", "eDepartureDate": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_DAY": 1, "nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_MONTH": 5}} iBookFlight
2 2-1.wav Book a flight from New York to Atlanta for tomorrow {"eArrivalCity": "Atlanta", "eDepartureCity": "New York City", "eDepartureDate": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_REL.nuance_STEP": "day", "nuance_CALENDAR.nuance_DATE.nuance_DATE_REL.nuance_INCREMENT": 1}} iBookFlight
.trsx Truth File
When using the run.py wrapper process, you can specify a Mix .trsx file as your input truth file, and the samples contained within it will be used as test cases. Internally, the tool converts the samples in the .trsx file to the .tsv format described above before execution.
One may want to use a .trsx as the truth file if exporting samples directly from Mix that are intended specifically for testing. Then you do not have to manually create a .tsv truth file.
Simply specify a .trsx as your input truth file, and run.py will sniff the file extension and handle the conversion automatically.
Limitations:
- The
.trsxformat does not support specifying an associated audio file, so using.trsxas the truth file format should only be used for NLU-only tests - Only the literal values of entity annotations can be automatically populated from the
.trsxannotations; if you want to use the canonical value instead of or in addition to the literal, the generated.tsvfile should be updated before running your test (see below)
Manual .trsx to .tsv conversion
You can convert a .trsx file to Mix Testing Tool .tsv format manually by calling the conversion module directly. This could be useful if you want to first export samples to test on from a Mix project and then update the generated .tsv file to point to audio files corresponding to each test case so that an ASR+NLU test can be done.
python -m nuance.conversion -i <path_to_your_trsx_file>
The location of the output .tsv file will then be displayed.
Run python -m nuance.conversion -h for more information.
Including only excluded="true" samples
When samples are excluded from model training, upon project export the XML attribe excluded is set to true for those samples. You may want to then test specifically on these samples since it is considered a best practice to separate training data from testing data.
To only convert and/or test on samples where the excluded XML attribute is set to true, use the -eo or --excluded-utts-only argument. This option is available when using run.py to perform an end-to-end test or when using the nuance.conversion module directly as described above.