Troubleshooting
Overview​
Logs are produced and can be piped out to a file.
The following sample contains all the raw details from the execution of the test run.
There are numerous other files produced, and one noteworthy one during troubleshooting isout.json.
Sample log.txt​
The test execution logs are rendered to stdout.
The following sample is with the INFO log_level in config.json.
2022-06-20 00:10:31,501.501 INFO [__main__:1465] - init_logging: [logging] configured
2022-06-20 00:10:31,501.501 INFO [__main__:129] - main: Validating test set...
|████████████████████████████████████████| 22/22 [100%] in 0.0s (11698.54/s)
2022-06-20 00:10:31,536.536 INFO [__main__:162] - main: Test set validated!
2022-06-20 00:10:31,763.763 INFO [__main__:1465] - init_logging: [logging] configured
2022-06-20 00:10:31,763.763 INFO [__main__:1405] - run_tests: ======================================================================================================================================================
2022-06-20 00:10:31,763.763 INFO [__main__:1406] - run_tests: 1 of 2: Execute tests with <_io.TextIOWrapper name='./truth.tsv' mode='r' encoding='utf-8'> (./audio)
2022-06-20 00:10:32,122.122 INFO [__main__:1159] - execute_tests: ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
2022-06-20 00:10:32,122.122 INFO [__main__:1165] - execute_tests: Processing row: {'speaker': '1', 'codedWvnm': '1-1.wav', 'transcription': 'Book a flight to Atlanta', 'annotation': '{"eArrivalCity": "Atlanta"} ', 'intent': 'iBookFlight'}
2022-06-20 00:10:32,122.122 INFO [__main__:492] - xasr: start
2022-06-20 00:10:32,126.126 INFO [__main__:467] - xasr_client_stream: >>>>>> stream generator <wave.Wave_read object at 0x10df68df0>
2022-06-20 00:10:33,007.007 INFO [__main__:714] - xasr: complete
2022-06-20 00:10:33,010.010 INFO [__main__:761] - xnlu: start
2022-06-20 00:10:33,290.290 INFO [__main__:825] - xnlu: complete
2022-06-20 00:10:33,295.295 INFO [__main__:1165] - execute_tests: Processing row: {'speaker': '1', 'codedWvnm': '1-2.wav', 'transcription': "I'd like to go to Atlanta on May 4th", 'annotation': '{"eArrivalCity": "Atlanta", "eDepartureDate": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_DAY": 4, "nuance_CALENDAR.nuance_DATE.nuance_DATE_ABS.nuance_MONTH": 5}}', 'intent': 'iBookFlight'}
2022-06-20 00:10:33,296.296 INFO [__main__:492] - xasr: start
2022-06-20 00:10:33,298.298 INFO [__main__:467] - xasr_client_stream: >>>>>> stream generator <wave.Wave_read object at 0x10df99400>
2022-06-20 00:10:34,146.146 INFO [__main__:714] - xasr: complete
2022-06-20 00:10:34,156.156 INFO [__main__:761] - xnlu: start
2022-06-20 00:10:34,320.320 INFO [__main__:825] - xnlu: complete
2022-06-20 00:10:34,329.329 INFO [__main__:1165] - execute_tests: Processing row: {'speaker': '1', 'codedWvnm': '1-3.wav', 'transcription': 'Is there a flight from New York to Phoenix?', 'annotation': '{"eDepartureCity": "New York City", "eArrivalCity": "Phoenix"}', 'intent': 'iBookFlight'}
2022-06-20 00:10:34,329.329 INFO [__main__:492] - xasr: start
2022-06-20 00:10:34,331.331 INFO [__main__:467] - xasr_client_stream: >>>>>> stream generator <wave.Wave_read object at 0x10df99c70>
2022-06-20 00:10:35,230.230 INFO [__main__:714] - xasr: complete
2022-06-20 00:10:35,236.236 INFO [__main__:761] - xnlu: start
2022-06-20 00:10:35,347.347 INFO [__main__:825] - xnlu: complete
2022-06-20 00:10:35,349.349 INFO [__main__:1165] - execute_tests: Processing row: {'speaker': '1', 'codedWvnm': '1-4.wav', 'transcription': 'Can I fly to San Francisco?', 'annotation': '{"eArrivalCity": "San Francisco"}', 'intent': 'iBookFlight'}
2022-06-20 00:10:35,349.349 INFO [__main__:492] - xasr: start
2022-06-20 00:10:35,351.351 INFO [__main__:467] - xasr_client_stream: >>>>>> stream generator <wave.Wave_read object at 0x10dfa44c0>
2022-06-20 00:10:36,487.487 INFO [__main__:714] - xasr: complete
2022-06-20 00:10:36,489.489 INFO [__main__:761] - xnlu: start
2022-06-20 00:10:36,571.571 INFO [__main__:825] - xnlu: complete
2022-06-20 00:10:36,574.574 INFO [__main__:1165] - execute_tests: Processing row: {'speaker': '1', 'codedWvnm': '1-5.wav', 'transcription': "I'm going to New York tomorrow", 'annotation': '{"eArrivalCity": "New York City", "eDepartureDate": {"nuance_CALENDAR.nuance_DATE.nuance_DATE_REL.nuance_STEP": "day", "nuance_CALENDAR.nuance_DATE.nuance_DATE_REL.nuance_INCREMENT": 1}}', 'intent': 'iBookFlight'}
2022-06-20 00:10:36,575.575 INFO [__main__:492] - xasr: start
2022-06-20 00:10:36,577.577 INFO [__main__:467] - xasr_client_stream: >>>>>> stream generator <wave.Wave_read object at 0x10df7f040>
2022-06-20 00:10:37,246.246 INFO [__main__:714] - xasr: complete
2022-06-20 00:10:37,247.247 INFO [__main__:761] - xnlu: start
2022-06-20 00:10:37,389.389 INFO [__main__:825] - xnlu: complete
2022-06-20 00:10:37,394.394 INFO [__main__:1165] - execute_tests: Processing row: {'speaker': '1', 'codedWvnm': '1-6.wav', 'transcription': 'Did I lose my luggage in New York?', 'annotation': '{}', 'intent': 'iFAQLostLuggage'}
2022-06-20 00:10:37,395.395 INFO [__main__:492] - xasr: start
2022-06-20 00:10:37,398.398 INFO [__main__:467] - xasr_client_stream: >>>>>> stream generator <wave.Wave_read object at 0x10dfd0670>
....
2022-06-20 00:10:54,373.373 INFO [__main__:1429] - run_tests: 2 of 2: Prep results for scoring - 21 items
2022-06-20 00:10:54,373.373 INFO [__main__:1385] - score_prep: {'transcription': 'Book a flight to Atlanta.', 'mft': 'Book a flight to Atlanta.', 'error_type': None, 'error_text': None, 'asr_confidence': 0.974, 'asr_request_id': '4632f644-08f0-46cb-8ad4-6447cc9decec', 'intent': 'iBookFlight', 'entities': {'eArrivalCity': 'Atlanta'}, 'literal': 'Book a flight to Atlanta.', 'origin': 'GRAMMAR', 'nlu_confidence': 0.9999524, 'nlu_request_id': '4632f644-08f0-46cb-8ad4-6447cc9decec'}
....
2022-06-20 00:10:54,375.375 INFO [__main__:1431] - run_tests: ======================================================================================================================================================
To re-run:
python -m nuance.testaccuracy -c config.json -t ./truth.tsv -a ./audio -o ./output/2022.06.20-00.10.31 --log ./output/2022.06.20-00.10.31/log.txt
nuance.score version: 0.5
Loading conf from nuance/score/config.py
Loaded: <module 'config' from 'env/lib/python3.9/site-packages/nuance/score/config.py'>
Loading conf: <module 'config' from 'env/lib/python3.9/site-packages/nuance/score/config.py'>
Class: NLUCSV_HYP PASR: {'hyp_intent': <function hyp_intent at 0x10c65bf70>, 'hyp_annotation': <function hyp_annotations at 0x10c661040>, 'hyp': <function hyp_transcription at 0x10c6610d0>, 'error': <function error at 0x10c661160>, 'error_type': <function error_type at 0x10c6611f0>}
Class: NLUCSV_REF PASR: {'intent': <function ref_intent at 0x10c6563a0>, 'annotation': <function ref_annotations at 0x10c65be50>, 'transcription': <function ref_transcription at 0x10c65bee0>}
Ref recordCount: 21
Hyp recordCount: 21
syncing media files: DISABLED FOR NOW
Creating nlu score csv: ./output/2022.06.20-00.10.31/results.nuanscored.tsv
[4 %]
[9 %]
[14 %]
[19 %]
[23 %]
[28 %]
[33 %]
[38 %]
[42 %]
[47 %]
[52 %]
[57 %]
[61 %]
[66 %]
[71 %]
[76 %]
[80 %]
[85 %]
[90 %]
[95 %]
[100 %]
To re-run:
python -m nuance.score --csvref ./truth.tsv --csvhyp ./output/2022.06.20-00.10.31/results.tsv --map ./mappings.adjmap --out ./output/2022.06.20-00.10.31 --runtimeConfig config.json
To re-run:
python -m nuance.report ./output/2022.06.20-00.10.31/results.nuanscored.tsv config.json ./output/2022.06.20-00.10.31
Run complete.
open ./output/2022.06.20-00.10.31/report.html
...
ASR and NLU Raw Responses​
Each test execution response is logged in a subdirectory: xaas_logs.
Sample 0001_asraas.log​
{
"result": {
"absEndMs": 3680,
"utteranceInfo": {
"durationMs": 3680,
"dsp": {
"snrEstimateDb": 21.0,
"level": 16385.0,
"numChannels": 1,
"initialSilenceMs": 100,
"initialEnergy": -34.8569,
"finalEnergy": -18.4933,
"meanEnergy": 70.4414
}
},
"hypotheses": [
{
"confidence": 0.974,
"averageConfidence": 0.993,
"formattedText": "Book a flight to Atlanta.",
"minimallyFormattedText": "Book a flight to Atlanta.",
"words": [
{
"text": "Book",
"confidence": 0.993,
"startMs": 980,
"endMs": 1320
},
{
"text": "a",
"confidence": 0.993,
"startMs": 1320,
"endMs": 1460
},
{
"text": "flight",
"confidence": 0.995,
"startMs": 1460,
"endMs": 1740
},
{
"text": "to",
"confidence": 0.992,
"startMs": 1740,
"endMs": 1900
},
{
"text": "Atlanta",
"confidence": 0.993,
"startMs": 1900,
"endMs": 2500
},
{
"text": ".",
"confidence": 0.992,
"startMs": 2500,
"endMs": 2520
}
],
"encryptedTokenization": "..."
}
],
"dataPack": {
"language": "eng-USA",
"topic": "GEN",
"version": "4.7.3",
"id": "GMT20220208215217"
}
}
}
Sample 0001_nluaas.log​
{
"status": {
"code": 200,
"message": "OK"
},
"result": {
"literal": "Book a flight to Atlanta.",
"interpretations": [
{
"singleIntentInterpretation": {
"intent": "iBookFlight",
"confidence": 0.9999524,
"origin": "GRAMMAR",
"entities": {
"eArrivalCity": {
"entities": [
{
"textRange": {
"startIndex": 17,
"endIndex": 24
},
"confidence": 1.0,
"origin": "GRAMMAR",
"entities": {
"eCity": {
"entities": [
{
"textRange": {
"startIndex": 17,
"endIndex": 24
},
"confidence": 1.0,
"origin": "GRAMMAR",
"stringValue": "Atlanta",
"literal": "Atlanta",
"formattedLiteral": "Atlanta",
"formattedTextRange": {
"startIndex": 17,
"endIndex": 24
},
"audioRange": {
"startTimeMs": 1900,
"endTimeMs": 2500
}
}
]
}
},
"literal": "Atlanta",
"formattedLiteral": "Atlanta",
"formattedTextRange": {
"startIndex": 17,
"endIndex": 24
},
"audioRange": {
"startTimeMs": 1900,
"endTimeMs": 2500
}
}
]
}
}
}
},
{
"singleIntentInterpretation": {
"intent": "NO_INTENT",
"confidence": 1.50801e-05,
"origin": "STATISTICAL",
"entities": {
"eArrivalCity": {
"entities": [
{
"textRange": {
"startIndex": 17,
"endIndex": 24
},
"confidence": 0.8680555,
"origin": "STATISTICAL",
"entities": {
"eCity": {
"entities": [
{
"textRange": {
"startIndex": 17,
"endIndex": 24
},
"confidence": 0.8680555,
"origin": "STATISTICAL",
"stringValue": "Atlanta",
"literal": "Atlanta",
"formattedLiteral": "Atlanta",
"formattedTextRange": {
"startIndex": 17,
"endIndex": 24
},
"audioRange": {
"startTimeMs": 1900,
"endTimeMs": 2500
}
}
]
}
},
"literal": "Atlanta",
"formattedLiteral": "Atlanta",
"formattedTextRange": {
"startIndex": 17,
"endIndex": 24
},
"audioRange": {
"startTimeMs": 1900,
"endTimeMs": 2500
}
}
]
}
}
}
},
{
"singleIntentInterpretation": {
"intent": "iFlightStatus",
"confidence": 1.02592e-05,
"origin": "STATISTICAL"
}
}
],
"formattedLiteral": "Book a flight to Atlanta."
}
}