N-Best

By default, the testing tool uses the top NLU interpretation (the result with the highest confidence) and compares that result to the reference during the scoring step.

Optionally, the tool can consider all NLU interpretations returned by NLUaaS, and if any one of those interpretations is correct with respect to the reference test case, count that test case as correct.

This behavior can be set with the nBest parameter:

	"nlu": {
		"use_asr_results": true,
		"nBest": true,
		...
	}

The intermediate results.tsv that contains the NLUaaS interpretations will contain lists for the predicted intent, predicted entities, NLU origin, and NLU confidence in order from most confident to least confident for each test case.

The final output in the report.html file will only contain one result:

• The fully correct result that matches the reference if such a result exists among all returned NLU interpretations;
• A partially correct result that matches the intent of the reference if such a result exists; or
• The result with the highest confidence if none of the candidate results' intents match the reference intent.

See InterpretResult for further details about interpretation results.