Advanced Conversation Analytics Telemetry

[ryanisgrig]

Spec Title
Advanced Conversation Analytics Telemetry

Area
Enterprise Template

Spec Content
The Virtual Assistant requires additional telemetry logged from the Bot Framework to support new Advanced Conversation Analytics PowerBI dashboards. This is being done in conjunction with updates to the SDK at microsoft/botframework-sdk#5177.

In order to implement unit tests, we will provide interfaces for ITelemetryLuisRecognizer and ITelemetryQnAMaker. This will allow developers to mock and fake classes, avoiding roundtrips to the server and stage the results that the tests expects from the services.

Key	Description
❌	Missing
⚠	Existing component will be deprecated
✔	Supported

Logging

TelemetryLoggerMiddleware

CustomEvent: BotMessageReceived

Logged when the bot receives a new message.

Key	Value	Current
user_id	activity.ChannelId + activity.From.Id Telemetry Initializer	❌ 4.2 dependency
customDimensions.conversationId => session_id	activity.ConversationId Telemetry Initializer	⚠ 4.2 dependency
customDimensions.activityId	activity.Id Telemetry Initializer	⚠ 4.2 dependency
customDimensions.activityType	activity.Type Telemetry Initializer	❌ 4.2 dependency
customDimensions.channel => customDimensions.channelId	activity.ChannelId Telemetry Initializer	⚠ 4.2 dependency
customDimensions.text	activity.Text	✔ Supported
customDimensions.fromId	activity.From.Id	✔ Supported
customDimensions.fromName	activity.From.Name	✔ Supported
customDimensions.conversationName	activity.Conversation.Name	✔ Supported
customDimensions.locale	activity.Locale	✔ Supported
customDimensions.recipientId	activity.Recipient.Id	❌ Adding in next release
customDimensions.recipientName	activity.Recipient.Name	❌ Adding in next release

CustomEvent: BotMessageSend

Logged when bot sends a message.

Key	Value	Status
user_id	activity.ChannelId + activity.From.Id Telemetry Initializer	❌ 4.2 dependency
customDimensions.conversationId => session_id	activity.ConversationId Telemetry Initializer	⚠ 4.2 dependency
customDimensions.activityId	activity.Id Telemetry Initializer	⚠ 4.2 dependency
customDimensions.activityType	activity.Type Telemetry Initializer	❌ 4.2 dependency
customDimensions.channelId	activity.ChannelId Telemetry Initializer	⚠ 4.2 dependency
customDimensions.text	activity.Text	✔ Supported
customDimensions.conversationName	activity.Conversation.Name	✔ Supported
customDimensions.locale	activity.Locale	✔ Supported
customDimensions.recipientId	activity.Recipient.Id	✔ Supported
customDimensions.recipientName	activity.Recipient.Name	✔ Supported
customDimensions.replyToId	activity.ReplyToId	❌ Adding in next release

TelemetryLuisRecognizer

CustomEvent: LuisIntent.IntentName(LUISResult)

Logs results from LUIS service.

Key	Value	Status
user_id	activity.ChannelId + activity.From.Id Telemetry Initializer	❌ 4.2 dependency
customDimensions.conversationId => session_id	activity.ConversationId Telemetry Initializer	⚠ 4.2 dependency
customDimensions.activityId	activity.Id Telemetry Initializer	⚠ 4.2 dependency
customDimensions.activityType	activity.Type Telemetry Initializer	❌ 4.2 dependency
customDimensions.channel => customDimensions.channelId	activity.ChannelId Telemetry Initializer	❌ 4.2 dependency
customDimensions.intent	topLuisIntent.intent	✔ Supported
customDimensions.intentScore	topLuisIntent.score.ToString("N2")	✔ Supported
customDimensions.sentimentLabel	sentiment.label	✔ Supported
customDimensions.sentimentScore	sentiment.score	✔ Supported
customDimensions.question	context.Activity.Text	✔ Supported
LUIS Entities customDimensions.key	value	✔ Supported
customDimensions.dialogId	WaterfallStepContext.ActiveDialog.Id	✔ Supported

TelemetryQnaMaker

CustomEvent: QnAMessage

Logs results from QnA Maker service.

Key	Value	Status
user_id	activity.ChannelId + activity.From.Id Telemetry Initializer	❌ 4.2 dependency
customDimensions.conversationId => session_id	activity.ConversationId Telemetry Initializer	⚠ 4.2 dependency
customDimensions.activityId	activity.Id Telemetry Initializer	❌ 4.2 dependency
customDimensions.activityType	activity.Type Telemetry Initializer	❌ 4.2 dependency
customDimensions.channel => customDimensions.channelId	activity.ChannelId Telemetry Initializer	❌ 4.2 dependency
customDimensions.username => customDimensions.fromName	activity.From.Name	✔ Supported
customDimensions.originalQuestion	activity.Text	✔ Supported
customDimensions.sentimentLabel	sentiment.label	✔ Supported
customDimensions.sentimentScore	sentiment.score	✔ Supported
customDimensions.question	commas-separated string of queryResult.Questions	✔ Supported
customDimensions.answer	queryResult.Answer	✔ Supported
customDimensions.score	If article found, queryResult.Score	✔ Supported
customDimensions.articleFound	true/false	✔ Supported

Links
https://github.com/daveta/analytics/blob/master/va_reports.md
#134
#135
#343