Evaluate agent performance in Connect Customer using generative AI

Use generative AI to answer questions that only require the conversation transcript. Examples are questions on soft skills, questions that check for call flow, or compliance statements, among others.

Split complex questions into multiple simpler ones. For example, instead of "Did the agent exhibit active listening?", ask two questions: "Did the agent understand the customer's problem the first time, without the customer needing to repeat themselves?" and "Did the agent summarize the issue after the customer explained it?".

Use conditionally enabled questions to enable or disable questions that are only applicable in certain situations. For example, you may have one question, "Did the customer buy a product during the conversation?", and a subsequent conditionally enabled question, "Did the agent provide mandatory fee disclosures before completing the sale?". For more details, see Step 4: Conditionally enable questions.

Don't use generative AI to answer questions that need information outside the conversation transcript. Generative AI cannot analyze screen recordings, access your internal or third-party systems such as CRM applications, or evaluate conversations across multiple contacts.

Don't use generative AI to evaluate quantifiable activities such as "Was the customer put on excessive hold?" or "Was the customer frequently interrupted?". Instead, set the question type to Number and use metrics such as the longest hold duration or the number of interruptions. For more details, see Step 6: Enable automated evaluations.

Don't automate questions that assess interactions between multiple parties (another agent, a partner institution, or a second customer). Contact Lens is aware of only two participants at a given time. For example, avoid a question like "If another person other than the primary account holder joined the conversation, did the agent first confirm with the primary account holder before proceeding?".

Don't ask questions that depend on tone of voice. Generative AI cannot determine the agent's or customer's tone.

Don't use generative AI for highly subjective questions, such as "Was the agent attentive during the call?".

Word questions as complete sentences. Instead of "ID validation", ask "Did the agent attempt to validate the customer's identity?".

Add detailed evaluation criteria in the Instructions to evaluators for the question. For "Did the agent try to validate the customer's identity?", add the instruction: Answer is Yes if the agent asked for the customer's membership number and postal code before addressing their questions. Answer is No otherwise.

Define business-specific terms the question depends on. If the agent must name a department in the greeting, list example department names in the instructions.

Use "agent" and "customer" consistently. Avoid variations like "colleague", "representative", or "associate", and "member", "caller", or "subscriber".

Make it clear who is being evaluated. "Did the agent avoid the usage of profanity?" can be read as asking whether profanity occurred anywhere, returning "No" even when only the customer used it. Ask "Did the agent use profanity?" instead.

Phrase questions positively. Use "Did the agent greet the customer?" rather than "Did the agent skip the greeting?". Phrasing questions positively provides better AI-evaluation reasoning and references.

Specify when the answer is Not Applicable (N/A). For example: The answer is N/A if the call resulted in a transfer.

Clarify whether all or any of the specified agent behaviors is required. "The agent must ask the customer's name and phone number" fails if the agent asked for the name but not the phone number.

Include examples for non-standard scenarios, not just the standard call flow. If you expect the agent to say "It typically takes 3 to 5 business days" on a standard call, you should also include callback phrasing like "I'll call you back within 30 minutes with an update". Standard-flow-only examples lead to inconsistent answers on callbacks, escalations, and transfers.

Give auto-fail questions the most attention, since one failing answer affects the whole form's evaluation score. Cover edge cases and non-standard scenarios. For more on scoring, see Step 5: Assign scores and ranges to answers.

Don't use double quotes unless you need exact wording. If the instruction checks for "Have a nice day", the AI won't match Have a nice afternoon. Write instead: The agent wished the customer a nice day.

Don't use acronyms. Spell out the full term, for example "CFPB", so the AI can interpret it correctly.

Don't use proper nouns likely to be misspelled in the transcript. A product name like Klarity Pay may be transcribed differently, preventing a match.

Don't use vague questions. Instead of "Did the agent use appropriate language?", ask "Did the agent use profanity?".

Don't use long verbatim scripts. Checking for "Thank you for calling ABC Bank. How may I assist you?" rarely matches, since minor transcription differences break the full script.

Use simple, short answer options, such as Yes, No, and Partial.

Enable the Optional question setting when a question may not apply. This lets evaluators skip the question or mark it Not Applicable.

Don't use spelling errors or special characters in answer options, as they can reduce the accuracy of generative AI answers.

Don't use too many answer options. For "How was the customer experience?", a long list like Great, Good, OK, Poor, Very Poor, and Horrible reduces accuracy. Use a smaller set of distinct options instead.

Don't use long text in answer options, since the generative AI model might reproduce it incorrectly.