Optimize AI Agent based on insights from your ticketsUpdated 2 days ago

You can improve AI Agent’s performance and increase its ability to successfully automate tickets by reviewing the Optimize page.

On the Optimize page, you can explore tickets grouped by intent (the reasons your customers contact you). For each intent, you can see AI Agent’s success rate, its average CSAT score, and review previous tickets that the AI either automated or handed over to your team.

You can then use these insights to identify gaps in AI Agent’s knowledge and improve its setup to better answer common inquiries from customers. As you make improvements, you can return to the Optimize page to see how your changes improve the AI’s ticket coverage and success rate.

Requirements

• You must have an active Automate subscription
• You must have Lead or Admin permissions

Filtering and reading data

On the Optimize page, you can filter data by date range and interact with various metrics to review tickets or drill down for more detail.

• Open the Optimize page → select AI Agent from the main menu, then select Optimize under one of your connected stores
  
• Filter by date → use the Date filter to adjust the results on the page
  
  
  • Choose a preset date range (either Past 7 days or the Last week starting on Sunday or Monday)
  • Or use the calendars to select custom dates within a 7 day period
• Review Tickets → select metrics like Coverage rate to see a list of tickets including their intent and outcome
  
  
  • Next to each intent, select Tickets to see all AI Agent tickets associated with the intent
• Drill down on intents → select any intent to see more granular ticket topics detected by AI Agent

Review AI Agent’s overall performance

You can access the Optimize page by going to AI Agent from the main menu, then selecting Optimize. At the top of the page, you can review key metrics to understand AI Agent’s performance.

Over time, you can use the performance section to measure how improvements to AI Agent’s knowledge sources and setup affect its ticket coverage, the volume of automated interactions, success rate and CSAT.

Here are details about each of the 4 performance metrics:

• Coverage rate → the percentage (%) of your store’s total tickets that AI Agent attempted to respond to
  
  
  • This metric can include tickets that AI Agent directly responded to, as well as tickets that the AI immediately handed over because of a listed handover topic in your settings.
  • Connect AI Agent to Email and Chat to increase your coverage rate
• Automated interactions → the number of interactions that AI Agent fully automated without agent intervention
  
  
  • learn about Automated interactions
• Success rate → of the tickets that AI Agent has covered, the percentage (%) that were fully automated
  
  
  • Add knowledge, Guidance, and connect AI Agent to other apps to increase the the number of tickets AI Agent can fully automate
• Customer satisfaction → the average CSAT score for tickets that AI Agent handled and for which shoppers gave a rating
  
  
  • this metrics include any tickets for which a survey was sent within your selected timeframe
  • CSAT surveys are sent after a ticket has been closed

Explore AI performance by ticket intent

Exploring intents helps you assess how AI Agent performs on each of the reasons your customers contact you.

From here, you can identify opportunities to improve AI Agent’s coverage (the % of total tickets it handles), as well as its success rate (the % of tickets for each intent the AI successfully automates).

For each intent in the table, you can review:

• Success rate uplift opportunity  → an estimate of how much AI Agent’s success rate could increase if you improve how it responds to shoppers with a specific intent
  
  
  • Example: if AI Agent has 100 tickets total and the order/status intent has 15 tickets that have not been automated, then the uplift opportunity is 15%
• Tickets → the number of tickets AI Agent handled with the intent
  
  
  • Click on the number of tickets to open an expanded list of all the tickets AI Agent handled with the intent
  • Select Download all tickets to export a .csv file and review ticket data in a spreadsheet
• Success rate → the % of tickets with the intent that AI Agent fully automated
  
  
  • learn about Automated interactions
• Average CSAT → the average CSAT score given to AI Agent for tickets with the intent

• Drill down for details → select any intent to see more granular ticket topics detected by AI Agent
  
  
  • For example, you may have a lot of Return/Request tickets, but you want to understand what topics are driving returns
  • Here you can see detailed ticket topics like “I ordered the wrong size” or “product description is wrong”
  • Some tickets may initially be classified as Other. Over time, they may be grouped into more specific topics as AI Agent learns from past tickets

Intents detected by AI Agent

AI Agent uses the AI Intent managed ticket field to automatically detect and assign an intent to each of the tickets that it handles. You can find the full list of intents below: