Automatically score the quality of your tickets with Auto QAUpdated 22 days ago

Whenever your agents close a new conversation with a customer, Gorgias uses Auto QA to automatically evaluate and assign a quality score to the ticket using AI.

Rather than manually checking tickets, your reviewers can use the scores from Auto QA to better understand your team’s performance and focus on individual coaching.

And with the Auto QA report, you can review the overall quality of your customer conversations and identify trends that affect your CSAT and the experience of shoppers who interact with your brand.

Requirements

• You must have an active Automate subscription
• Auto QA scores are only visible for Admin and Lead users

Understanding scoring criteria

Auto QA uses artificial intelligence (AI) to automatically score the quality of your agent’s tickets based on 3 predefined criteria: Resolution completeness, Communication and Language proficiency.

The AI only evaluates public messages in your ticket conversations (no internal notes or metadata), and ignores messages from Flows, Rules-based messages, or Gorgias Bot when assigning a score.

• Resolution completeness: did the agent answer and resolve all issues brought up by the customer? (binary score [yes/no])
• Communication: did the agent show empathy, active listening and clear messaging? (scale of 1 to 5)
• Language: did the agent display a high proficiency in the language of the conversation: flawless spelling, grammar and syntax? (scale of 1 to 5)

The remaining 4 criteria are not automatically scored by Gorgias's AI. Team Leads and Admins may use these quality metrics when assessing agents' tickets based on your brand's unique policies and guidelines. Learn more how to score tickets manually.

• Accuracy: did the agent use the correct resolution flow to solve the customer's inquiry and provide accurate information? (scale of 1 to 5)
• Efficiency: did the agent handle the ticket quickly and minimize the number of touches? (scale of 1 to 5)
• Internal compliance: did the agent follow company guidelines and procedures: did they apply the relevant tags and macros, follow proper escalation and merging procedures? (scale of 1 to 5)
• Brand voice: did the agent use your brand's correct vocabulary, sign-offs and tone and voice? (scale of 1 to 5)

Setting up Auto QA

Auto QA is enabled by default. There is no set up required.

Viewing a ticket’s Auto QA Score

You can find and review the score given to a ticket within the AI Feedback tab of a conversation. Tickets are given a score 12 hours after the ticket closes.

1. Go to Tickets
2. Select a closed conversation
3. Select AI Feedback to view the score
4. For more detail, select the triangle icon to expand each part of the score and review the AI’s written comments.

Training the Auto QA AI

If you or another reviewer disagree with Auto QA’s assessment of a ticket, you can manually edit and change any aspect of the score.

If you want to update the comments associated with one of the quality scores, you can remove the entire AI-generated text and write your own assessment as if you were rating the quality of the ticket yourself.

When you edit the Auto QA’s assessment, Gorgias collects your changes to train and fine-tune the AI model so that it improves future scoring for similar tickets. Any changes you make will be reflected in the Statistics of your Auto QA report.

Manually scoring tickets

Some of the criteria for scoring tickets is not determined by Gorgias's AI and can only be scored manually. These include: Accuracy, Internal compliance, Efficiency, and Brand voice.

When reviewing tickets, Team Leads and Admins can use these criteria to assess agents based on your brand's unique policies and guidelines.

Manually scoring tickets with these 4 criteria helps to build a more complete picture of your ticket quality so that you can identify trends in the Auto QA report.

1. Go to Tickets
2. Select a closed conversation
3. Select AI Feedback to open the QA scores for the tickets
4. Score the ticket using the available criteria
   • Open the dropdown menu next to any criteria to change its score
   • Use the triangle icon to expand each criteria and write additional comments in the textbox

Using the Auto QA report

To get a high-level picture of the quality of your team’s work, you can use the Auto QA report to see both overall and agent-specific performance based on auto-reviewed tickets.

FAQ

Why was a ticket not scored?

Tickets are automatically scored 12 hours after the ticket closes. If you review a conversation prior to 12 hours since the ticket was closed, you may not see a score from Auto QA.

Additionally, not all tickets are eligible for a score from Auto QA.

• At the moment, Auto QA does not score tickets from public social media channels or voice. These include:
  
  
  • Facebook messages, mentions and recommendations
  • Instagram mentions, comments and ad comments
  • Twitter (X)
  • Gorgias Voice
  • Aircall
• Conversations that are too short do not receive an Auto QA score:
  
  
  • Only tickets with at least 1 agent message and 1 customer message are scored
  • The total conversation (all messages combined) must be longer than 250 characters

What happens to scores when two tickets are merged?

When you or a member of your team merges tickets, we will keep the current score of the “target ticket” (the ticket you are merging with).

Can I create custom QA criteria?

No, not at this time. Auto QA is designed to evaluate tickets based on commonly used criteria that most CX- focused brands use to assess the quality of customer interactions.

What can I do if I disagree with a score?

If you or another reviewer disagree with how Auto QA scored a ticket, you can always edit any aspect of the score in the AI Feedback tab of the conversation. Any changes you make will be reflected in the Statistics of your Auto QA report.

How is my ticket data used by the AI?

Auto QA uses OpenAI’s GTP-4o to automatically score categories. Your data is processed under a strict Data Processing Agreement (DPA) and zero retention policy. Your data is never used to train OpenAI models.

What languages does Auto QA support?

Auto QA should be able to read and assess tickets in any language supported by OpenAI’s GTP4 model.

Any comments that Auto QA provides with its assessment of a ticket will be written in English.