Confidence Score of the AI Classifier: How to Configure

Confidence Score of the AI Classifier: Configuration for Optimal Automation

In Postigo, the AI classifier analyzes responses to your cold email campaigns, determining their relevance and intent. A key parameter affecting classification accuracy is the confidence score. This article explains how to interpret the confidence score and adjust its threshold to achieve your goals.

What is Confidence Score?

Confidence score is a numerical value from 0 to 1 that the AI classifier assigns to each response. It reflects the algorithm's confidence in the correctness of the assigned classification (e.g., "Positive Response", "Rejection", "Not Relevant").

• 0: Minimum confidence. The classifier is hardly confident in the correctness of the result.
• 1: Maximum confidence. The classifier is absolutely confident in the correctness of the result.
• Values between 0 and 1: Reflect the degree of confidence. For example, 0.8 means that the classifier is 80% confident in the correctness of the assigned label.

For example, if the classifier identifies a response as "Positive Response" with a confidence score of 0.9, it means that it is very confident that the recipient is interested in your offer. Conversely, a score of 0.3 indicates a high probability of error.

Default Confidence Score Threshold

By default, Postigo sets the confidence score threshold at 0.7. This means that only responses that receive a score of 0.7 or higher will be automatically classified and, depending on your campaign settings, may be used for automatic follow-up or other actions.

This threshold was chosen as a compromise between accuracy and coverage. It allows you to filter out most irrelevant responses, minimizing the number of errors during automatic processing.

When to Raise the Confidence Score Threshold (0.8-0.9)

Raising the confidence score threshold to 0.8-0.9 is recommended in the following situations:

• Strict automatic follow-up: If you want to fully automate the follow-up process and minimize the need for manual verification, raising the threshold will ensure that follow-ups are sent only to the most promising leads. This is especially important if you are using AI content generation for personalized emails and want to be sure of their relevance.
• High volume of incoming responses: If you receive a large number of responses and do not have the resources to process them manually, raising the threshold will help you focus on the most promising leads, filtering out less relevant ones.
• Critical classification accuracy: In some cases, such as recruiting, an error in classification can lead to the loss of a valuable candidate. Raising the threshold minimizes the risk of such errors.

Example: Imagine you are running a cold email campaign to attract B2B clients in the financial services industry. An error in classifying a response could lead to sending a commercial offer to a company that is already your client. In this case, raising the confidence score threshold to 0.8-0.9 will help avoid awkward situations and increase the effectiveness of your campaign.

When to Lower the Confidence Score Threshold (0.5-0.6)

Lowering the confidence score threshold to 0.5-0.6 is recommended in the following situations:

• Need for greater coverage: If maximum coverage is important to you and you are willing to spend time manually reviewing responses, lowering the threshold will allow you to identify potentially promising leads that may have been missed at a higher threshold.
• Insufficient number of responses: If your campaign receives few responses, lowering the threshold can help you increase their number and get more opportunities for interaction.
• Complexity of classification: In some cases, for example, when working with highly specialized industries, it may be difficult for the classifier to unambiguously determine the relevance of the response. Lowering the threshold will allow you to see more responses and independently assess their prospects.

Example: You are running a campaign to find new suppliers for your production. Even a response containing general phrases about cooperation may be potentially interesting. Lowering the threshold to 0.5-0.6 will allow you to see more of these responses and conduct a more detailed analysis. It is important to remember the importance of spam protection, even at a lower confidence score threshold.

How to Change the Confidence Score in Postigo

The confidence score is configured in the settings of your AI classification in Postigo. The specific location of the setting may vary depending on the platform version, but is usually located in the "AI and Automation" or "Campaign Settings" section.

Attention: Changing the confidence score threshold will affect all future classifications. It is recommended to test different values on a small group of responses to determine the optimal threshold for your campaign.

Additional Recommendations

• Regularly analyze the results: Track how changing the confidence score threshold affects the number and quality of automatically classified responses. Use Postigo analytics to evaluate the effectiveness of your campaign.
• Use A/B testing: Conduct A/B testing with different confidence score values to determine the optimal threshold for your target audience and campaign type.
• Improve data quality: The better the quality of the data used to train the AI classifier, the higher the classification accuracy will be. Make sure your email templates and target lists are optimized for maximum relevance. Don't forget about Spintax to diversify content.

Properly configuring the confidence score of the AI classifier in Postigo will allow you to optimize the response processing, increase the effectiveness of your cold email campaigns, and focus on the most promising leads. Do not forget about the importance of SMTP warm-up and compliance with SPF, DKIM, and DMARC security rules to ensure high deliverability of your emails.