CivicPlus Agent Confidence Score

This page explains what the CivicPlus Agent confidence score means, how it is determined at a high level, and how to interpret it when reviewing conversations and analytics.

In short: Confidence measures how well-supported an answer is by relevant sources. It reflects evidence quality, not writing quality.

What is a Confidence Score?

Every answer the CivicPlus Agent gives is paired with a confidence score. This score is a signal for how well the answer is supported by real, verified information sources available to the agent.

Confidence is meant to help interpret how grounded an answer is. It is not a score for tone, readability, or how polished the response sounds.

A short answer with strong supporting sources can have higher confidence than a longer answer that sounds helpful but is not well supported by source material.

What Confidence Measures

Confidence reflects the quality and quantity of supporting evidence behind a response.

• Strong confidence (70 to 100%): The answer is backed by multiple relevant, authoritative sources that directly address the resident’s question.

• Moderate confidence (40 to 70%): The answer is backed by at least one relevant source, but the support may be partial, limited in scope, or less authoritative.

• Low confidence (0 to 30%): No relevant sources were found, or the available sources do not actually answer the question that was asked.

• Not applicable: Confidence does not apply to simple conversational exchanges such as greetings, clarifying questions, or off-topic redirects, because there is no factual claim to support.

Examples of authoritative sources can include city documents, official web pages, and verified meeting records.

What Confidence Does Not Measure

Confidence should not be interpreted as a general quality score for the conversation.

• It does not measure how well written the answer is.

• It does not measure whether the resident liked the answer.

• It does not guarantee the answer fully resolved the resident’s need.

• It does not indicate whether a topic is important, popular, or trending.

This distinction matters because an answer can be well supported by sources and still fail to fully help the resident, or it can be phrased nicely while lacking reliable evidence.

What Happens at Low Confidence

When confidence is low, the CivicPlus Agent is designed to avoid guessing or filling in gaps with unsupported information.

Instead, the agent should communicate that it could not find a reliable answer and direct the resident toward a safer next step, such as:

• Contacting the appropriate department.

• Checking an official city resource.

• Rephrasing or narrowing the question.

Note:

This cautious behavior is intentional. The goal is to reduce the risk of giving residents confident-sounding but incorrect information.

Why This Matters

The confidence score helps explain how much evidence the agent had available when producing an answer.

This is important because the CivicPlus Agent is designed to prioritize trustworthy answers over overly broad or speculative ones. When strong supporting sources exist, the agent can answer more directly. When they do not, the agent should be transparent about that limitation.

This approach helps protect residents from inaccurate answers and gives administrators a clearer view into where knowledge coverage may be strong, limited, or missing.

How to Interpret Confidence in Practice

• High confidence usually means the agent found multiple strong sources that directly support the answer.

• Moderate confidence usually means the agent found some support, but the evidence may be incomplete or less definitive.

• Low confidence usually means the source set was weak, missing, or not relevant enough to safely answer the question.

Confidence is best used as one signal alongside conversation review, resolution outcomes, and topic patterns.

Frequently Asked Questions