Start with ordinary failure handling
We put several controls around the chatbot. It had spending limits, a cap on how many messages one sender could submit, timeouts, and a circuit breaker. The circuit breaker stopped the system from repeatedly calling the AI model when it kept failing. Incoming messages also had length and validity checks.
These controls helped manage resource use and prevent the system from being overloaded. However, they could not tell us whether the chatbot’s replies were correct, appropriate, or safe to send. If we counted all these controls as one success measure, we would miss this important difference.
Give people a clear way out
An off switch turned the chatbot off. We made sure that requests to speak to a person were handled before the message was sent to the AI model. If a check blocked a response, the intended fallback directed the sender to a human contact.
These choices gave people another way to get support when the chatbot could not respond. A system should never leave someone stuck just because its automated process failed. It needs a clear backup option, and that option must actually work.
An output filter has limits
Before sending a reply, the chatbot checked its replies for certain problems. It looked for signs that private instructions or secret information had been exposed, and it checked whether the answer was unrelated to the person’s question. If a reply matched one of these problems, the chatbot did not send it.
However, these checks only catch problems that someone has already thought of. They cannot guarantee protection against every prompt injection, where someone uses carefully worded input to try to make a model ignore its instructions. They also do not confirm that answers are factually correct, and they cannot replace a human review of harmful replies.
Keep unresolved decisions visible
Our review also identified work that had not been fully addressed. This includes four areas: removing private or sensitive details from messages, protecting the chatbot from prompt injection, reviewing harmful replies, and investigating problems after they happened.
The chatbot logs recorded each call to the model and the resources it used. However, they did not store enough of the conversation to show exactly what a customer had been told.
Saving more conversation data would have helped with investigations, but it would also have raised privacy questions and data-retention responsibilities. This needed a clear policy, rules about who could access the data, how long to keep it, and who was responsible for the policy. Calling this just a “logging task” would make the work seem simpler than it really was.
Review whether the feature still earns its place
We removed the chatbot when that channel was no longer worth keeping. Retiring a feature that no longer helps is also part of maintenance.
This experience shapes our automation and AI work. For any new automation, we would first start with the task boundary, the person responsible, the failure path, and the evidence needed to review the incident. Only after making those decisions would we decide how many checks or controls to add.