Frequently Asked Questions
Q. Can I test any chatbot using Reputable?
A. Provided you have login details, and the bot-to-be-tested can be addressed over the internet, the Reputable Test Bot can run a conversation.
Q. Do you handle audio?
A. Not yet. For reasons of accuracy, we are currently focussing on text-based conversations.
Q. If a bot-under-test passes a Reputable test, does that mean it’s guaranteed as safe?
A. No, the Pass indicators are a sign that the bot-under-test is exhibiting desirable behaviours but it’s not a guarantee of safety or any other aspect of performance.
Q. If you are using a bot to test a bot, does the Reputable Test Bot reflect human conversations and human psychology?
A. The Reputable Test Bot is trained on millions of hours of conversation, so whilst it’s not human, it will reflect human patterns. It also reflects the persona that it is given. Our Library Personas are checked by professional psychologists. Custom Personas are written by Reputable.bot customers and are not checked by us.
Q. If I develop a Custom Persona could my competitors see it?
A. No. A Custom Persona is only visible to the customer who created it.
Q. Do you use the test results generated by Reputable.bot to train other bots?
A. No. When you run a test, the results are yours. They are only visible to you when you login. They are not used to train other bots.
Q. How many tests should I run?
A. This depends on the function of your bots and the likely composition of your customer base. Reputable.bot tests how bots respond to different user psychologies, which we simulate using personas. It therefore helps to have a test strategy. If your bots are helping customers whom you expect to be annoyed or anxious, choose personas that reflect this. If the bots are handling routine, less pressured customer queries, then choose personas that are polite, and reflect low levels of customer anxiety or annoyance.
Q. If I change the configuration of my service bots, should I retest?
A. Yes. This is what we call regression testing: every time a change happens in a complex system it requires retesting. Assuming your bots run on top of an LLM, anytime the underlying model version changes, you should retest the performance of your bots.
Q. How do you test a chatbot for safety?
A. Chatbot safety testing is done through "red teaming"—actively trying to break or exploit the chatbot using adversarial prompts to see if it bypasses its safety guardrails. Reputable.bot automates this by using AI-driven testing personas (such as adversarial red teamers) that systematically probe your chatbot for vulnerabilities, toxic output, and prompt injection risks.
Q. What is red teaming in large language models (LLMs)?
A. Red teaming is the process of simulating adversarial attacks or abusive user behaviors against an AI system to uncover security flaws, safety issues, or compliance gaps. Reputable.bot automates LLM red teaming by deploying specialized AI personas that interact with your chatbot in multi-turn conversations to pressure-test its limits.
Q. What are adversarial prompts in AI testing?
A. Adversarial prompts are inputs specifically designed to trick an LLM into ignoring its system guidelines (jailbreaking) or generating harmful/restricted output. Proactively testing your chatbot with adversarial prompts exposes vulnerabilities in its guardrails before they are discovered by real users.
Q. How do you evaluate LLM response quality?
A. LLM response quality is evaluated by measuring how well the chatbot adheres to desired outcomes (like helpfulness and compliance) while avoiding undesirable behaviors (like hallucinations, bias, or leakage of sensitive data). Reputable.bot defines these criteria, runs automated multi-turn test conversations, and outputs structured pass/fail analytics.
Q. Is a company legally liable for its chatbot's output?
A. Yes, courts and regulatory bodies increasingly hold companies legally responsible for agreements, misleading statements, or harmful advice generated by their AI agents. Continuous, regression testing with platforms like Reputable.bot is crucial to demonstrate due diligence and mitigate legal and reputational liability.
Q. What are the EU AI Act transparency rules for chatbots?
A. The EU AI Act requires organizations deploying conversational AI to clearly disclose to users that they are interacting with an AI system, protect user data, and verify the safety of high-risk systems. Reputable.bot helps organizations verify compliance by auditing chatbot guardrails and safety limits.
Q. Does using an LLM chatbot violate GDPR?
A. Using an LLM chatbot does not inherently violate GDPR, but organizations must ensure the chatbot does not collect, store, or leak personally identifiable information (PII) in an uncompliant manner. Reputable.bot tests how chatbots handle sensitive inputs, ensuring they enforce data minimisation and reject attempts to extract PII.