The large language models (LLMs) that power Slackbot can sometimes generate responses that are factually inaccurate or that don’t make sense in the context of a conversation. These responses, known as hallucinations, are often plausible-sounding and delivered confidently.

Read on to learn more about recognizing hallucinations, ways to avoid them, and how to report suspected hallucinations to Slack. 

Note: It’s important to review AI-generated responses critically and verify information yourself before using it to make decisions or sharing responses with others.

How to recognize hallucinations

AI responses in Slack only contain information you have access to (like public channels, private channels you belong to, and your direct messages), so if you see unfamiliar information it’s likely a hallucination. 

Hallucinations Slack can detect

Here are the types of hallucinations we can identify and flag in Slackbot responses: 

• Files and messages
  Links to files and messages that lead to a dead end because they don’t exist in your Slack workspace or Enterprise organization.
• People and channels
  References to people or channels that don’t exist.

When we detect one of these hallucinations, links, channel names, or people will be replaced with a Link not found, Channel not found, or User not found tag. If you see flagged hallucinations in a response, you should carefully review the full response for accuracy. 

What to do if Slackbot hallucinates

Slackbot has been trained to be as helpful as possible, but in some cases it won’t have enough information or context to provide a quality response. If Slackbot says “I don’t know” or “I don’t have enough information to answer that question” and you ask it to try again without providing new instructions, it’s likely to start hallucinating in order to give you an answer you’re satisfied with. 

Use these tips to verify information Slackbot sends to you or help get back on track when it’s having trouble providing an answer: 

• Ask Slackbot again
  Ask Slackbot the same question in a new chat and see if the new response is consistent with the original reply. 
• Keep conversations short
  If you’ve sent several messages in the same conversation, start a new discussion to reduce the risk of Slackbot getting confused by earlier replies. 
• Provide specific sources
  Rather than saying “Look other places for more info,” try “Those aren’t the right channels, please use #channel-name and #channel-name instead. You can also look for messages from @person.”

Report hallucinations

If AI response feedback submissions are enabled for your workspace or organization, you can report responses with hallucinated information. These reports help us identify patterns and improve Slackbot. Here’s how: