The large language models (LLMs) that power Slackbot can sometimes generate responses that are factually inaccurate or that don’t make sense in the context of a conversation. These responses, known as hallucinations, often sound plausible and are delivered confidently.

Read on to learn more about recognising hallucinations, ways to avoid them and how to report suspected hallucinations to Slack. It’s important to review responses critically and verify information yourself before using it to make decisions or sharing responses with others.

Note: When you work with Slackbot, it’s important to review responses critically and verify information yourself before using it to make decisions or sharing responses with others.

How to recognise hallucinations

AI responses in Slack only contain information that you have access to (such as public channels, private channels that you belong to and direct messages), so if you see unfamiliar information, it’s likely a hallucination. 

Hallucinations that Slack can detect

Here are the types of hallucinations that we can identify and flag in Slackbot responses: 

• Links to content
  References to files that don’t exist in your Slack workspace or Enterprise organisation.
• People, channels and messages
  References to people, channels or messages that don’t exist.

When we detect one of these hallucinations, links, channel names or people will be replaced with a Link not found, Channel not found or User not found tag. If you see flagged hallucinations in a response, you should carefully review the full response for accuracy. 

What to do if Slackbot hallucinates

Slackbot has been trained to be as helpful as possible, but in some cases it won’t have enough information or context to provide a quality response. If Slackbot says ‘I don’t know’ or ‘I don’t have enough information to answer that question’ and you ask it to try again without providing new instructions, it’s likely to start hallucinating in order to give you an answer that you’re satisfied with. 

Use these tips to verify information that Slackbot sends you or to help get back on track when it’s having trouble providing an answer: 

• Ask Slackbot again
  Ask Slackbot the same question in a new chat and see if the new response is consistent with the original reply. 
• Keep conversations short
  If you’ve sent several messages in the same conversation, start a new discussion to reduce the risk of Slackbot getting confused by earlier replies. 
• Provide specific sources
  Rather than saying ‘Look in other places for more info’, try ‘Those aren’t the right channels, please try #channel-name and #channel-name instead. You can also look for messages from @person.’

Report hallucinations

If AI response feedback submissions are enabled for your Slack workspace or Enterprise organisation, you can report responses with hallucinated information. These reports help us to identify patterns and improve Slackbot. Here’s how: