Error Handling

It's well known that AI can deviate from the source of truth, basically tell lies. In this module we'll take you through a few basic things that will help reduce hallucinated responses and also how to use the scoring value to direct users to other places in your workflows.

Manage hallucinated responses

Thinking back to the previous module where we spoke about Embedding Match & Completions, we'll start there. So head back in to your flow and navigate to the Embedding Match and Completions action step. Once in the action step let's focus on the Introduction bar.

As mentioned in the previous module: The introduction is just like a prompt, so if you want to add an additional prompt or instruction here then you can 'optional'

OK, so let's add a basic prompt to the Introduction tab.

"Question {{last_text_input}}. assistant: extract data only from {{embed_text}} do not respond with any context that is not within the extracted data."

This prompt using the information that we captured in the embedding match action step {{embed_text}}, so what this will do is, provide the AI the exact content from the embedding again. With the intent to only use the provided context to extract an answer from.

Another way to help reduce hallucinations is by actually heading back to the embedding and placing a prompt there instead. So head over to your Integrations tab > OpenAI and select the embeddings. Let's look inside the embedding

So we mentioned back in the 'How to create embeddings' module that you could inject a prompt within the embed itself. We actually find this a better way than placing the prompt in the Introduction tab shown earlier. Here's a simple prompt that we inserted into one of the embeddings called Integrations:

This really helped the AI stay on track and respond perfectly. All we done here was tell the AI in a heading called 'Assistant' that we only integrate with the list below, and any other integrations can be fulfilled via Zapier, Make or using external requests. If you have API documentation, then you can build your own application within Flochat.

And it done exactly that! We threw lots of questions about integrations that we currently don't have 'Yet' and to use Zapier etc etc.

A quick way to check if your AI diverts from the source of truth, is by testing a well know question. Try this: How far is the moon from earth.

By doing this it shows you that by not responding to it, your AI is only focused on your content. This is done via the Scoring.

How to use scoring

Scoring is possibly the most important part for your AI set up. When you test your Embedding Match action, you would of seen the data sent back from the AI containing a score.

So once we have the score saved in a number custom field, we can now use it with a condition to filter the AI response based on the score. in the below image is the basic setup, then we'll show you how to create the condition and what to do with it.

To create the condition based on the Score, open a condition node - type in your score custom field - set the operator to: greater than or equal to and the value at 0.2 or up to 0.3.

The reason why we chose 0.2 or 0.3 is that anything below that score could not be found in your embeddings, ie asking how far it is from the moon to earth. You can then either display a message that asks them to try again, or send to another flow where you may handle a human take over flow.

As always be sure to test your self when using the scoring, to help you get the best output and also reduce the hallucinated responses.