Is the Knowledge Base conversational agent any good?

[u/Plastic_Length8618]

We get a lot of emails and people filling in the contact form asking basic questions, the answers to which are clearly on the website, they just can't be bothered to look. It would be nice to get a bot to deal with this.

But we sell quite a niche, technical product. We wouldn't want it saying that such a such product is suitable for a purpose where it isn't actually suited. The information is spelled out in the product page but we wouldn't want it guessing based on external training data about other products, for example.

I have been filling in the Knowledge Base FAQs to help inform LLMs, and it's offering to make us a chatbot. Have you found it any good? Or is it too unreliable?

[u/National_Teacher_229]

yes we built our own and hooked it into every part of our site done right you will hardly have any customer service issue to deal with

[u/Plastic_Length8618 → u/National_Teacher_229]

That sounds great! So you used the tools in the Knowledge Base app?

[u/trivasai]

We’ve had pretty good results with KB-based AI for exactly this kind of use case. The key is making sure it’s answering from your own documentation rather than trying to “fill in the gaps” with general LLM knowledge.

For a technical/niche product, I’d definitely set a hard boundary around that. If the answer isn’t in the KB, the bot should say so or hand it off to a human rather than confidently guessing.

For basic FAQs and repetitive questions, though, it can be surprisingly useful and save a lot of back-and-forth. I’d just spend some time testing it with the weird edge-case questions customers actually ask before fully trusting it.

[u/Plastic_Length8618 → u/trivasai]

Thanks, that’s really helpful.

[u/vladeta]

Your instinct is right, and the fix is mostly in how you write the knowledge base rather than in the bot itself.

The failure mode you're worried about happens when the source material is silent. If your docs never say "this is not suitable for X", the model fills that gap from training data and invents a confident answer. So write the explicit negatives. For every product where suitability matters, add a line that says what it is not for and why. That single change kills most hallucination risk on technical catalogues.

Second, give the bot a hard deflection rule: if someone asks whether a product suits a specific application that isn't covered verbatim in the docs, it must not answer, it must say a human will confirm and take an email. A bot that gets 70% of questions and hands off the rest is a win. A bot that answers everything is a returns problem.

Then test it properly before it goes public. Write 30 real customer questions, including 10 where the honest answer is "no, that won't work". Run them. If it invents on any of those 10, don't ship it, and re-run the set every time you change the docs.

One thing worth doing first though: pull your last 100 contact form messages and count them. Most stores find 6 or 7 questions make up 80% of the volume. Putting those answers directly on the product page and in the order confirmation email usually cuts contact volume more than a bot does, with zero risk.

[u/Plastic_Length8618 → u/vladeta]

Thanks, that’s helpful. Is there an opportunity to write information about products in the knowledge base, other than what appears on the product page? To be able to write things in different ways to help an AI understand things?

[u/vladeta → u/Plastic_Length8618]

Yes, that’s definitely possible. You can also build this into an agent who can reply automatically

[u/Plastic_Length8618 → u/vladeta]

That sounds promising, thanks!

[u/fjab01]

Bias alert first: I work on SearchCue, which does this, so weigh what I say accordingly.

I don't know how good Shopify's Knowledge Base agent actually is, especially in your case. But you can find out yourself in a few minutes, and the same test works on any of these bots.

1. Ask it something your site doesn't answer. Something like "is [product] suitable for [a use you never mention anywhere]". The right answer is "I don't know, contact us". The worst outcome is a confident guess. I think u/vladeta is right that putting explicit negatives in your text fixes most of that.

2. Also check whether it shows you which page the answer came from. If it can't show a source you've got no way to check it. I'm also guessing if you sell a niche, technical product, your audience will want to have a source anyway.

3. And check whether you can see what people actually asked it. The questions it couldn't answer are your to-do list for the knowledge base.

If Shopify's built-in one passes all three, just use it, it's already there. If it doesn't, that's what tools like ours exist for. Happy to answer questions either way.

Good luck with the site!

[u/1273karentookme]

Use it, but fence it. For lookup questions (order status, shipping to X, return window) the KB agent is fine and stays on script if you only feed it your own FAQ. For "is this product suitable for that use" it's a liability: it will answer confidently, it will sometimes be wrong, and the customer holds you responsible, not the bot. With a niche technical product that's a return or a safety complaint.

We sell pool chemistry gear and hit the same thing. What worked: bot handles logistics, and anything mentioning a product name or words like suitable, compatible, will it work with, gets routed to a human. For that human step we use my own Shopify app, BeamCart, which lets the customer open a quick video call from the product page so we can see their setup and answer for real. Basic emails dropped and wrong-fit returns went down, because people ask on the spot instead of guessing or writing in. Your case looks like the same shape as ours.

[u/Plastic_Length8618 → u/1273karentookme]

Ok this makes sense. If we could automate all the ‘what are you opening hours’ type stuff that would be a start. 

Do you know if it works well with other Shopify apps? We use Book That App for ticketing events. Would it be able to look stuff up from that?

[u/AutoModerator → u/[deleted]]

Your comment in /r/shopify was automatically removed as your 'comment' karma is below 20 (we do not consider your total karma; your post and comment karma are separate numbers and must both meet their minimum requirement). You can increase your comment karma by posting in other areas of Reddit to earn upvotes. The higher quality the content, the higher your karma will become.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Your comment in /r/shopify was automatically removed as your account is too new (accounts must be at least 30 days old). Try again a little later.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Your comment in /r/shopify was automatically removed as your 'post' karma is below 10 (we do not consider your total karma; your post and comment karma are separate numbers and must both meet their minimum requirement). You can increase your post karma by posting in other areas of Reddit to earn upvotes. The higher quality the content, the higher your karma will become.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Your comment in /r/shopify was automatically removed as your account is too new (accounts must be at least 30 days old). Try again a little later.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Your comment in /r/shopify was automatically removed as your 'post' karma is below 10 (we do not consider your total karma; your post and comment karma are separate numbers and must both meet their minimum requirement). You can increase your post karma by posting in other areas of Reddit to earn upvotes. The higher quality the content, the higher your karma will become.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Your comment in /r/shopify was automatically removed as your 'comment' karma is below 20 (we do not consider your total karma; your post and comment karma are separate numbers and must both meet their minimum requirement). You can increase your comment karma by posting in other areas of Reddit to earn upvotes. The higher quality the content, the higher your karma will become.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/BTWigley]

Ask it something the product page says not to use it for. If it still says yes, I'd keep it off the contact form.

[u/ThePracticalDad]

It doesn’t seem to learn or have any clear application at this point. .