Thoughts on how to prevent your AI chatbot from becoming a very expensive hat

Are Public Support Knowledge Bases Ready for AI Agents?

Jun 09, 2026

Everyone’s racing to fashion an AI chatbot onto their existing knowledge base and move on. But a chatbot can only answer from what it can actually retrieve. So how do you know your knowledge base is solid enough to put one in front of customers?

Summary

I examined the external help content of four major consumer platforms — DoorDash, Uber, Lyft, and Instacart — and behaved like a customer looking for help regarding a recent ride or a delivery order. I searched each company’s customer support site and asked questions like: “My order says delivered but I never received it,” and “My driver never arrived but I was still charged,”

66% of the time, I couldn’t find the help I needed, not because the information wasn’t there, but because I couldn’t retrieve it. If the right article exists but cannot be reliably retrieved from natural customer wording in regular search, then the AI chatbot will inherit that retrieval problem. The model may be capable of writing a good answer, but it will be grounded in the wrong source, or no source at all.

Taking inspiration from Wix, IBM, and Databricks + Stanford University

AI support agents can only give a safe and useful answer if the underlying knowledge base is findable, current and complete, well-structured, and explicit about policy boundaries. While I know this from my experience as a technical writer, I wanted more backup, so I turned to academic and industry teams with likeminded goals.

I reviewed the following three papers:

WixQA’s core finding was that enterprise RAG evaluation needs to be grounded in a fixed knowledge-base snapshot. I adapted that idea into a smaller public-support snapshot across Uber, Lyft, DoorDash, and Instacart. I also borrowed WixQA’s emphasis on realistic customer questions. My benchmark questions are written in the way that mimics how a customer might ask for help: in messy, natural language.

From MultiDoc2Dial, I took the idea that real support interactions often move across topics and documents. I tracked whether each benchmark question could be answered by a single article, multiple articles, or an in-app-only flow.

From ARES, I took their three-part evaluation frame and used it to shape my rubric categories:

Context relevance: Can the right support source be found, and does it contain the needed information?
Answer faithfulness: Can an AI answer stay within the source without inventing policy or overpromising refunds?
and Answer relevance: Does the answer actually help the customer take the next safe step?

Why I looked at DoorDash, Uber, Lyft, and Instacart

I chose to look at Uber, Lyft, DoorDash, and Instacart because they service some of the most fundamental aspects of living: sustenance and transportation. Because of the nature of their services, their customer satisfaction and business success are directly tied to adequate customer support.

How I designed the experiment and organized my findings

I limited the audit to support issues that are high-frequency and high-risk, such as:

Missing service or delivery
Wrong or incomplete order
Cancellation fees
Duplicate, unexpected, or disputed charges
Refund status or refund eligibility
Support escalation

1. Create realistic benchmark questions

I put myself in the customer’s shoes to create a small benchmark of 24 realistic customer questions, 6 questions per company. The questions were written in natural language, because I wanted to test whether public support content could be found from the way users actually describe problems.

For a full list of the question, refer to Appendix A.

2. Design, calibrate, and lock the evaluation method

I designed a scoring rubric around three evaluation categories:

Source and context quality
Groundedness and policy safety
Customer resolution usefulness

Under source and context quality, I scored whether the right article could be found, whether the public KB contained enough information to answer, what type of source evidence was available, and whether the article had a visible freshness signal.

Under groundedness and policy safety, I scored whether policy rules were clear, whether the answer required account-specific data, whether an AI agent could answer safely from the public source, and whether the source created risk of unsupported claims.

Under customer resolution usefulness, I scored whether the customer would know what to do next and whether the escalation path was clear when the public documentation was insufficient.

I used a simple 1–3 scale for each scoring dimension:

3 = clear, complete, or agent-ready
2 = partially answerable, but requires caveats or caution
1 = risky, missing, ambiguous, or not answerable from public documentation

I also assigned one primary failure mode to each evaluated question. This helped separate different kinds of problems, such as hard-to-retrieve content, query phrasing sensitivity, unclear refund eligibility, account-specific dependency, missing escalation paths, or guidance scattered across multiple pages.

Before scoring the full dataset, I ran a calibration pass on four representative questions: one each for DoorDash, Uber, Lyft, and Instacart. This helped me test whether the source fields, scoring definitions, and failure modes were clear enough to use consistently across the four companies.

After calibration, I froze the rubric so the rest of the audit would be scored consistently.

3. Document the search experience on the company help site, Google search, and Google AI Overview

I plugged in each question into the company’s help site and google. I took note of my search experience in the company help site. Was I able to find the correct article by their heading and excerpt? Did the correct article show up high enough in the ranking? Did the article I chose have the answers I needed? Did I have to click on another link embedded in the first link I clicked on? Did I have to leave the help site to find answers elsewhere, like Google?

Similarly, I took note of my experience using Google Search, including the Google AI Overview.

4. Capture metadata from each relevant support article

For each benchmark question, I was able to find 1 or more articles that answered the question adequately enough. I considered each article a “source”.

For each source, I recorded the source URL, article title, access date, visible update dates when available, and a short summary.

I also captured metadata about what kind of information the article contained: policy language, procedural steps, app-only instructions, support routing, refund or credit language, escalation paths, and whether the answer depended on account-specific information.

Now each question was mapped to an expected source article or source set. for each question, I also labeled whether it required a policy answer, account-specific action, escalation, or synthesis across multiple sources.

5. Evaluate...without AI

Once I gathered all the information, I manually scored the question-source article pairs using the rubric I created. When in doubt, I erred on the conservative side.

Results: The information is there, so why couldn’t I find it?

Across all four companies, I found that the support articles themselves were detailed and well-written. But I could not, for the life of me, search and retrieve them through the company help site. I’d amend my query twice and thrice, but even then, the help site would not surface the right article or showed no results at all. In these cases, I turned to Google, which would often find me the official company article I needed, either in the search results or the AI Overview.

Across the 24 questions, the three dominant failure modes were hard_to_retrieve, query_phrasing_sensitivity, and missing_policy_condition. Together, they account for 19 of 24 rows, more than two-thirds of the benchmark.

That means the article often existed and was usable, but the company help-center search did not surface it from realistic customer wording.

Company-level scores were close

No single company scored dramatically better than the others. Instacart came out highest overall at 2.5, DoorDash at 2.4, and Lyft/Uber at 2.3.

Companies differed more by failure style than by overall score

The more interesting differences were failure styles:

Instacart: strongest content and escalation clarity, but weak internal search.
DoorDash: good support routing and some strong policy articles, but many natural-language searches failed.
Uber: some of the strongest policy content, especially cancellation fees, but many issues branch into multiple charge types.
Lyft: good rider policy pages, but noisy retrieval.

Billing disputes and refund statuses were trickiest

The question types that were hardest to get answers for were:

duplicate_or_unexpected_charge — 1.8
wrong_or_problematic_service — 1.8
refund_status — 2.2

The easiest were things like missing_items, cancellation_fee, lost_item, membership_or_subscription_charge, and support_escalation.

Google AI Overview behavior: fluent answers sometimes added unsupported certainty

Google AI Overview answers tended to add information that was unsupported by the source material. This included:

Incorrect refund timing
Fare-adjustment promises
Extra recovery steps
Non-official sources, like Reddit, Quora, Facebook groups, etc

So what does this mean for teams who want to attach AI to their existing customer support knowledge bases?

If your team’s existing support content is available and detailed, but not retrievable through messy, symptom-based customer search, it’s not ready for AI. Here are a couple ways teams can improve their knowledge bases in preparation for AI-agent support.

Attach audience labels to your support articles

Support articles should clearly identify who they apply to: customer, rider, driver, shopper, merchant, subscriber, or support agent.

This matters because similar support language can apply to different audiences. One example of this problem I saw was on Lyft’s help site. When I asked about a cancellation fee as rider, the top result was driver-facing, while the rider-facing article ranked lower. Audience metadata should be used in search ranking and AI retrieval so that rider questions retrieve rider content and driver questions retrieve driver content.

Create machine-readable decision tables

High-risk policy content should be converted into structured decision tables, with approval from legal and PR teams. They should not be left as narrative help articles.

This is especially important for refunds, cancellation fees, disputed charges, and fare adjustments. These issues often depend on conditions the public article may not fully expose: charge type, order status, cancellation timing, payment method, account history, or support review.

A decision table should clearly demonstrate:

What facts are required
What the agent can safely say
What requires account-specific review
What outcomes are possible
What the agent must not promise
When to escalate and connect the customer to live support channels

This would help prevent AI agents from turning “You may be eligible for a refund.” into “You will get a refund.”

Improve natural-language retrieval

Companies should test whether realistic customer wording retrieves the correct support article.

The dominant failure pattern in this audit was retrieval. The right article often existed, but company help-center search did not surface it for natural phrases like “I was charged twice,” “my driver never came,” “my order says delivered but it never showed up,” or “I need to talk to someone.”

Support teams should add customer-language synonyms, symptom phrases, and query variants to high-risk articles. They should also maintain a golden question set to test whether natural customer questions retrieve the right source before deploying AI support agents.

For AI support, the goal is not to retrieve just any answer for the customer. The goal is to retrieve the right answer for the customer’s issue, based on audience, and policy boundary.

Appendix

A. Benchmark questions

DoorDash

My order says delivered, but I never received it. What should I do?
My food showed up but some items are missing. Can I get money back?
I canceled my DoorDash order but I still got charged. Why?
DoorDash charged me twice for the same order. How do I fix it?
DoorDash said I got a refund, but I don’t see it in my bank account.
The app isn’t helping with my order problem. How do I talk to someone?

Uber

My driver never showed up, but Uber still charged me. Can I get a refund?
My Uber fare was way higher than expected because the driver took a weird route.
I canceled because my Uber driver was late. Why did I get charged a fee?
There’s an Uber charge on my card that I don’t recognize.
Uber said I’m getting a refund, but I don’t know when it’ll hit my card.
I can’t figure out how to contact Uber about a ride problem.

Lyft

I canceled because the driver wasn’t moving. Why did Lyft charge me a cancellation fee?
I left something in my Lyft. How do I get it back?
My Lyft ride cost more than the price I saw before booking.
Lyft put a pending charge on my card. Did I actually get charged?
I disputed a Lyft charge. How long does it take to get my money back?
I need help with a Lyft charge but I can’t find where to contact support.

Instacart

My groceries are missing several items. Can I get a refund?
I got the wrong item in my Instacart order. What do I do?
My Instacart order says delivered but it never showed up.
Instacart charged me the wrong amount. How do I report it?
Instacart says it refunded me, but the money isn’t back in my account.
I got charged for Instacart+ and I don’t want it. How do I cancel or get help?

Limitations

This is a Knowledge Base-readiness audit, not a RAG benchmark. This project evaluates whether public docs are ready to back an AI agent. It does not measure the performance of any specific retrieval or generation pipeline.
The audit scores publicly visible help-center articles only. I don’t have access to internal policy, agent macros, routing logic, internal tools, account-state systems, or proprietary playbooks.
The information used in this article is contained to early June, 2026. KBs change frequently. Findings reflect the source state on the dates listed in data/raw/support_kb_sources.csv (primarily 2026-06-01). Any rule, rate, or routing path may have changed since.
English-only. Localized content was not evaluated. Findings may not generalize to non-English versions of the same KBs.
All rubric scoring was performed by a single rater, me. Scores are auditable but subjective.

References

Cohen, Dvir, et al. “WixQA: A Multi-Dataset Benchmark for Enterprise Retrieval-Augmented Generation.” arXiv, 2025, https://arxiv.org/abs/2505.08643.

Feng, Song, et al. “MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents.” Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2021, pp. 6162–6176.

Saad-Falcon, Jon, et al. “ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems.” Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, 2024, pp. 338–354.

Discussion about this post

Ready for more?