chatJPt Update: a Tribal Knowledge Switch, and Why a Search Kept Answering the Wrong Question

chatJPt (chat.jpaul.me) answers Zerto, HPE Morpheus, HVM, and OpsRamp questions from the official docs. That’s mostly true. The docs servers behind it also carry a layer of curated material that isn’t on help.zerto.com: hand-written notes on Zerto REST API gotchas, version pairing rules, per-platform capabilities, upgrade traps, and the working code samples from Zerto’s public zerto-api-quickstart repo. I call that layer Tribal Knowledge. Until this week you couldn’t turn it off, and you couldn’t easily tell when an answer leaned on it.

Now you can. And making that switch work properly led to three rounds of search fixes I didn’t plan on. One of them only happened because Claude got a Zerto fact wrong and I had to correct it.

Disclaimer up front

Personal side project. Not a Zerto or HPE product, not supported by Zerto or HPE, not endorsed by my employer. The official docs side is 100% publicly available documentation. The Tribal Knowledge side is curated notes and public sample code, and it’s exactly the part you should check before you trust it. That’s why it now has an off switch.

The switch

There are two places to set it:

• Your default, from the menu under your email in the bottom-left corner. It’s on unless you turn it off.
• Per chat, from the ⚙ button next to the message box. You can flip it in the middle of a conversation and it applies to the next answer. When it’s off, the ⚙ gets a small orange dot so you can see it at a glance.

The obvious way to build this would have been to hide the curated tools from the model when the switch is off. That doesn’t work. Curated content doesn’t only live in its own tools. It also shows up mixed into normal search_docs results, ranked right alongside the vendor pages. Hiding three tools would leave the curated notes coming in through the main search.

So the switch is a flag sent to the docs servers themselves. Every search call from chatJPt now carries official_docs_only=true or false, and the server filters its own results. All four servers (Zerto, Morpheus, HVM, OpsRamp) take the same flag, and every curated result is marked [curated] so it’s visible either way. chatJPt also removes that flag from what Claude sees, so Claude can’t override your choice. If you turn the switch off halfway through a chat, curated results from earlier turns are hidden from Claude on the next answer. They stay in your saved chat, and turning the switch back on brings them back.

You’ll also notice a “What’s new” pop-up the first time you sign in after an update. The first post in it explains this switch, and older posts stay listed under “What’s new” in the same menu.

Then I checked whether the curated stuff was even showing up

Once the switch worked, I ran a set of test questions against the production Zerto docs server with it on and off. The off side was clean: zero curated results on every query. The on side was the surprise. A lot of the time the curated notes didn’t appear at all, even for questions they exist to answer.

“Which ZVM versions can be paired together” returned no curated results. Neither did “zerto upgrade gotchas,” although there’s a curated note written specifically about an upgrade blocker. The vendor docs won every time, and the curated pages sat below the top 5, where the model never sees them.

The cause was formatting. The curated docs were written like reference sheets: a heading, then a big table. The final ranking step in the search (a reranker, a second model that re-scores the top candidates against the question) consistently scores tables lower than prose. The vendor pages open with a sentence that answers the question. Mine opened with a grid.

So the Claude session that maintains the Zerto docs server rewrote the four hand-written curated docs “answer-first”: every section opens with one to three plain sentences that answer its heading, and long sections are split so each one fits in a single search chunk. Facts were not supposed to change, and it checked. A diff of every URL, number, and quoted vendor sentence before and after showed zero dropped. Across 19 test questions, a curated result at rank 1 went from 10 to 14, and a curated result anywhere in the top 5 went from 14 to 17. The official-docs score on the 25-question eval set didn’t move (0.440 both before and after).

One lesson worth passing on if you write docs for a search index: question words in a heading help that section, but a long top-level heading gets prepended to every subsection under it, and that hurts all of them. I’d asked for “Authentication” to become “Authentication: how to get a working token for the Zerto REST API.” That change knocked the auth section from rank 4 to not returned at all. Short parent headings; question words in the leaf headings and the first sentence.

The VPG question, and where Claude got it wrong

Two of the test questions still didn’t surface curated content after the rewrite, and that was on purpose. “Can a ZVM 10.0 site pair with a 9.5 site?” is answered directly by the vendor’s pairing matrix, and the curated pairing notes say in writing that they don’t cover pre-10 pairing. A curated note shouldn’t outrank the vendor on a question it declines to answer.

The interesting one was a control question: “how to create a VPG in the vSphere web client.” With Tribal Knowledge on, all five results were code samples from the API quickstart. Python, Bash, PowerShell. No UI page at all. Both Claude sessions flagged it as an intrusion: someone asked a UI question and got handed API scripts.

I’ll be honest, that framing was wrong, and it took me a minute to spot it. In Zerto 10.x you can’t create a VPG in the vSphere web client. You create it in the Zerto UI or through the API. The question has a false premise, so the API samples weren’t wrong, just incomplete. The right answer is “you can’t do it there; here are the two places you can.”

Then I asked the obvious follow-up: is there anything in the docs that would make an AI think you can? Yes, two things.

• The 9.0 docs describe a Zerto plug-in inside the vSphere Web Client, with a Zerto tab and a PROTECT AS A STANDALONE VM button that opens the Create VPG wizard. That plug-in is gone in 10.x.
• The current 10.9 page, Creating Virtual Protection Groups (VPG), still has a table row for “To a different site using the Client Console or VMware Web Client” with an empty procedure cell. The 10.0 page has the same row. It’s a leftover from 9.x. I submitted a doc bug for it.

Nothing in the corpus actually said “VPGs are created in the Zerto UI or the API, not the vSphere client.” So I wrote that down, and the Zerto docs session added it as a new section in the curated platform capabilities doc, with each claim linked to its source page. That is exactly what Tribal Knowledge is for: the fact everyone in the field knows and nobody wrote down.

Things that already broke

Adding the fact didn’t fix the search. The new section ranked first in both the keyword search and the vector search for that question, and the reranker still put it sixth. Four of the five results above it were chunks of the same file: one Python script, split into pieces, each piece scoring slightly higher than my one-paragraph answer.

That turned out to be the real bug, and it wasn’t specific to curated content. search_docs never merged results from the same page, so any page split into enough chunks could take every slot. The best part is that the Zerto docs server’s own eval code already merged repeated pages before scoring, with a comment saying two chunks of the same page “aren’t two answers, they’re one.” The search users actually get never did it. So the eval scores could never see the problem: 12 of 95 result slots across the test questions were repeats of a page already shown, and no number moved.

The fix shows each page once and allows at most two pages from any one doc set (one “bundle”), before cutting the list to the top 5. Nothing gets filtered out; it only changes the order. The vSphere web client question now returns five different pages:

1. The API quickstart’s Python create-VPG script
2. The API quickstart’s Bash create-VPG script
3. The new curated section on where a VPG can be created
4. The 10.9 Creating Virtual Protection Groups page, with the empty table row right there in the result
5. Zerto’s official API workflow for creating a VPG

That’s the correction, the evidence for it, and both real ways to do it, in one result set. A plain “how do I create a VPG” still leads with the vendor’s create-VPG page, which is correct. And with Tribal Knowledge off, every test question still returns zero curated results.

A couple more potholes from the same two days:

• The flag got renamed mid-build. It started as tribal_knowledge=true and became official_docs_only=true once the docs servers settled the contract, because “Tribal Knowledge” is only chatJPt’s name for it. Every test, the README, and three issues had to follow.
• The AI sessions talk to each other now, and that cuts both ways. This work was split between a Claude Code session on chatJPt and one on the Zerto docs server, passing messages and issue numbers back and forth. It’s fast. It also means one of them merged and deployed the duplicate-page fix before I’d said go ahead. The fix was right and I verified it, but deciding when to change what every search returns is my call, not the model’s.

What’s next

If chatJPt gives you an answer that leans on Tribal Knowledge and it’s wrong, I want to know. Email me at [email protected] with the question. And if there’s a fact about Zerto that everybody in the field knows and nobody has written down, send that too. It’s probably the next section in the curated docs.