What I set up so AI assistants can read and cite this site

I built pappfer.hu with AI assistants in mind as well as human visitors. If someone asks ChatGPT for a freelance Laravel developer in Hungary, I want an accessible, clear description of my work available as a source. Some of the setup is ordinary SEO; some is experimental. Here is what each part does, and what I would not promise it can do.

One thing first, because it saves money: nobody can guarantee that an assistant will cite you. Google says plainly that AI Overviews and AI Mode need no special files, markup or optimization beyond the usual SEO requirements. A page must be indexed and eligible to show a snippet to qualify; that does not mean it will be selected as a source. What you can control is whether the crawlers get in, whether the content is easy to extract, and whether the facts about you are consistent everywhere they appear.

The short version

1. Check robots.txt: one User-agent: * group is enough when all crawlers get the same rules. Add a per-bot group only when its rules need to differ.
2. Make sure OAI-SearchBot and Claude-SearchBot can reach you. They collect content for search; allowing training crawlers does not replace that access.
3. Describe yourself or your company in JSON-LD, with a stable @id for each entity, reused wherever you refer to it.
4. Give quotable definitions and sections stable anchors, and preserve existing identifiers when editing.
5. Let sitemap <lastmod> move only when the content moves.
6. Ping IndexNow after the deploy is live, not during the build.
7. Add llms.txt if you like, but don't expect it to do the heavy lifting.

robots.txt: why there is no GPTBot group

The robots.txt on this site is three lines plus three comments:

User-agent: *
Allow: /

Sitemap: https://pappfer.hu/sitemap.xml

# Structured summary for LLMs: https://pappfer.hu/llms.txt
# Full site content as plain text: https://pappfer.hu/llms-full.txt
# Machine-readable CV (JSON Resume): https://pappfer.hu/resume.json

A lot of guides tell you to add User-agent: GPTBot, User-agent: ClaudeBot and so on, each with Allow: /. That grants nothing. Under RFC 9309 a crawler looks for the group that names it, and only falls back to the * group if none does. So the explicit allow says what the wildcard already said.

It also sets a trap. Once GPTBot has its own group, it ignores the * group entirely. Six months later someone adds Disallow: /admin/ under *, and every bot with its own group never sees that line. I've kept one group so that a rule added later applies to every crawler following the protocol. robots.txt is not access control: an admin area still needs authentication.

The bots worth knowing by name are the search ones. Three names in the OpenAI documentation matter here: GPTBot collects training data, OAI-SearchBot collects content for ChatGPT search, and ChatGPT-User fetches a page when a user asks for it, where robots.txt "may not apply". Anthropic splits it the same way with ClaudeBot, Claude-SearchBot and Claude-User. If you want to stay out of training but be found in search, that's the split to use: disallow GPTBot and ClaudeBot, leave the search bots alone. That controls these providers' own crawlers; it does not guarantee your content cannot enter training data through another route.

Check your CDN and firewall rules too. Allowing a bot in robots.txt does little if Cloudflare blocks its requests or requires a browser challenge. Inspect requests from verified crawlers in your logs; OpenAI publishes IP ranges to help identify its bots.

Structured data: one person, one @id

The homepage, service pages, glossary and articles use the same JSON-LD identifier for me: https://pappfer.hu/#ferenc-papp. The homepage holds the full Person (job title, location, languages, sameAs links to LinkedIn, GitHub, Stack Overflow and X). The ProfessionalService names that Person as founder, the service pages as provider, the blog articles as author. That explicitly marks these references as the same person. It does not guarantee a search engine or assistant will resolve them into a single entity.

The detail that matters most is that the JSON-LD and the visible text agree. If the schema says "15+ years" and the About section says 12, a crawler has no idea which to believe, and neither does a person.

The FAQ is mirrored in a FAQPage block, and only there, with no microdata on the visible questions: marking the same answers up twice adds weight without adding anything a parser needs. Be honest about why you keep it: Google stopped showing FAQ rich results on May 7, 2026. The visible questions and answers can still appear in ordinary search results. I keep the block because it is the cleanest machine-readable form of question-and-answer content I can give. The documentation linked here does not tell me which assistants use this block when generating answers.

The glossary: anchors that never move

The AI glossary has 50 terms, each with its own anchor (for example, the RAG definition) and each mirrored in a DefinedTermSet:

{
  "@type": "DefinedTerm",
  "@id": "https://pappfer.hu/en/ai-glossary/#rag",
  "name": "RAG (retrieval-augmented generation)",
  "termCode": "rag",
  "inDefinedTermSet": { "@id": "https://pappfer.hu/en/ai-glossary/#termset" }
}

Definitions are what assistants answer "what is X" questions from, so this is the most quotable content on the site. The rule I follow: a term id is permanent. It is the anchor and the schema @id at once, so renaming rag to retrieval-augmented-generation would break every link and citation that already points to it. If a name changes, the display text changes; the id stays.

The same goes for ## headings in articles here: they become anchors, so they stay short and don't get reworded after publishing.

Accurate lastmod: hash the content, not the build

A sitemap whose <lastmod> is the build date on every page is worse than no date. Google says it uses <lastmod> only if it is "consistently and verifiably" accurate, and a site that claims every page changed on every deploy fails that test fast.

Here each URL gets a hash of the content that feeds it (its slice of the translation file, never the rendered HTML), and the date only moves when the hash does. The manifest is committed, so a CSS tweak or the year in the footer leaves the dates alone:

function trackLastmod(urlPath, source) {
  const hash = crypto.createHash('sha1')
    .update(JSON.stringify(source)).digest('hex').slice(0, 16);
  const prev = lastmodStore[urlPath];
  if (prev && prev.hash === hash) return prev.date;
  lastmodStore[urlPath] = { hash, date: BUILD_DATE };
  return BUILD_DATE;
}

Where a page has a JSON-LD dateModified field, it draws its date from the same manifest.

IndexNow: ping after the deploy, not before

IndexNow tells Bing, Yandex, Seznam, Naver and the other participants that a URL changed, and one submission is shared with all of them. Google doesn't take part; sitemaps and crawlable links remain ways to help it discover pages. Even for participating engines, a submission does not guarantee indexing.

The catch is timing. Cloudflare Pages runs the build before the deploy is published, so a ping from the build sends crawlers to the old page. A GitHub Action does it instead: after a push it rebuilds, polls the live sitemap.xml until it matches the one it just built, and only then submits the URLs whose hash changed. A CSS-only deploy submits nothing.

The key is a plain text file at the root (/<key>.txt). It's public by design. I keep mine stable, but it can be rotated: publish the new key file and use the new key for submissions.

llms.txt: useful, but don't oversell it

llms.txt was proposed by Jeremy Howard in September 2024: a markdown file at the root with an H1, a one-paragraph summary and lists of links an agent might want. You can read this site's llms.txt, plus an llms-full.txt with the homepage, service and glossary content in all three languages and the EN/HU articles.

I think it's worth an hour. It costs nothing at runtime, it gives an agent a clean entry point without HTML around it, and writing it forces you to state plainly who you are and what you do. But the OpenAI and Anthropic crawler documentation linked here does not promise their search systems use it, and Google says outright that no new "AI text files" are needed. If someone sells you llms.txt as the thing that gets you into ChatGPT, that's the oversell.

What actually carries the weight

Most of the work isn't in a file. It's in the writing: sentences that make sense when lifted out of the page ("Ferenc Papp is a full-stack developer based in Debrecen, Hungary"), FAQ answers that stand on their own, and headings that say what's underneath. A RAG system often splits documents into chunks and retrieves the ones relevant to a question. A chunk that only makes sense with the paragraph before it is easy to misread on its own. I also build semantic search and RAG systems, and I use that same check when writing my own site.

Need a hand?

If you want to know how assistants describe your company today and what's technically stopping them from reading your site, that's what my AI search optimization work covers. Tell me about your site.