How readable are Hungary's top websites for AI? I measured 500
I checked homepage HTML and AI crawler rules in robots.txt for the first 500 .hu domains in the Tranco ranking. 409 returned usable HTML. Most had no crawler-specific AI rules; blocking affected training crawlers much more often than search crawlers. This measures technical conditions for access, not whether an assistant understands or cites a page.
The raw data and the script are linked at the end, so you can check the numbers or run it again.
The short version
Across the 409 homepages that answered:
- 84.6% have a robots.txt that names none of the AI crawlers tested, or no successfully retrieved robots.txt. The general (
*) rules apply to the crawlers there. - 12.2% (50 sites) block at least one training crawler (GPTBot, ClaudeBot, CCBot, Google-Extended). Only 3.2% (13 sites) block at least one of the three search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
- 9.8% (40 sites) send fewer than 50 words in the first response. Crawlers that don't run JavaScript won't receive text that only JavaScript loads.
- 43.5% use JSON-LD, and 34.2% include an organization-related type in their HTML. Two homepages carry
FAQPagemarkup. - 7.3% (30 sites) publish an llms.txt.
- 41.3% have no detected
<h1>on the homepage, and 11.7% have no detected language declaration in thelangattribute.
How I measured it
The sample comes from the Tranco list. It combines several popularity rankings and gives each version an ID, so the same list can be downloaded later. Rank is not a direct measurement of visitor numbers. List 8P8NV contains 2,921 .hu domains. I took the first 500 and measured them on 9 November 2026, starting at 14:04 UTC.
The script requested each homepage over HTTPS, retrying with www if the first request did not return HTTP 200. For usable HTML responses, it also requested /robots.txt, /llms.txt and /sitemap.xml on the final host, following redirects. The script identified itself as pappfer-research/1.0 and never pretended to be an AI crawler. I evaluated robots.txt against the homepage path (/) using the group-selection principle in RFC 9309: the group that names the crawler applies, otherwise the * group, and within it the longest matching rule wins.
91 domains returned no usable HTML. 46 failed at the network level (for example a nonexistent address or a certificate error), 42 answered with HTTP 403 and 3 with another error code. A 403 can indicate bot protection or another access restriction. It does not establish whether those sites let the real AI crawlers in, so they are left out of the percentages.
The sample follows the Tranco ranking rather than selecting businesses: news portals, shops, government sites, classifieds and adult sites are all in it. I only looked at homepages, so anything deeper in a site (an FAQ page with markup, say) isn't counted. Robots rules were always evaluated against /, even when the homepage redirected to another path. Failed robots.txt requests are treated as an empty rule set, so an allowed result is not always a confirmed permission. HTML markers are detected with simple text patterns, which can miss unusual formatting.
robots.txt: training is blocked, search mostly isn't
AI companies run three kinds of crawler:
- Training crawlers collect data for model training: GPTBot, ClaudeBot and CCBot, while Google offers the Google-Extended control token. Google-Extended is not a separate crawler: it controls both Gemini training and certain uses of search results to ground answers.
- Search crawlers build an assistant's search index: OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot and PerplexityBot. Blocking them restricts how your content is used for search; the exact consequences depend on the provider's policy.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page while someone is asking a question.
How many homepages block each crawler:
- GPTBot: 46 (11.2%)
- CCBot: 43 (10.5%)
- ClaudeBot and Google-Extended: 36 each (8.8%)
- PerplexityBot: 12 (2.9%)
- OAI-SearchBot: 10 (2.4%)
- Claude-SearchBot: 8 (2.0%)
- ChatGPT-User: 13 (3.2%), Claude-User: 7 (1.7%), Perplexity-User: 6 (1.5%)
robots.txt is a weaker control for the user-triggered fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User, because the user initiates those visits. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says all its crawlers honor it. That is why the numbers that follow count only the three search crawlers.
Of the 50 sites that block at least one training crawler, 37 let all three search crawlers in. Those rules allow training and search to be controlled separately, but they do not establish the operator's intent. Seven block all three search crawlers; four of those also block Googlebot. The CSV stores the evaluated permissions, not the original robots.txt files, so it cannot establish the exact directives used.
19 robots.txt files contain a Content-Signal line. Cloudflare's managed robots.txt can add that line alongside rules blocking training crawlers. 15 of the 50 training blockers have it. That is consistent with use of Cloudflare's feature, but does not prove which settings the operator chose.
I think blocking training is a perfectly reasonable choice. Blocking Google-Extended, for example, has no effect on Google Search, and Google says so in its documentation. I'd keep search crawler blocks only if they are intentional. OpenAI, for example, says pages blocking OAI-SearchBot will not appear in ChatGPT search answers, though they can still appear as navigational links.
Homepages without text
40 homepages (9.8%) send fewer than 50 words in the first response. The median for the other 369 is 1,031 words.
Of the 40, 3 redirect to google.com. Of the remaining 37, 27 send at most 13 KB of HTML and 10 send tens or hundreds of kilobytes. Size and word count alone cannot distinguish JavaScript apps from protection pages, login screens or other responses with little text.
Googlebot can run JavaScript, though rendering may be delayed, and other crawlers may not have that capability. In Vercel and MERJ's December 2024 analysis, the OpenAI crawlers tested and ClaudeBot did not execute JavaScript. That is a historical observation, not proof of how every AI crawler works today. My script does not run JavaScript either: it counts text extracted from the downloaded HTML.
If your company's public site is itself a Vue or React app, I'd put server-side rendering or prerendered HTML under at least the public pages. Google benefits from that too.
JSON-LD on 43.5% of homepages
178 homepages (43.5%) include JSON-LD, and only one has a broken block. The most common types are Organization (133) and WebSite (117). 140 sites (34.2%) include an organization-related type (Organization, LocalBusiness, NewsMediaOrganization or a related type). The script collects type names; it does not check whether they describe the site's operator or whether the details are complete and accurate. It does not analyze microdata or RDFa, so missing JSON-LD does not establish that a page has no structured data.
Only two homepages have FAQPage markup, and two have LocalBusiness. That stands out, because an address, opening hours and an FAQ are exactly what assistants answer questions about for local businesses. Then again, this sample is high-traffic sites, not small local firms.
llms.txt: 30 sites
I found an llms.txt on 7.3% of the hosts, including shops, classifieds sites and a university. The script counts successful, non-HTML text responses starting with #; it does not validate their contents. I covered which assistants use it, and for what, in my previous article: I publish one, but I wouldn't count on it to get cited.
A few basics are missing too
None of these are AI settings, but crawlers work from them as well:
- 71.6% have a detected meta description of at least 20 characters;
- 41.3% have no detected
<h1>on the homepage; - 11.7% have no detected language declaration in the
<html>element'slangattribute; - 66% have a sitemap reference in robots.txt or a recognizable sitemap XML response at
/sitemap.xml. Files referenced in robots.txt were not checked separately.
What I'd check on your own site
- Open
https://your-site.com/robots.txt. If OAI-SearchBot, Claude-SearchBot or PerplexityBot appears withDisallow: /, decide whether that's on purpose. - If you use Cloudflare, look at the managed robots.txt and the AI crawler blocking settings. The firewall can reject a crawler even when robots.txt allows it.
- Check how much text your homepage contains without JavaScript (command below).
- Put your company details (name, address, contact, social profiles) in JSON-LD as
OrganizationorLocalBusiness. - Make sure the homepage has a descriptive
<h1>and the<html>element has the correctlangattribute.
For the third check, this follows redirects and gives a rough word count from the downloaded HTML:
curl -fsSL --max-time 30 https://your-site.com/ | perl -0777 -pe 's/<(script|style)\b.*?<\/\1>//gis; s/<[^>]+>/ /g' | wc -wIt's a rough count. If it shows only a few dozen words, view the source in your browser (Ctrl+U) and see whether the text is there or just an empty <div id="app">.
The data
- results.csv: one row per domain, 500 rows.
ok_<crawler>is 1 when the script found no robots.txt rule blocking that crawler on/,named_<crawler>is 1 when robots.txt has a group for it by name.words_no_jsestimates words extracted from HTML;jsonld_typeslists the JSON-LD types on the homepage. The script counts words of at least two characters, excluding content inside scripts, styles,noscript,templateand several other elements. It can still count text hidden by CSS, so this is not an exact count of visible words. - measure.py: the script, Python standard library only. Run against Tranco list 8P8NV it measures the same sample; the results will differ as the sites change.
If you find a mistake in the method, tell me and I'll fix it.
Need a hand?
If you want to know what ChatGPT, Claude or Google actually get from your site, I'll go through your robots.txt, firewall settings, structured data and what reaches the crawlers without JavaScript, and tell you what I'd change. Get in touch.
Sources
- Tranco: list 8P8NV
- RFC 9309: Robots Exclusion Protocol
- OpenAI: overview of its crawlers and how to control them
- Anthropic: how its web crawlers work and how to block them
- Perplexity: its crawlers
- Google: JavaScript SEO basics
- Google: common crawlers, including Google-Extended
- Cloudflare: the Content Signals Policy and managed robots.txt
- Vercel and MERJ: how AI crawlers handle JavaScript