User-agent: * # 2026-09-28 (founder decision ID-122, A): our AI policy, in the Content Signals # form assistants and Cloudflare read. Search: yes. Answering a person's # question from our pages: yes. Training models on them: no. A stated # preference, not a lock: the per-crawler training blocks below stay (FU-16). # No llms.txt is published (settled by the same decision). Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /admin/ # 2026-09-11 (reach run, technical SEO) — BARE /admin ADDED. `Disallow: /admin/` # matches /admin/ and NOT /admin itself, which returns HTTP 200. Disallow: /admin # 2026-09-11 (reach run, technical SEO) — /analytics ADDED. It is the # admin-key-gated founder dashboard (AnalyticsPage.tsx:411 renders "Enter admin # key ... Locked to staff"), it was in NO Disallow rule, and it returns 200 to # any crawler. Its only protection was accidental: the SPA emits # `rel=canonical -> https://yachtfinder.co/` for unrecognised routes, which # consolidates it onto the homepage. Disallow: /analytics # 2026-06-08 (COPY-FIX P2-11) — /ai-tools is an internal staff AI media # workbench (raw endpoints, per-call $ costs, R2 paths). Keep it out of the # index; and it admin-gates client-side. # # ⚠️ 2026-09-11 — THE ORIGINAL OF THIS COMMENT CLAIMED THE PAGE "also # noindex-tags". IT DOES NOT, AND NO PAGE ON THIS SITE DOES. MEASURED against # live served bytes: `` count is 0 on /ai-tools, /admin/ai, # /analytics, /admin, /my-deals, /numbers and the homepage; 0 in the built # dist/index.html; and 0 in BOTH SSR handlers (api/spa-shell.ts, api/og-yacht.ts). # The distinction is load-bearing: a Disallow stops CRAWLING but does not stop # INDEXING of a URL Google finds via a link, so for an operator surface the # correct instrument is a noindex tag, which this comment asserted we had. # Recorded rather than silently fixed, because adding noindex site-wide is a # founder decision (it would need per-route logic in the shell). Disallow: /ai-tools Disallow: /aitools Disallow: /admin/ai # 2026-05-25 (SEO PR) — explicit disallow on the broker-portal admin URL # (lives at a top-level path rather than /admin/) and on the buyer-brief # admin queue. Both are operator-only surfaces and shouldn't be indexed. Disallow: /buyer-brief/admin Disallow: /portal/admin # 2026-05-31 (SEO-index PR) — signed-in sub-routes inside otherwise-public # marketing surfaces. /family-office and /seller-portal etc. are public # landing pages (marketing meta in PAGE_META); only the onboarding / # authenticated sub-paths should be excluded from index — they require # signed-in session and surface no content for anonymous crawlers. Disallow: /family-office/onboard Disallow: /family-office/portfolio Disallow: /family-office/transactions Disallow: /family-office/calendar Disallow: /family-office/team Disallow: /my-deals # 2026-09-28 (founder decision ID-143) — My fleet: a signed-in account's # private list; noindex too (api/og-page.ts NOINDEX_PATTERNS). Disallow: /my-fleet Disallow: /account Disallow: /settings Disallow: /profile Disallow: /vessel-cockpit/ Disallow: /buyer-profile/ # 2026-06-25 (SEO) — allow the OG/structured-data image renderer through the # /api/ block. og:image + Product JSON-LD `image` point at /api/og-card, and # Google fetches structured-data images respecting robots.txt — a blanket # Disallow: /api/ was dropping the rich-result image on photoless hulls. # Longest-match wins in Google's robots parser, so this Allow overrides the # Disallow below for the og-card path regardless of order. Allow: /api/og-card Disallow: /api/ # 2026-05-26 ip-protect — Worker JSON endpoints. These return aggregated # catalog + Tidemark typical-ask + sanction data that's intended for the # SPA at runtime, not for crawler indexing. Per-page yacht detail (/y/) # is indexed via the SPA shell rendered by api/og-yacht.ts — the JSON # payload underneath should not be crawled directly. Disallow: /ai/ Disallow: /featured/inventory Disallow: /yacht/*/priv Disallow: /charter/inventory Disallow: /b2b/ # v789cc — Static design prototypes left in /public from early UI exploration. # They render outside the SPA shell and have their own , so when Google # indexed them they showed up in SERPs as "YachtFinder — desktop directions" # alongside the real homepage. Keep on disk for reference; don't index. Disallow: /design-v2.html Disallow: /design-v3-desktop.html Disallow: /design-v4.html Disallow: /design-v5.html Disallow: /design-v6.html Disallow: /design-v7.html Disallow: /design-v9.html Disallow: /design-icons.html Disallow: /design-icons-map.html Disallow: /yacht-detail.html Disallow: /yacht-cards.html # 2026-05-21 — Added two more dev artifacts that were leaking into the # Googlebot index. Both files (map-styles.html, mapbox-features.html) are # referenced from code comments only (yachtfinderMapStyle.ts + AtlasMap.tsx) # for historical design-iteration context — not loaded by the SPA at runtime. # Without these entries, Googlebot was free to index them. Disallow: /map-styles.html Disallow: /mapbox-features.html # 2026-05-21 — Block AI model crawlers from training on our content. # Our valuations, comp data, AI prompts, and editorial copy are proprietary. # Allow Google, Bing for search indexing (already covered by * above) but # deny model-trainers that scrape for LLM corpora. # # 2026-09-16 — THE RULE OVERSHOT ITS OWN STATED PURPOSE, and this splits it. # The rationale above is about TRAINING. But three of the agents blocked below # are not trainers — they are LIVE RETRIEVAL agents that fetch a page only # because a human just asked a question about it: # ChatGPT-User a user asked ChatGPT something; it fetches to answer # OAI-SearchBot powers ChatGPT Search results # Perplexity-User a user asked Perplexity; it fetches to answer # Blocking these protects NO training corpus. It only guarantees that when a # buyer asks an assistant "what is the asking price on ARMADA", every source # EXCEPT us can be the answer. # # MEASURED 2026-09-16, and the asymmetry is total: northropandjohnson.com, # fraseryachts.com and boatinternational.com carry ZERO AI-crawler directives # between them. A competitive study found the same across 13 further hosts, # including yachtvalues.com — our closest valuation competitor — whose # robots.txt welcomes ClaudeBot BY NAME. # # And there is nothing here to protect: 28-day search is 318 clicks, of which # 106 are brand and brand typos, leaving roughly 11 non-brand clicks spread # one-per-query across 1,137 rows. We are defending a channel we do not hold # against a channel that is replacing it. # # TRAINING blocks below are DELIBERATELY KEPT — GPTBot, anthropic-ai, # Claude-Web, CCBot, Google-Extended, Bytespider, Applebot-Extended, cohere-ai # and the rest. That is a genuine IP position on proprietary valuations and it # is the founder's to change, not a bug. Only the user-directed retrieval # agents are opened. Reverse by restoring `Disallow: /` on the three below. User-agent: GPTBot Disallow: / # RETRIEVAL, not training — see the note above. User-agent: ChatGPT-User Allow: / # RETRIEVAL, not training — see the note above. User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / # 2026-10-01 (founder decision ID-623, option A): his Sep 28 rule kept, each # assistant written in BY NAME. Anthropic's search crawler and its user-directed # fetcher answer a person's question with a link, like OAI-SearchBot and # ChatGPT-User above; they were already allowed by `User-agent: *`, now # deliberately. ClaudeBot (training) stays blocked above. Reverse with `Disallow: /`. User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / # 2026-09-30 (founder decision ID-566, option A): OPENED. Perplexity documents # PerplexityBot as the crawler that indexes sites for its search answers, not # for model training, so blocking it cost us citations and protected nothing. # Every TRAINING block above and below stays. Reverse with `Disallow: /`. User-agent: PerplexityBot Allow: / # RETRIEVAL, not training — see the GPTBot note above. User-agent: Perplexity-User Allow: / User-agent: Bytespider Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: FacebookBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: cohere-ai Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: omgili Disallow: / User-agent: Timpibot Disallow: / User-agent: YouBot Disallow: / # 2026-06-01 (ip-protect) — Named competitor crawlers explicitly disallowed # from the entire site. robots.txt is not a legal instrument, but combined # with our Terms §05 and §19, an explicit disallow strengthens the argument # that any subsequent scraping was knowing and wilful — relevant to CFAA # (18 U.S.C. § 1030) "exceeds authorized access" analysis and to DTSA # trade-secret misappropriation under 18 U.S.C. § 1836. # Named parties: Yatco (yatco.com / Steve Meyers), Boats Group (yachtworld.com / # boattrader.com / boats.com), One Water Marine (onewatermarine.com). User-agent: YatcoBot Disallow: / User-agent: BoatsGroupBot Disallow: / User-agent: OneWaterBot Disallow: / User-agent: YachtWorldBot Disallow: / User-agent: BoatTraderBot Disallow: / # 2026-06-01 (ip-protect) — Commercial SEO-audit and data-aggregator crawlers # that are commonly used for competitive intelligence and bulk data harvesting. # Disallowed globally; they serve no SEO benefit for yachtfinder.co and their # crawl activity is the primary vector for catalog extraction at scale. # SemrushBot, AhrefsBot: used for competitive site-auditing and backlink # analysis — exact tools a well-resourced competitor would run. # MJ12bot: Majestic SEO, used for link graph analysis and content harvesting. # DataForSeoBot: DataForSEO API crawler, frequently used to bulk-extract SERP # and page content for resale. # SeznamBot: Czech crawler with history of aggressive re-indexing. # Screaming Frog: desktop crawler commonly used for full-site extraction; its # default UA is "Screaming Frog SEO Spider". User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: ia_archiver Disallow: / # 2026-05-31 (SEO-index PR) — /sitemap.xml is a sitemap INDEX that lists the # marketing sub-sitemap (/sitemap-pages.xml, built from MARKETING_PAGES in # api/sitemap.ts) and the yacht sub-sitemap (/sitemap-yachts.xml, built from # the live /featured/inventory catalog). # 2026-08-11 — the two counts that used to be written here, "~80 URLs" and # "~1,074 URLs", measured 180 and 15,700 against production that day: wrong by # 2.2x and 14.6x. Both are derived at request time, so neither has any reason # to be restated in a static file. Name the source, never the number. # Per sitemaps.org spec, crawlers fetch the index first and recurse into # each sub-sitemap independently. Listing the sub-sitemaps explicitly here # (Google/Bing accept multiple Sitemap: lines) helps crawlers that only # parse robots.txt and not the index. Sitemap: https://yachtfinder.co/sitemap.xml Sitemap: https://yachtfinder.co/sitemap-pages.xml Sitemap: https://yachtfinder.co/sitemap-yachts.xml