Concept v1.0, updated 2026-07-07

The Topkay crawl format, an open standard for crawls

A portable format for a website crawl: pages, links and scores, built on JSON Graph Format.

There is no portable format for a website crawl. Every crawler keeps its results in its own store, so you cannot move a crawl from one tool to another, or into your own systems, without crawling it again. We think that is worth fixing, so Topkay exports your twin in an open format anyone can read.

The format is JSON Graph Format with a small crawl profile. A crawl is a directed graph: each page is a node, each link an edge. The graph carries metadata about the crawl. Each node carries the page's url, status, depth, word count, title, canonical, indexability, internal PageRank, inlinks and when it was last fetched. Each edge carries its link type, anchor text and rel. It reads cleanly in any graph tool and loads into a data warehouse.

Export a verified site three ways from the dashboard, or by posting to the export endpoint with your project key: the JGF graph for interchange, NDJSON with one page per line for BigQuery or Snowflake, and CSV as a floor. See the digital twin and internal PageRank.

The format is versioned and source available. If it is useful, use it. If you build a crawler, emit it, and a crawl becomes something you own and can move, not something locked in a tool. That is the origin thesis applied to your own data.

Read next

What is ARMX

Agent Readiness and Machine eXperience: the five things every machine does with your site, and the one score that measures them.

What is answerability

How easily an AI answer engine can find and lift the answer from your page.

SEO, AEO and GEO, the difference

Ranking pages, being the answer, being cited by models. Three jobs, not three names.

All topics
Ask Topkay the analyst×
Ask me anything about your site. I read your twin and the Learn corpus, and cite both. Try a skill below.