Plain-language definitions of the words that keep coming up around indexing.
Adding a page to the search engine database. Only indexed pages can appear in results.
A search bot visiting a page and reading its content. The first step before indexing.
An external link from another site to yours. It affects authority, but only once a search engine has crawled and counted it.
A list of your pages for the search engine. Helps bots find new pages faster.
A protocol a site uses to tell search engines (Bing, Yandex and others) about a new or changed page.
An instruction not to add a page to the index. Set via the robots meta tag or the X-Robots-Tag header.
A file at the site root with crawl rules. It can forbid bots from entering certain sections.
Specifies the main version of a page among duplicates. The engine indexes that one.
How many pages a bot will crawl per visit. On large sites it is important to spend it on the right pages.
AI search crawlers: GPTBot, PerplexityBot, ClaudeBot and others. They collect content for AI answers.