Technical SEO Basics: Crawling, Indexing and Site Health

Technical SEO is the part of search engine optimization that makes sure search engines can reach your pages, read them, understand them and add them to their index. The best content in your field won't appear in results if its page is blocked from crawling, carries a forgotten noindex tag, or exists under ten different URLs. This guide covers those technical foundations; writing the content itself is covered in writing for search engines without losing the reader, and store-specific issues in SEO for online stores.

How a search engine reaches your page

Every page goes through four stages before it can rank:

1. Discovery and crawling: crawlers find the URL through a link on another page or through your sitemap, then request it from your server.
2. Rendering: the engine runs JavaScript to see the page as a visitor would. This step can be delayed, so keep the main content and links in the HTML your server sends.
3. Indexing: the engine analyses the content and decides whether to store the page. Not every crawled page gets indexed; duplicates and thin pages may be left out.
4. Ranking: for each search, the engine picks the most relevant pages from its index and orders them.

Technical SEO works on the first three stages. "Crawl budget" mainly matters for very large sites or sites that add and change many pages constantly. For a typical company site, the problem is usually indexability, not how often crawlers visit.

Site structure and internal links

• Every important page needs at least one internal link pointing to it. An orphan page that nothing links to may never be discovered.
• Keep key pages within a few clicks of the home page.
• Use real HTML links with an href, not buttons that navigate only through JavaScript, because crawlers follow links.
• Breadcrumbs explain your structure to visitors and search engines alike.

robots.txt vs noindex: the difference people get wrong

The robots.txt file controls crawling; the noindex directive controls indexing. Mixing them up is one of the most common technical mistakes.

• robots.txt is a text file at the root of your site asking crawlers not to visit certain paths. It does not prevent indexing: if other sites link to a blocked page, it can still appear in results, usually without a description.
• noindex sits in a robots meta tag inside the page or in an X-Robots-Tag HTTP header, and asks engines to keep the page out of results. For it to work, the engine has to crawl the page and read it.

That creates a classic trap. If you block a page in robots.txt and add noindex at the same time, the engine never sees the noindex, and the page may stay indexed. To remove a page from results, allow crawling, add noindex, and wait until it drops out of the index.

User-agent: *
Disallow: /cart/
Disallow: /search/

<meta name="robots" content="noindex">

You can also add a Sitemap line to robots.txt with the full address of your sitemap. And remember that the file is public: listing a private path there points people straight to it.

XML sitemaps

An XML sitemap lists the URLs you want indexed and helps engines discover them, especially on new sites, large sites, or sites with weak internal linking. The rules:

• List canonical pages only: URLs that return 200, are indexable and point to themselves as canonical. Leave out redirected, deleted and noindexed URLs.
• Keep lastmod honest, changing it only when the content really changes.
• One file holds up to 50,000 URLs or 50 MB uncompressed; beyond that, split it into several files tied together by a sitemap index.
• On a bilingual site, include both the Arabic and the English URLs.
• Submit the sitemap in Google Search Console and reference it in robots.txt.

Most content management systems generate a sitemap automatically, but open yours and look. It often includes tag archives or attachment pages with no value.

Canonical tags and duplicate content

One page can be reachable at many URLs: over HTTP and HTTPS, with and without www, with and without a trailing slash, and with utm tracking or sort parameters. Search engines see these as duplicates, split signals between them, or pick a version you didn't intend.

Handle this at two levels:

1. Redirect the versions nobody needs, such as HTTP or the www variant you didn't choose. The setup is covered in choosing hosting, a domain and an SSL certificate.
2. Use a canonical tag for versions that must stay reachable, such as a page with sort parameters. Give every page a canonical pointing to its preferred full URL, even when that is its own address (a self-referencing canonical).

A canonical is a hint, and engines follow it more readily when other signals agree: internal links point to the preferred version and the sitemap lists only that one. Two common mistakes are pointing every paginated page (page 2, 3 and so on) to page 1, and pointing the Arabic version to the English one. Language versions are not duplicates, so each gets a canonical to itself.

hreflang: connecting language versions

The hreflang annotation tells search engines that a page has versions in other languages or for other regions, so each searcher is shown the right one. The essentials:

• Each version lists every version, including itself, using full URLs.
• Annotations must be reciprocal. If the Arabic page points to the English one but not the other way round, they may be ignored.
• Language codes follow ISO 639-1, such as ar and en. Add a region, such as ar-AE or ar-EG, only when the content genuinely differs by country.
• An x-default value marks the fallback page for users whose language matches none of the versions.
• Annotations can go in the page head, in HTTP headers or in the XML sitemap.

Building a site in both languages, from URLs to text direction and fonts, is covered in building a bilingual Arabic–English website.

Structured data

Structured data describes your content in a machine-readable format using the Schema.org vocabulary, so the engine understands that a page is a product with a price and reviews, an article with an author and date, or a business with an address and opening hours. Google recommends the JSON-LD format.

Common types on business sites include Organization for the company, logo and profiles; LocalBusiness for branches with addresses and hours; BreadcrumbList for navigation paths; Article for articles; and Product for products. The rules that keep you safe:

• Markup must match what visitors see. Never add ratings or prices that aren't on the page.
• Structured data makes a page eligible for rich results; it doesn't guarantee them.
• Rich result types change, and some get restricted over time, so check Google's current documentation before relying on one.
• Test pages with Google's Rich Results Test and follow the enhancement reports in Search Console.

Status codes and redirects

Every page request returns an HTTP status code that tells browsers and crawlers what happened:

Good redirect habits:

• Send each old URL to the closest matching new page, not to the home page in bulk; mass redirects to the home page can be treated as missing pages.
• Keep redirects to a single hop. Chains slow visitors down and waste crawling, and loops break the page.
• Update internal links to point straight at the final URL.
• Keep redirects in place long after a move, commonly at least a year, and longer if old links still bring visits.
• Watch for soft 404s: pages that say "not found" or are nearly empty but still return 200.

Before a redesign or any change to your URL structure, prepare a redirect map: a sheet listing every important old URL next to its new equivalent.

Speed and mobile-first indexing

Google uses mobile-first indexing, meaning the mobile version of a page is what it primarily indexes and ranks. In practice, your mobile pages need the same content, headings, structured data and meta tags as the desktop version. Content hidden from mobile visitors to simplify the design may never be seen by Google at all. The guide to responsive web design covers the details.

Speed is measured through the Core Web Vitals: loading of the largest visible element (LCP), responsiveness to interaction (INP) and layout stability during loading (CLS). They are part of page experience, which is one signal among many; relevance to the search still matters most. But a slow page loses visitors before it loses rankings, and practical fixes are in the guide to website speed.

Google Search Console: your technical dashboard

Search Console is Google's free tool and the first thing to set up after launch. Verify the whole domain through a DNS TXT record so the property covers every protocol and subdomain. What to watch:

• Page indexing report: which pages are indexed, which aren't, and why, for example blocked by robots.txt, excluded by noindex, duplicate without a user-selected canonical, or crawled but not currently indexed. Interface labels may change, but the logic stays the same.
• URL Inspection: shows how Google sees a specific page, whether it can be indexed and which canonical Google selected, and lets you request indexing after a fix.
• Sitemaps: submit yours and see how many URLs were read.
• Core Web Vitals and enhancement reports: for speed and structured data.
• Manual actions and security issues: act immediately if anything appears here.

Bing offers a similar tool, Bing Webmaster Tools, which is worth setting up as well.

Practical checklist: a quick technical audit

• robots.txt blocks no important pages and no CSS or JavaScript files needed to render them.
• No noindex on pages that should rank, and no page that is both noindexed and blocked in robots.txt.
• The sitemap contains canonical pages only and is submitted in Search Console.
• Every page has a canonical pointing to its preferred URL, and language versions are linked with reciprocal hreflang.
• Every variant of the site redirects in one hop to a single HTTPS version.
• No broken internal links, no orphan pages, no redirect chains.
• Main content is in the HTML and fully present on mobile.
• Structured data matches visible content and passes testing.
• The page indexing report is reviewed monthly and after every major update.

For the complete build process these foundations sit on, see building your business website.