Search engine optimisation has a reputation for secret tricks, but most of what makes a site findable is documented by the search engines themselves. This guide sticks to that documentation: Google Search Central, Google's web.dev performance guidance and Bing's own webmaster material. It is written for people who run a site, whether on a hosting plan, a website builder or their own server, and want to know what actually matters. One honest warning first, in Google's own words: "Google doesn't accept payment to crawl a site more frequently, or rank it higher," and it "doesn't guarantee that it will crawl, index, or serve your page" even when you follow its rules (Google Search Central). Anyone promising a guaranteed ranking is selling something Google says does not exist.
How search works: crawl, index, rank
Google describes three stages, and not every page makes it through each one (Google Search Central):
- Crawling: automated programs called crawlers download text, images and video from pages they have found, mostly by following links and reading sitemaps.
- Indexing: Google analyses the page and stores what it learns in its index. Low-quality content, a robots rule that blocks indexing, or a design that hides content can stop a page at this point.
- Serving: when someone searches, Google returns the indexed pages it considers most relevant and useful. Ranking is done programmatically.
Most SEO problems are really one of these three: the page was never found, it was found but not indexed, or it was indexed but something else answers the question better.
Technical basics
HTTPS
Google's page experience guidance asks, among other things, whether "your pages [are] served in a secure fashion," and points to Search Console's HTTPS report (Google Search Central). Most hosts include a free certificate; make sure every HTTP address redirects to its HTTPS version.
robots.txt
A robots.txt file tells crawlers which URLs they may request. Google is explicit that it "is not a mechanism for keeping a web page out of Google"; to keep a page out, use a noindex rule or password protection (Google Search Central). A common mistake is blocking a page in robots.txt and adding noindex: if the crawler cannot fetch the page, it never sees the noindex.
XML sitemaps
A sitemap lists the URLs you want found. Google's limits are 50MB uncompressed or 50,000 URLs per sitemap; larger sites split them and can submit a sitemap index file. URLs should be absolute and canonical, and you can submit through Search Console or reference the sitemap in robots.txt (Google Search Central). Bing supports the same protocol and asks for accurate last-modified dates so it can prioritise recrawling.
Canonical URLs and duplicate content
The same content often lives at several addresses: with and without www, with tracking parameters, or in print versions. Google says duplicate content "is not a violation of our spam policies," but it can waste crawling and split signals (Google SEO Starter Guide). Google lists the ways to state your preferred URL in order of strength: redirects and rel="canonical" annotations are strong signals, sitemap inclusion a weak one (Google Search Central).
Redirects and 404s
When a page moves for good, use a permanent server-side redirect (HTTP 301 or 308) (Google Search Central). Google's site move guide says these "don't cause a loss in PageRank," that Googlebot follows up to 10 hops in a chain, and that you should still redirect straight to the final destination (Google Search Central). A page that is gone should return a real 404 or 410. A "not found" message served with a 200 status shows up in Search Console as a soft 404 (Google Search Central).
Languages and hreflang
If you publish the same page in several languages, hreflang annotations tell Google which versions belong together. Each version must list itself and all the others, links should point both ways, and an x-default value can mark the fallback page. Google notes it does not use hreflang to detect a page's language (Google Search Central).
JavaScript
Google processes JavaScript sites in three phases, crawling, rendering and indexing, and pages can wait in a queue for rendering. Its advice is that "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript" (Google Search Central). If your main content only appears after scripts run, check how Google sees it with the URL Inspection tool.
Site structure and internal links
Crawlers find pages mainly through links, so every important page should be reachable from a normal HTML link on another page of your site, not only from a search box or a script. Group related pages in folders that make sense to a visitor, use short descriptive URLs, and link from general pages to specific ones and back. Google's starter guide notes that keywords in the domain name or URL path "alone have hardly any effect beyond appearing in breadcrumbs," so pick names for people, not for rankings (Google SEO Starter Guide).
Titles and meta descriptions
The title element is usually what becomes the clickable headline in results. Google asks for a unique, descriptive and concise title on every page, and warns against vague titles such as "Home", keyword stuffing and boilerplate titles that differ by only one word (Google Search Central). Snippets are "primarily created from the page content itself," but Google sometimes uses the meta description when it describes the page better (Google Search Central). Write one for each important page as a plain summary of what the reader will find.
Content that helps people
Google's guidance on helpful content asks you to create "people-first content" rather than "search engine-first content made primarily to gain search engine rankings" (Google Search Central). It describes E-E-A-T, meaning experience, expertise, authoritativeness and trustworthiness, as aspects its systems try to identify, and says trust is the most important. It also suggests asking "Who, How, and Why" of each page: who made it, how it was made, and why it exists. Google's starter guide is equally clear that E-E-A-T itself is not a ranking factor (Google SEO Starter Guide). In practice: say who is behind the site, explain how you reached your conclusions, cite sources, keep pages accurate, and answer the question the searcher actually had.
Images and alt text
Put images near the text they relate to and give each one descriptive alt text. Google calls alt text "a short, but descriptive piece of text that explains the relationship between the image and your content" and says it helps search engines understand the image (Google SEO Starter Guide). Alt text is also what screen readers announce, so it is good for visitors as well as search.
Structured data and rich results
Structured data is code, usually JSON-LD, that labels what a page contains so Google can show richer results. Google keeps a gallery of the features it supports, including article, breadcrumb, product and merchant listings, review snippets, recipes, events, job postings, local business, organisation and video (Google Search Central). Be careful with older advice: Google removed HowTo rich results from search, first limited FAQ rich results to well-known government and health sites, and has since stopped showing FAQ rich results altogether and removed their documentation (Google Search Central updates). Markup makes a page eligible, never guaranteed, and it must describe content that is visible on the page.
Page speed and Core Web Vitals
Google recommends that site owners "achieve good Core Web Vitals for success with Search," noting this aligns with what its core ranking systems seek to reward (Google Search Central). The three metrics and their "good" thresholds are:
- Largest Contentful Paint (LCP): the main content should load within 2.5 seconds.
- Interaction to Next Paint (INP): 200 milliseconds or less.
- Cumulative Layout Shift (CLS): 0.1 or less.
web.dev says to measure these at the 75th percentile of page loads, separately for mobile and desktop (web.dev). Google also says not to focus on only one or two aspects of page experience: avoid intrusive interstitials, excessive ads and layouts where the main content is hard to find (Google Search Central). The usual fixes are compressed and correctly sized images, fewer third-party scripts, caching, and reserving space for images and ads so the page does not jump.
Mobile
Google "uses the mobile version of a site's content, crawled with the smartphone agent, for indexing and ranking." It asks that the mobile site carry the same primary content as the desktop site, because content missing on mobile cannot rank (Google Search Central). A responsive design that serves the same HTML to every device avoids the problem.
Local SEO basics
For a business with premises or a service area, the Google Business Profile matters as much as the website. Google says local results are "mainly based on relevance, distance, and popularity," and that "there's no way to request or pay for a better local ranking on Google" (Google Business Profile Help). Keep the profile complete and accurate, including category, hours and contact details, and make sure the same name, address and phone number appear on your website.
Links: earning them, and what counts as spam
Links from other sites still help search engines discover and assess pages. Google defines link spam as "creating links to or from a site primarily for the purpose of manipulating search rankings," and its examples include buying or selling links, exchanging goods or services for links, and excessive link exchanges. Paid and sponsored links are fine when marked with rel="sponsored" or rel="nofollow" (Google spam policies). The same policies cover cloaking, keyword stuffing, scaled content abuse (mass-produced pages with little value), expired domain abuse and site reputation abuse. Durable links come from things people want to cite: original data, useful tools, clear explanations and good customer relationships.
Measuring: Search Console and Bing Webmaster Tools
Google Search Console is free and shows "how often your site appears in Google Search, which search queries show your site, how often searchers click through for those queries, and more" (Search Console Help). Use it to submit sitemaps, check which pages are indexed and why others are not, inspect individual URLs, and watch the Core Web Vitals and HTTPS reports. Bing Webmaster Tools does the same for Bing, accepts sitemaps, and supports IndexNow, a simple ping that tells participating search engines a URL "has been added, updated, or deleted" so they can reflect it sooner (IndexNow).
Common mistakes and myths
- Meta keywords: "Google Search doesn't use the keywords meta tag" (Google SEO Starter Guide).
- Keyword density: there is no target. Repeating words is "tiring for users," and keyword stuffing is against Google's spam policies.
- Word count: Google says length alone "doesn't matter for ranking purposes."
- Top-level domains: a .com is not better than a .org; the ending matters only when targeting one country, and even then is usually a low-impact signal.
- llms.txt files: Google says they are not needed for Google Search and will not help or hurt rankings (Google Search Central updates).
- Blocking with robots.txt to deindex: it does not remove a page; noindex does.
- Launching with the staging block still on: a leftover "disallow everything" rule or sitewide noindex is one of the most common causes of a new site not appearing at all.
Launch checklist
- Every page loads over HTTPS, and HTTP and non-preferred hostnames redirect permanently to one version.
- robots.txt does not block pages you want found, and no staging noindex remains.
- An XML sitemap lists canonical URLs and is submitted in Search Console and Bing Webmaster Tools.
- Each page has a unique title, a meta description and one clear main heading.
- Important pages are linked from navigation or other pages with normal HTML links.
- Images have descriptive alt text and are compressed.
- Core Web Vitals are checked on mobile, against the 2.5 second, 200 millisecond and 0.1 thresholds.
- Old URLs, if any, redirect one-to-one to their new equivalents, and removed pages return 404 or 410.
- Structured data, where used, matches visible content and passes Google's Rich Results Test.
- A local business has a complete Google Business Profile with details matching the site.
After launch, give it time. Check Search Console's indexing reports after a few weeks, fix what it flags, and put your effort into pages that answer real questions better than what already ranks.
