The short answer.
A technical SEO agency works on the parts of a website that visitors never notice and search engines depend on: which pages can be crawled and indexed, what the server sends back, how fast pages respond and how the site is described to machines. It does not write your articles. It removes the reasons good pages fail to appear.
The work falls into six areas. Each is described in Google’s public documentation, so you can check an agency’s advice against the source. Every chapter below quotes it and says what competent work looks like.
Technical work is one part of search, not all of it. On our own SEO services for B2B companies it sits beside content and authority, because a perfectly crawlable site with nothing useful to say still earns no enquiries. Technical work makes sure nothing is wasted: every page you paid for can be found, read and understood.
When you need one.
You need specialist technical help when something structural stands between your pages and the index: a migration, a JavaScript-heavy build, a large or fast-changing site, or a drop in traffic nobody can explain. A small brochure site on a standard platform usually does not.
These are the situations in which an engagement is most likely to be worth commissioning:
- ✓A redesign, platform change or domain move is planned.
- ✓The site is built on a JavaScript framework, with content that appears only after scripts run.
- ✓Search Console shows many pages “not indexed” and nobody knows whether that is intended.
- ✓The site has thousands of URLs from filters, search pages, locations or product variants.
- ✓Traffic fell after a release and nobody has found why.
- ✓Pages are slow on phones and developers need a prioritised list.
The type of site changes the emphasis. A software company with documentation and integration pages tends to have rendering and structure problems; see our guide to SEO for complex B2B SaaS products. A business with many branches has duplication problems; see building multi-location pages without thin copies.
Crawl and index control.
Crawl and index control means deciding, on purpose, which URLs search engines may fetch and which may appear in results. Most problems come from mixing up four tools that do different jobs.
| Control | What it does | Common mistake |
|---|---|---|
| robots.txt | Manages crawling. Google: “A robots.txt file is used primarily to manage crawler traffic to your site.” | Using it to hide a page. Google: it is “not a mechanism for keeping a web page out of Google.” |
| noindex | Keeps a page out of results. Google: “To keep a web page out of Google, block indexing with noindex or password-protect the page.” | Blocking the same page in robots.txt, so the crawler never sees the noindex. |
| Canonical tag | Names the preferred version among duplicates. Google calls rel="canonical" “a strong signal”; sitemap inclusion is “a weak signal”. | Canonicals that point at redirected, blocked or non-existent URLs. |
| Status codes and redirects | Tell crawlers whether a page exists, moved or failed. Google treats a 301 as a strong signal and a 302 as a weak one. | Error pages that return a success code, known as soft 404s. |
One detail catches many sites out. Google says “a page that’s disallowed in robots.txt can still be indexed if linked to from other sites.” A blocked page can sit in the results with no description, the opposite of what the owner wanted.
Crawl budget: mostly a large-site problem
Google defines crawl budget as “the set of URLs that Google can and wants to crawl.” Its guide says who it is written for: “large sites (1 million+ unique pages) with content that changes moderately often (once a week)” and “medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”. It adds a test for everyone else: “If your pages seem to be crawled the same day that they are published, you don’t need to read this guide.”
So if a proposal for a 300-page site leads with crawl-budget optimisation, ask why. For larger sites, the guide lists what wastes crawling: soft 404 pages, long redirect chains and duplicate content. Server errors matter too: Google says they prompt its crawlers to slow down.
What good work looks like: a URL inventory that sorts every page type into “index”, “crawl but do not index” and “do not crawl”, with reasons, and a before-and-after view of Search Console’s Page indexing report.
Rendering and JavaScript.
Rendering is the step where a crawler runs a page’s code to see the finished content. If your important text exists only after JavaScript runs, Google may see it late and many other crawlers may not see it at all.
Google “processes JavaScript web apps in three main phases”: crawling, rendering and indexing. Before rendering, a page waits in a queue: “the page may stay on this queue for a few seconds, but it can take longer than that.” Its own advice is cautious: “Keep in mind that server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript.”
The same document has two rules every audit should test:
- Links must be real links. “Google can only discover your links if they are <a> HTML elements with an href attribute.” Buttons that change the page without a proper link can hide whole sections.
- Missing pages must say they are missing. Single-page applications often show “not found” while the server reports success, which Google calls a soft 404.
Vercel, a hosting company, reported from its own network on 17 December 2024 that “none of the major AI crawlers currently render JavaScript.” The data is nearly two years old, but the safe design is unchanged: send main content, navigation and metadata in the HTML from the server.
What good work looks like: for each page template, a side-by-side of the raw HTML and the rendered page, listing what is missing from the raw version and a fix your developers agree is feasible.
Speed and Core Web Vitals.
Core Web Vitals are three measurements of how a page feels to a real visitor: how fast the main content appears, how quickly the page responds to a tap or click, and how much the layout jumps about. Google describes them as “a set of metrics that measure real-world user experience for loading performance, interactivity, and visual stability of the page.”
| Metric | What it measures | Good | Typical causes of failure |
|---|---|---|---|
| Largest Contentful Paint (LCP) | Loading: when the main content appears | Within 2.5 seconds | Slow server, large images, render-blocking scripts |
| Interaction to Next Paint (INP) | Responsiveness to taps, clicks and key presses | 200 milliseconds or less | Heavy JavaScript, third-party tags |
| Cumulative Layout Shift (CLS) | Visual stability | 0.1 or less | Images without set dimensions, late-loading banners |
Two points matter in an agency’s report. First, the target applies to real visitors, not one lab test: web.dev says “a good threshold to measure is the 75th percentile of page loads, segmented across mobile and desktop devices.” Second, INP “became a stable Core Web Vital metric in 2024”, replacing First Input Delay. A proposal that still talks about FID is out of date.
How much do they matter for ranking?
They matter, in proportion. Google states that “Core Web Vitals are used by our ranking systems”. It also says “Google Search always seeks to show the most relevant content, even if the page experience is sub-par.” Speed will not rescue a weak page; its larger effect is usually on visitors, who leave slow pages before they enquire.
Where the web actually stands
Passing all three is far from universal. The HTTP Archive’s 2025 Web Almanac, using Chrome’s real-user data from July 2025, found that 48% of mobile websites and 56% of desktop websites had good Core Web Vitals. On mobile, loading was the hardest to pass.
What good work looks like: real-user data by page template, the element or script causing each failure, fixes ranked by effort and effect, and a re-measurement some weeks after release.
Structured data.
Structured data is a block of code that labels what is on a page (this is a company, this is an article, these are questions and answers) in a vocabulary machines share.
Google’s introduction says “Google uses structured data that it finds on the web to understand the content of the page.” It sets one rule that is easy to break: “Don’t add structured data about information that is not visible to the user, even if the information is accurate.”
An engagement should do three things here:
- Inventory what markup each template currently outputs, including what plugins add unasked.
- Correct anything that describes content not on the page, such as review ratings or FAQs that visitors cannot see.
- Validate with Google’s Rich Results Test.
Be wary of structured data sold as a route into AI answers. Google’s guidance for its AI features lists “making sure your structured data matches the visible text on the page” as ordinary good practice. See our guide to llms.txt, schema and AI crawler rules.
What good work looks like: a table of templates against schema types, validation results per template, and a named owner for the markup.
Migrations and redesigns.
A migration is any change to URLs, platform, domain or site structure, and it is where technical mistakes are most costly. Missed redirects or a stray noindex tag can take established pages out of the results.
Google’s documentation on site moves sets out the sequence: prepare the new site, “prepare a URL mapping” from current URLs to new ones, start the move by redirecting, and “monitor the traffic” on both old and new URLs. It is realistic about what follows: “Expect temporary fluctuation in site ranking during the move.” On timing: “a small to medium-sized website can take a few weeks for most pages to move, and larger sites take longer.”
Check three of its instructions against any migration plan:
- Permanent redirects. “Use server side permanent redirects if technically possible.”
- No chains. “Avoid chaining redirects.”
- Keep them. “Keep the redirects for as long as possible, generally at least 1 year.”
The agency’s role is specific: crawl the old site before anything changes, build and test the redirect map, check staging for blocked crawling and stray noindex tags, verify the launch within hours, and watch Search Console for weeks afterwards. It should be involved before design is signed off.
The wider project, including trust and conversion, is set out step by step in the website redesign checklist.
AI-crawler access.
AI-crawler access is the newest part of an engagement: checking that the bots behind ChatGPT search, Google’s AI features and similar tools can fetch and read your pages. It is mostly a robots.txt and firewall question.
Google’s AI features use the normal index. To appear as a supporting link in AI Overviews or AI Mode, “a page must be indexed and eligible to be shown in Google Search with a snippet”, and “there are no additional technical requirements.” Among the practices that still apply is “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure”. Security tools can block bots before robots.txt is ever read.
Other companies run their own crawlers. OpenAI documents that “OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features”, separately from GPTBot, which relates to model training. Blocking the wrong one by accident is an easy mistake to make.
Buyer, in an AI assistant: “Which payroll software for UK firms with 50 to 250 staff integrates with Xero and publishes its implementation time?”
What decides whether a vendor can be named: the assistant’s crawler must be allowed through robots.txt and the firewall; the integrations and implementation time must be in the HTML, not loaded by script; and the page must be indexed. Each is a technical SEO check.
An invented question, written to show the pattern. It is not a real query or answer.
An engagement should therefore include a written crawler policy, a check of firewall and CDN bot settings, and a look at server logs to confirm allowed bots get through. Access only makes you eligible; what your pages and other sites say about you decides the rest, which is the subject of our AI search visibility service.
What the deliverables look like.
The deliverable of a technical SEO engagement is implemented change, supported by a small set of documents your developers can act on. If the only output is a long audit, you have bought a diagnosis without treatment.
Google’s own advice on hiring an SEO sets the standard for the first document: “An SEO audit should be about giving you realistic estimates of improvement, and an estimate of the work involved.” An export of automated warnings does neither.
| Deliverable | What it should contain | Sign of a poor one |
|---|---|---|
| Technical audit | Findings by area, each with evidence, likely effect and effort | A raw tool export sorted by “severity” with no business context |
| Prioritised backlog | Developer tickets: what to change, where, and how to test it | Advice such as “improve site speed” with no named cause |
| URL and indexing plan | Every page type marked index, noindex or blocked, with canonical rules | No view on which pages should not be indexed |
| Redirect map (if migrating) | Every old URL mapped to a final destination and tested before launch | “Redirect everything to the homepage” |
| Monitoring and reporting | Indexed pages, crawl errors, real-user Core Web Vitals and organic enquiries, with release dates marked | A health “score” from a tool as the headline number |
The implementation gap
Engagements often fail not from bad advice but from good advice waiting in a queue. The only survey we found on this is old: in September 2021 the software company seoClarity asked 1,195 SEO practitioners from its own audience. Half said they were waiting for technical SEO fixes, 53% named “lack of dev resource” as a hurdle, 37% said fixes normally take weeks and 15% said months. Treat those 2021 figures as a pattern, not a current measure.
So ask any agency “who implements?” Some only advise, some write tickets and test, some make the changes. The proposal must say which. For how the first three months should unfold, see what an SEO agency should deliver in 90 days.
In-house, consultant or agency.
Choose by how much technical work you have and how continuous it is. A steady stream on a large site justifies a person in-house. A one-off diagnosis suits a consultant. A project that also needs hands to implement suits an agency.
| In-house specialist | Independent consultant | Agency | |
|---|---|---|---|
| Best when | Large or fast-changing site with releases every week | You need a diagnosis, a second opinion or migration oversight | You need audit, tickets, implementation support and reporting together |
| Strength | Knows the codebase and sits with developers | Senior attention; no hand-offs | Range of skills; cover when someone is away |
| Weakness | Hard to hire and to keep busy on a small site | Limited capacity; usually advises rather than implements | Quality depends on who is actually assigned |
Many organisations use a mix: outside help for the audit and migration, and a named person inside who owns the backlog afterwards.
If you are considering a team abroad, our guide to hiring an India-based agency for US or UK work covers time zones, data access and contracts.
How to judge a proposal.
Judge a technical SEO proposal on five things: whether it starts from your data, whether it fits your site, who implements, how results are verified, and what it promises.
- 01 · EVIDENCEDoes it start from your Search Console and crawl data?
- 02 · SCOPEAre the areas matched to your site, not a template?
- 03 · HANDSDoes it say who writes, makes and tests each change?
- 04 · PROOFIs every fix checked on the live site after release?
- 05 · HONESTYAre outcomes described as estimates, not promises?
A proposal that fails test 3 produces only a document.
On the fifth test Google is blunt. Its guidance on hiring an SEO says that if someone promises their changes will give you first place in search results, “find someone else.” It also recommends a cautious start: “only grant read access to Search Console (at this stage, don’t grant them write access).”
Red flags specific to technical work
- Crawl-budget work proposed for a site of a few hundred pages.
- Speed reported as a single lab score, or with a metric Google has retired.
- Structured data or an llms.txt file presented as the way into AI answers.
For the wider conversation, including how to check a case study, use our 20 questions to ask an SEO agency. Software companies can also use the proposal scorecard in our SaaS SEO agency buyer’s guide.
The readiness self-check.
Tick each statement that is true today. You get a readiness score out of 100 and the gaps to close before you sign. Nothing you tick is sent anywhere.
Readiness score: 0 / 100
Tick what is true today. If you are not sure, leave it unticked.
The two heaviest items are developer time and a named owner: findings nobody implements change nothing. And if you cannot see enquiries by source, you cannot tell whether the work paid off; our guide to tracking which pages create real opportunities shows how to set that up.
Technical work pays off when it is joined to content worth finding. To have both handled together, see how we approach technical SEO, content and authority as one programme.
Questions buyers ask.
What does a technical SEO agency do?
It makes sure search engines and AI crawlers can fetch, render, understand and index your site, and that pages load quickly. The work covers crawl and index control, rendering, speed, structured data, migrations and AI-crawler access.
What is included in a technical SEO audit?
A site crawl, a review of Search Console data, checks on robots.txt, indexing, canonicals, redirects, rendering, real-user speed and structured data, and a prioritised list of fixes with the effort each needs.
Are Core Web Vitals a ranking factor?
Google states that “Core Web Vitals are used by our ranking systems.” It also says relevance comes first, so treat them as one factor among many.
Should I hire a technical SEO consultant or an agency?
A consultant suits a diagnosis or second opinion. An agency suits a project that needs audit, tickets, implementation support and reporting together. Either way, ask who implements.
How long does a site migration take to settle in Google?
Google says “a small to medium-sized website can take a few weeks for most pages to move, and larger sites take longer”, and to expect temporary fluctuation in ranking.
Sources: Google, robots.txt; Google, canonical URLs; Google, HTTP status codes; Google, crawl budget; Google, JavaScript SEO basics; Google, Core Web Vitals; Google, page experience; web.dev, Web Vitals; Web Almanac 2025; Google, structured data; Google, site moves; Google, AI features; OpenAI, crawlers; Vercel, 17 Dec 2024; Google, do you need an SEO?; seoClarity survey, September 2021. All accessed 12 November 2026.