Website migrations: preserve discovery for AI and search
Plan a website migration with URL mapping, content parity, redirect tests and post-launch checks that preserve useful discovery paths for search.
By Crawlito · 6 min read
In short
- A migration inventory should include important deep pages and their replacement content, not just top navigation.
- Test old URLs, new URLs and the links between them as separate parts of the release.
- A successful deployment is not proof that external search systems have processed the move.
On this page
A website migration should preserve the route from an old reference to the information a reader expects. Build a URL map, verify that replacements contain the intended content, and test the public responses after launch. AI and search discovery can be disrupted by the same practical failures that frustrate visitors: missing guides, incorrect redirects and inaccessible new pages.
Do not treat migration as a homepage test. An assistant, search result, bookmark or partner article may reference a deep URL published years ago. The quality of that arrival experience depends on the mapping and content decisions made before the switch.
Inventory content before choosing destinations
Combine a crawl with CMS exports, sitemaps and useful traffic records. Include documentation, downloadable files, localized pages and campaign routes that still serve a purpose. Keep the original URL, title, content type, owner and proposed outcome in one table.
For each page, choose whether to keep it, move it, merge it into a genuine replacement or retire it. Avoid defaulting every unmapped URL to the homepage. That may reduce visible not-found responses while leaving readers without the answer they came for.
Google's site-move guidance recommends preparing URL mappings, implementing redirects and monitoring the move. It also sets expectations that processing takes time. Use it for Google-specific migration requirements while independently verifying your site's public delivery.
A useful mapping row is more than two URLs:
| Old address | New address | Content decision | Owner | Verification |
|---|---|---|---|---|
| Old installation guide | New installation guide | Preserve supported steps | Documentation team | Compare prerequisites and examples |
| Retired feature page | None | Explain removal where appropriate | Product team | Confirm intended status |
| Duplicate tutorial | Consolidated tutorial | Keep unique troubleshooting notes | Editor | Review merged sections |
This keeps the content obligation visible during implementation.
Test content parity as well as status codes
A redirect can work perfectly and still lead to the wrong information. Pick a few defining facts from each priority page and confirm they survive in the replacement. For a product guide, that may mean supported versions, limitations and configuration examples.
Compare the initial response as well as the browser display. A redesign may move formerly server-delivered text into an interaction or client-only fetch. That is a delivery change even if screenshots look similar. Use raw versus rendered HTML to identify the difference.
Check titles, headings, canonical URLs and public access rules on the new template. A copied staging noindex directive can affect an entire migrated section. The staging noindex guide provides a focused checklist for that failure.
Include representative edge cases: a long article, a paginated archive, an accented or non-Latin slug, a downloadable document and a deleted page. These often exercise routing branches that a short homepage smoke test never reaches.
Keep redirect behavior simple and explainable
For a permanent move, implement an appropriate permanent redirect from the old address to the chosen replacement. Preserve meaningful query parameters when the destination still uses them, and remove obsolete tracking only when doing so does not change content or behavior.
Inspect the complete chain:
curl -sS -L --max-redirs 8 \
-D /tmp/migration-headers.txt \
-o /tmp/migration-body.html \
-w 'final=%{url_effective} status=%{http_code}\n' \
'https://example.com/old-guide/'Use your own URL and review the saved headers. The final successful status alone does not reveal every intermediate hop. A chain through an old subdomain, an HTTP version and a temporary campaign route is harder to maintain than a direct intended destination.
Test without saved cookies or authentication. A redirect that depends on an administrator's session is not a public migration path. Also check that language and region remain appropriate when moving between old and new URL structures.
Use the redirect and status-code guide for response semantics. The migration-specific task is ensuring that each response matches the documented mapping and the real replacement content.
Update the discovery surfaces together
Change internal links to point directly at new destinations instead of relying on redirects forever. Update navigation, related articles, structured data references, sitemap entries and downloadable documents where practical. Search the repository or CMS for the old host and path patterns to find less obvious references.
Keep canonical URLs aligned with the intended public addresses. If a new page still declares its old URL or staging host, the site is sending conflicting information. Review absolute URLs in templates and environment configuration, not just visible links.
Sitemaps should reflect the intended new inventory. Check that generated entries are live and that the file is served from the correct public host. A successful local build does not prove the production sitemap has changed, particularly when a CDN or separate deployment serves it.
Assign each discovery surface an owner. Infrastructure may handle redirects, developers handle metadata, and editors maintain contextual links. A shared checklist prevents the migration from being declared complete when only one team's portion is finished.
Observe the live move and keep a rollback plan
Immediately after launch, request a fixed set of old and new URLs from the public domain. Save the timestamp, response chain, final content and any unexpected access challenge. Test the actual deployment rather than inferring success from the release job.
Monitor not-found responses and redirect loops by path pattern. Some requests may come from obsolete external references you did not inventory. Review whether those addresses deserve a useful replacement instead of treating every error as a reason to add a catch-all redirect.
Keep rollback criteria specific. An entire documentation section returning errors is different from one typo in a title. Decide who can restore the previous routing or content, and preserve the mapping files needed to understand the change.
External processing is a later observation. A search system may continue showing old references while it revisits the site. Do not repeatedly rewrite a sound mapping solely because an old URL appears in a result shortly after launch. First verify that following the reference leads to the correct public content.
Preserve evidence for the next migration
Store the final mapping, exceptions and representative verification results with the project. A future team should be able to tell why a route redirects to a particular guide and which content was deliberately retired.
Review the remaining old-path requests after the initial launch period. They can reveal valuable references, forgotten internal links or automated noise. Use that evidence to prioritize cleanup rather than relying on a blanket assumption that every old route is equally important.
The migration's technical completion criteria should be explicit: intended content is present, old references reach appropriate destinations, new pages are accessible, and discovery surfaces agree. Those are outcomes you can verify. Rankings and AI citations are downstream results that cannot be promised by a redirect map.
Frequently asked questions
Should every removed page redirect to the homepage?
No. Choose a destination only when it meaningfully replaces the old content. Otherwise communicate the removal with an appropriate response and a useful explanation for visitors.
Is a passing deployment enough to confirm the migration?
No. Verify public old and new URLs, their response chains and the actual content after deployment. Build success does not test every edge, cache or routing rule.
Do I need to change URLs during a redesign?
Only when the project requires it. Keeping suitable existing addresses can reduce migration work, but content, access and template checks still matter even when URLs stay the same.
What should be checked first after launch?
Priority commercial and help pages, their old references, the robots and indexing settings, canonical URLs and the sitemap. Use a prepared fixture set so the check is repeatable.
Use a full-site audit to review the new site's technical readiness alongside your migration mapping and public launch checks.
Filed under Access and robots.txt. Topics: migration, http-status, canonical