unavailable_after hides a page after an old expiry date
Find an expired unavailable_after directive in HTML or response headers, repair the campaign template, and verify the public page before requesting recrawling.
By Crawlito · 5 min read
In short
- Google supports unavailable_after as a dated search visibility rule.
- A page can return 200 while an old expiration directive remains active.
- Check response headers and HTML before changing a campaign's visible dates.
On this page
A page can disappear from Google after its campaign ends because an old unavailable_after directive remains in the HTML or response headers. If the page should still be available in search, remove or correct the directive at its source and verify the public response. Changing the visible offer date alone does not repair hidden indexing instructions.
Start with the page's intended future. An expired offer might deserve an archived explanation, a replacement page, or removal. Do not automatically make every old promotion searchable just because a report shows lost traffic.
Find the expiration instruction
Fetch the exact public URL using GET and retain both headers and body:
curl -sS -L -D campaign-headers.txt -o campaign.html \
'https://www.example.com/offers/autumn'Search both files for unavailable_after. Inspect the final response and any applicable crawler-specific instructions. An example of an expired rule is:
<meta name="googlebot"
content="unavailable_after: 2026-10-31T23:59:59+03:00">This is a hypothetical instruction, not a suggested date for your site. Google's robots metadata specification documents the rule as a date after which the page should not be shown in search results. It also describes supported date formats and says an invalid date is ignored.
Read the full date and timezone. A content team's local end-of-day plan may differ from the timestamp emitted by a UTC-based publishing job. Write the intended instant and the emitted instant side by side rather than calling both “the end of the month.”
Distinguish the offer's end from the page's end
The expiry of a price does not necessarily mean the page should vanish. An archived campaign may still explain eligibility, support existing customers, or link to current options. Decide whether the page has a useful public purpose after the promotion.
For example, a training provider might keep an event recap and materials at the old campaign address. A retailer might replace an ended offer with an honest notice and a link to the current category. These are editorial decisions about the content readers need.
Do not preserve a misleading expired price just to retain traffic. Update the main explanation and remove obsolete actions. A page that says an offer is available while checkout refuses it has a content problem independent of metadata.
Record the decision in a simple publishing contract:
Offer ends: the advertised discount is no longer available.
Page remains: an accurate archive and link to current options.
Search instruction: no scheduled expiry for the archive.
Owner: the campaign publishing team.Use the deleted page status guide when the decision is actual removal rather than an archive.
Trace the directive to its source
Inspect the CMS fields and the template that generates robots metadata. Some systems keep separate publication, offer, embargo, and search-expiry dates. A migration may map one field to another even though their meanings differ.
Check shared layouts and hosting headers as well. If several campaigns expire on the same date, the rule may live in a template or route pattern rather than each content record. A per-page edit can then appear to work locally while production still adds the header.
Compare one affected page with one campaign that behaves correctly. Keep the content type, environment, and deployment version as similar as possible. A difference tied to a particular template is more useful than a random comparison across unrelated sections.
Search the source repository for the literal directive and the field name that feeds it. If the value is generated, trace its transformation into the final response. Preserve legitimate expiry rules on pages that still require them.
The HTML and HTTP robots comparison helps explain why editing robots.txt is not the right repair for a directive delivered with the document.
Correct the rule and the visible content together
Remove the scheduled search expiry when the approved page is meant to stay available. If it still needs an expiry, emit a valid date that matches the editorial decision. Keep the configuration explicit so a later campaign clone does not silently inherit an old value.
Update the visible offer statement and structured data where relevant. Search instructions and factual content should describe the same state. Do not simply move every expiry far into the future to silence the symptom.
Preview a new campaign cloned from the repaired template. This catches the common recurrence in which old metadata is copied into a fresh page. Check a campaign without an end date as well; an empty CMS field should not become a stale default.
Deploy the fix through the normal content workflow. If a CDN stores the old HTML or headers, refresh the affected response using your hosting process and verify it publicly. Do not assume that saving the CMS record updates every delivery layer immediately.
Keep a short record of the old directive, approved decision, and repaired response. That evidence is enough to explain why the change was made.
Verify accessibility before requesting recrawling
Repeat the same GET request after deployment. Confirm the final URL, content, and robots instructions. Check for a separate noindex instruction or login barrier that could still prevent the page from being usable in search.
Open the page as a logged-out visitor. An accurate archive should explain that the offer ended and provide the intended next action. A successful HTTP response that contains only an error shell is not a completed repair.
Then use the site's Search Console inspection tools as appropriate. Treat a fresh live test and the stored indexed result as different observations. The repair is visible now; a search system's next processing step happens on its own timeline.
Do not repeatedly toggle metadata to chase immediate results. Save the deployment time and wait for evidence that the updated response was fetched. The URL Inspection guide helps distinguish live access from previously processed information.
If the page remains absent after a confirmed later fetch, investigate its content and other indexing signals rather than assuming the expired rule is still the only cause.
Add an expiry review to campaign publishing
Before launching a campaign, review four fields together: visible end date, offer conditions, page retention plan, and search-expiry rule. These should not be scattered across teams without one person checking the final output.
Build a small representative sample of campaign states: upcoming, active, ended but archived, and removed. Verify that each template emits the intended text and directives. This is more useful than checking only a new campaign that has not reached its end date.
For agencies, include the timezone and intended page state in the handoff. “Remove expired metadata” is vague if another team legitimately uses expiry controls elsewhere. A scoped ticket should identify the affected routes and explain which rules must remain.
Also review imported content. A historical timestamp can survive a site migration and trigger an unexpected search change long after the new design goes live. Testing one archived campaign during launch helps expose that inherited state before a larger rollout.
Keep an example of each approved campaign state in the handoff so future editors can recognize the intended behavior.
Frequently asked questions
Can a 200 page still have expired search instructions?
Yes. HTTP success describes delivery, while robots metadata can contain additional instructions about search visibility.
Is unavailable_after an access control?
No. It does not prevent someone with the URL from opening a public page. Use proper access controls for restricted content.
Does removing the directive restore results immediately?
No. Verify the repaired response first, then use dated crawl and indexing evidence to assess subsequent processing.
Check the repaired campaign URL with the free Crawlito check before expanding the review to the rest of your templates.
Filed under Access and robots.txt. Topics: meta-robots, http-status, troubleshooting