Sitemap index works but new pages are missing
Find why a working sitemap index omits new pages by checking child files, database pagination, release order, and URL counts at partition boundaries.
By Crawlito · 5 min read
In short
- A valid sitemap index can link to incomplete or outdated child files.
- Google limits each standard sitemap to 50,000 URLs or 50 MB uncompressed.
- Stable partitions and complete publication help prevent pages from disappearing between builds.
On this page
When your sitemap index loads but new articles or products are missing, trace one omitted URL through the child sitemap that should contain it. Compare the generated files with the eligible content records and the currently published index. A successful XML response can hide a missing partition, a stale child, or a database pagination error. Google’s guidance for large sitemaps
Identify where the missing URL belongs
Take a new page that is public, canonical, and intended for discovery. Record its content identifier, publication time, canonical URL, and content type. Determine whether your sitemap generator includes only published records and whether the page meets that condition. A product available only in a private sales channel may correctly be absent from a public sitemap even if an administrator can preview it.
Open the sitemap index and list its children. Follow the child for the affected section and search its parsed URL values for the exact canonical address. Differences in trailing slashes, language paths, or the public hostname can make a text search misleading. Compare decoded XML values with the URL the page actually declares. The canonical URL guide explains the wider consistency check.
Find a nearby record that is included. If the missing article and its neighbor use the same template, compare their publication state, identifier, and partition assignment. A page created at the first record of a new partition is a particularly useful clue. It may expose an index generator that never added the next child.
Count URLs within the protocol's limits
Google documents a limit of 50,000 URLs or 50 MB uncompressed for an individual sitemap. Split a larger eligible set into smaller sitemap files and describe those files in an index. Compression helps transfer size but does not change the uncompressed limit. Count actual URL entries and inspect file size before publication. A filename like products-50000.xml does not establish how many entries it contains.
Use comfortably sized partitions that suit the publisher's operational needs. A fictional catalog could divide products by a stable identifier range and use a separate collection for articles. Another system may use immutable publication periods for archival content. Choose a method that lets you account for records and update affected files without renumbering the entire site whenever one item changes.
| Boundary case | Failure to look for |
|---|---|
| First record after a full partition | New child absent from the index |
| Exact multiple of the page size | Extra empty child or skipped final record |
| Record removed during generation | Shifted offsets that omit a later record |
| New item inserted during generation | Duplicate entry across two children |
| Very long URLs | File exceeds the byte limit before the count limit |
The general sitemap guide covers entry selection. This boundary exercise concentrates on whether the publishing process represents that eligible set completely.
Check the data query used for each child
A generator that uses database offsets over a changing list can skip or repeat records. Suppose it reads the newest 1,000 items, then an editor inserts a new item at the top before the next query begins. The second offset now refers to a shifted list. The resulting XML can remain perfectly valid while one record appears twice and another disappears.
Use the system's established method for consistent reads, or page through a stable ordering with an explicit cursor. The cursor needs a dependable tie breaker when several records share the same publication timestamp. Test with two items published at exactly the same time. A query that advances only by timestamp can drop one when it moves to the next batch.
Track the expected eligible count, the generated entry count, and the unique URL count for the same generation snapshot. A comparison between today's database and yesterday's sitemap can legitimately differ. Record the generation cutoff so an engineer can explain which content revision the numbers describe. If a new publication arrives after that cutoff, verify it appears on the next scheduled refresh.
Publish the children before changing their index
Consider an index referencing products-20261114-03.xml that has not finished uploading. A crawler fetching the index during that window can encounter a missing child even though your final deployment later looks complete. Upload complete children first, verify they can be read, and then publish the index that references them using the existing release mechanism.
When old index copies may be cached, keep their referenced child files available for the intended cache transition. Renaming every child and immediately deleting the previous set can break consumers holding yesterday's index. An immutable release set plus a controlled index update is one possible design; a platform with stable child URLs may instead manage versioning and cache refresh internally.
Check the live index and children through the public host after publication. An origin file can be current while the CDN serves an older index or child. Save response dates and cache information when investigating a discrepancy. The CDN freshness guide helps distinguish a stale public representation from a generator that produced the wrong file.
Reconcile the final public set
Download the live index and every child relevant to the affected section. Parse their URLs, deduplicate them for comparison, and report both total and unique counts. Compare the set with the generation snapshot. Identify missing and unexpected entries by stable content ID where possible so a changed slug does not look like an unexplained deletion.
Sample the first and last URL in each changed partition and at least one newly published page. Open them directly and verify that the intended document appears. A matching sitemap count can conceal a set of wrong or repeated destinations if the generator built URLs from the wrong host or path. Counts are an early signal; identity completes the check.
Include a no-change generation in the acceptance test. The same input should produce the same intended URL set. If ordering changes unpredictably, review whether a later consumer or cache relies on that ordering. If an inactive record returns during regeneration, trace the publication filter that included it. Keep the reconciliation output with the normal release logs so the next missing-page report starts from evidence.
Test an item moving between partitions
Change a sample item's classification in a controlled fixture so it moves from one sitemap partition to another. Regenerate the affected children and inspect the combined set. The item should appear at its intended canonical URL under the chosen partitioning rules, with any obsolete entry removed consistently.
Then retry the same update. A replay should preserve the correct set rather than adding another copy or dropping the record. This small exercise checks the transition that a static count of an unchanged catalog cannot reveal.
Frequently asked questions
Does a successful index submission prove every child was fetched?
Inspect the child processing information and fetch them independently. A readable index can still reference a missing or malformed file.
Should every content change regenerate all sitemap files?
That depends on the platform's design. Stable partitions can limit the files requiring updates, provided the final public index accurately describes the complete set.
Can duplicate sitemap entries create another copy of the page?
An entry describes a URL for discovery. Repeated entries usually signal waste or a generator defect, and counting unique addresses helps find the underlying problem.
Inspect a page missing from the set
Use the free page check on an omitted public URL. Confirm its canonical and access state before deciding which sitemap partition should include it.
Filed under Structure and schema. Topics: sitemaps, indexing, deployment