Skip to content

Content type and encoding errors that confuse page readers

Check HTML content types, character encoding, and compressed responses when crawlers receive unreadable text or an unexpected document format.

By Crawlito · 5 min read

In short

  • Inspect the body as well as Content-Type when a page is delivered as an unexpected format.
  • Character encoding and compression are different layers and need different diagnostic checks.
  • Verify accented characters, non-Latin text, and structured data after changing response handling.
On this page

When a public URL returns unreadable text, a download, or JSON instead of a page, inspect its content type and encoding before rewriting the article. The intended HTML may be served with unsuitable metadata, transformed incorrectly, or replaced by an error document. Save the headers and body together so you can identify the layer that changed the response.

This problem is especially easy to miss on multilingual sites. A page can look acceptable for plain English test text while damaging accented names, Arabic descriptions, or currency symbols. Use representative content rather than assuming one ASCII heading validates the whole publishing pipeline.

Separate format, characters, and compression

There are three different questions. What kind of representation did the server send? How should its characters be decoded? Was the representation compressed for transfer? Treating all three as “encoding” can send a developer to the wrong configuration.

RFC 9110 defines representation metadata including Content-Type and Content-Encoding. Content-Type describes the media type; Content-Encoding identifies codings applied to the representation. A character set parameter, when applicable, concerns text interpretation rather than HTTP compression.

For an ordinary HTML page, a response might contain:

http
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip

Those headers are an example, not a requirement to enable gzip everywhere. The body and declared metadata must agree. Adding charset=utf-8 cannot repair bytes that were already corrupted in a database export. Adding Content-Encoding: gzip to an uncompressed body creates a different problem.

Make a vocabulary table in the incident ticket: “wrong media type,” “incorrect characters,” or “compression decode failure.” A clear symptom narrows the search to the component most likely to own the change.

Capture the response your client actually receives

Start with a request that allows curl to negotiate and decode supported compression:

bash
curl --silent --show-error --compressed \
  --dump-header document-headers.txt \
  --output document.html \
  'https://www.example.com/ar/services'

Read the header file and inspect the saved body. Is it an HTML document, a JSON error, a sign-in page, or an infrastructure message? A status code of 200 does not answer that question. If the body is not the intended page, first identify who generated it.

Repeat without requesting compression only when you need a comparison. Keep the URL and other conditions identical. If one variation fails and another succeeds, preserve both files. Do not silently use the successful variation as proof that all normal requests work.

The raw HTML guide explains another distinction: readable raw HTML can still omit browser-generated content. First establish that the representation can be decoded, then investigate whether it contains the expected text.

Test characters that reveal the failure

Choose real public strings from the page: an accented company name, an Arabic service heading, a currency symbol, and punctuation used in the article. Write the expected and observed strings side by side. This is much more actionable than “the text looks wrong.”

Trace the affected value through the workflow:

  1. Inspect the source field in the CMS or content file.
  2. Check the exported or generated document before delivery.
  3. Capture the public response body.
  4. Compare the browser's rendered text.
  5. Check any JSON-LD field containing the same value.

If the CMS already contains damaged text, fix that source and identify how it was imported. If the source is correct but generated HTML is wrong, inspect serialization and templating. If generated output is correct but the public response differs, investigate delivery transformations or cached artifacts.

For a hypothetical Arabic service page, the heading may be correct while its JSON-LD description is damaged. Treat those as two fields with possibly different serialization paths. A visible-page screenshot alone would miss the second defect.

Fix the owning layer without hiding errors

For a wrong content type, locate the response rule or object metadata that sets it. Static storage, a reverse proxy, and the application can each have a say. Make the change at the appropriate layer and verify the public response, rather than assuming a local server configuration reaches production unchanged.

For compression failures, inspect where compression is applied and whether an intermediary transforms the body again. Avoid copying an unrelated server snippet into production. Use your actual server's documentation and test a narrow route or staging deployment first.

For character corruption, correct the decoding or serialization boundary and republish the affected content. Preserve a backup of the original source before performing bulk conversions. A global text replacement can damage correct content while appearing to fix one example.

Do not mask an API error by relabeling JSON as HTML. If a page request reaches the API route accidentally, fix routing. Similarly, changing an error response to 200 does not make it a valid article. The desired outcome is the intended representation with accurate metadata.

Review JSON-LD errors when a repair touches structured data. Syntax validity and accurate text both matter, and neither should be inferred from the other.

Add a small multilingual delivery check

After the repair, test one page from each affected template or content pipeline. Include a public HTML page, a downloadable document if it shares the same storage rules, and a structured endpoint such as a feed. They may require different content types even when delivered by the same service.

Record a content marker for each test. The Arabic page should contain its actual heading; the feed should parse as its intended format; the article should contain its canonical title and body. Compare those artifacts after the next routine deployment to confirm that the fix survives regeneration.

Keep the test proportionate. You do not need to invent an exhaustive list of every Unicode character. Use the scripts, symbols, and fields your audience actually encounters. Add a regression example when a new failure exposes a previously untested path.

For content teams, a publishing checklist can simply ask whether the public page preserves the intended names, numbers, and language. For developers, the acceptance record should include headers and raw output. Together those checks connect human meaning to technical delivery.

Frequently asked questions

Does UTF-8 mean the response is compressed?

No. UTF-8 describes character encoding. Compression such as gzip is a separate concern represented through different HTTP metadata.

Can a browser display a page despite incorrect headers?

Sometimes a client may recover or interpret the content differently. That is not a reliable reason to keep contradictory response metadata; verify the intended format directly.

Should I replace every unusual character with plain English text?

No. Preserve the language and names your readers need. Repair the content pipeline instead of removing meaningful characters to conceal the issue.

Can schema validation prove that text is correct?

No. A syntactically valid field can still contain corrupted or inaccurate text. Compare it with the visible content and the authoritative source field.

Continue with the rendering checklist, or run a representative page through the free check before expanding the audit.

Filed under Rendering and content. Topics: html, http-status, rendering