Skip to content

NLWeb and MCP for websites: what they are and who needs them

What Microsoft's NLWeb is, how it builds on schema.org data, what an MCP endpoint on a website means, and whether your site needs either today.

By Crawlito · 6 min read

In short

  • NLWeb is an open-source Microsoft project, announced in May 2025, for adding natural-language question answering to a website, and its README describes the code as proof-of-concept demonstrations.
  • Every NLWeb instance is also an MCP server, with a core ask method that takes a natural-language question and returns answers as JSON using schema.org vocabulary.
  • MCP is an open standard for connecting AI applications to external tools and data; servers can expose tools, resources and prompts over stdio locally or Streamable HTTP remotely.
  • No AI search engine documents querying a website's NLWeb or MCP endpoint when it builds answers, so neither replaces crawlable HTML.
  • The groundwork NLWeb relies on, accurate schema.org markup and feeds, is useful today whether or not you ever run NLWeb.
On this page

NLWeb is an open-source project from Microsoft for adding a natural-language interface to a website: visitors or AI agents ask a question, and the site answers with structured results drawn from its own content. It builds on data many sites already publish, mainly schema.org markup and RSS, and every NLWeb instance also acts as an MCP server, so AI applications that speak the Model Context Protocol can query it. For most websites today, neither is required to be found in AI search. They are worth a look if you run a large catalogue or content library and want agents to query it directly.

This post explains what each piece is, how they fit together, what they do not do, and how to decide whether to invest now.

What NLWeb is

Microsoft introduced NLWeb on 19 May 2025. The project was conceived by R.V. Guha, who created RSS, RDF and schema.org. The code is in the NLWeb GitHub repository under the MIT licence.

The README describes two parts:

  1. A protocol for natural-language interaction with a website. Its core method, ask, takes a question and returns a response in JSON that uses schema.org vocabulary.
  2. An implementation that uses existing structured data, such as lists of products, recipes, attractions or reviews, to build a conversational interface on top of it.

The implementation loads your content into a vector database and uses a language model to answer questions over it. The README lists support for several vector stores, including Qdrant, Milvus, Snowflake, Azure AI Search, Elasticsearch, Postgres and Cloudflare AutoRAG, and model providers including OpenAI, Anthropic, Gemini, DeepSeek, Inception and Hugging Face.

Two statements in the README set expectations. It calls the code proof-of-concept demonstrations of one possible approach and invites better implementations. And it says most production deployments are expected to use their own interface and connect to live databases rather than copy content, to avoid freshness problems. In other words, NLWeb is a reference design, not a plugin you install and forget.

What MCP is, and what an MCP endpoint on a site means

The Model Context Protocol is an open-source standard for connecting AI applications to external systems. Claude, ChatGPT, Visual Studio Code, Cursor and many other clients support it. An MCP server offers capabilities to an MCP client inside an AI application.

According to MCP's architecture overview, servers can expose three core primitives:

PrimitiveWhat it isWebsite example
ToolsFunctions the AI application can callsearch_products(query, max_price)
ResourcesData the application can read for contextA product record, a help article
PromptsReusable templates for interacting with a model"Compare these two plans"

MCP has two transports. Stdio connects processes on the same machine, which is how most local servers run. Streamable HTTP uses HTTP POST, with optional server-sent events, and is what a remote server on the internet uses. MCP recommends OAuth for obtaining authentication tokens on remote servers.

So "an MCP endpoint on a website" means a URL where an AI application can connect and call your tools, for example to search your catalogue or check stock, instead of fetching and parsing your pages. NLWeb provides one such endpoint with an ask tool. You could also build your own MCP server with any of the official SDKs.

How NLWeb depends on schema.org

NLWeb's answers are only as good as the structured data behind them. Microsoft's announcement says it uses semi-structured formats like schema.org and RSS that websites already publish. If your product pages have complete Product markup with name, price, availability and reviews, there is something precise to load and return. If your markup is thin or wrong, the answers will be too.

That is the part worth doing regardless of NLWeb. The same JSON-LD helps search engines understand your pages today. Our guide to structured data for AI search covers what operators actually document about it, and the Product schema guide covers the most common type for commerce.

A quick test of how ready your data is: pick ten important pages and check that each has JSON-LD in the server HTML and that the values match the page.

bash
for u in $(cat urls.txt); do
  printf "%s " "$u"
  curl -s "$u" | grep -c 'application/ld+json'
done

A 0 means the page has no JSON-LD in its raw HTML.

What NLWeb and MCP do not do

It helps to be precise here, because "agent-ready" is easy to oversell.

  • They are not how AI search engines find you. ChatGPT search, Perplexity, Claude, Copilot and Google's AI features document crawlers and fetchers that read your web pages. None of them documents querying a site's NLWeb or MCP endpoint when building answers, at the time of writing. Crawlable HTML, robots.txt access and working pages remain the foundation.
  • They do not replace your pages. People still click through to a URL, and crawlers still need the content in the HTML. Our guide on whether AI crawlers run JavaScript covers that part.
  • An MCP server is an application, not a file. Unlike llms.txt, which is a static text file, an MCP server runs code, takes input from AI clients and needs security work: authentication, rate limits, input validation and monitoring. MCP's own documentation has a security best practices section for this reason.
  • Discovery is not settled. There is no single documented way for an AI application to find out that a given website has an MCP server. Users or developers typically add a server to their client by URL.

Who actually needs it now

Your situationSuggested step
A typical business site, blog or SaaS marketing siteSkip NLWeb and MCP for now; make pages crawlable and markup accurate
A large catalogue, listings or recipe site with good schema.org dataTry NLWeb on a copy of your data to test on-site conversational search
A product with an API that customers already useBuild an MCP server for your API so users can connect it to their AI tools
A publisher wanting agents to query an archivePrototype with NLWeb; watch how agents identify themselves first
A site that cannot get basic crawling right yetFix that first; nothing here works around it

If you expose an MCP server, you are serving software clients you did not write. The planned post on Markdown for agents and content negotiation covers a lighter way to serve agents from your existing URLs, and our guide to AI browser agents in server logs shows how to see which agents already visit.

How to try NLWeb without risk

If you decide to experiment:

  1. Start from your existing data. Export schema.org JSON-LD or an RSS feed for one section of the site.
  2. Run NLWeb locally following the repository's setup guide, with one vector store and one model provider.
  3. Ask the questions your visitors ask. Compare the answers with your pages. Wrong answers usually point to missing or stale markup.
  4. Decide on freshness. If prices or stock change, plan to query live data rather than a copied index, as the README suggests for production.
  5. Only then consider exposing it, with authentication and rate limits, and a clear note in your terms about automated access.

Keep the experiment separate from your public site until you know what it costs to run and maintain.

Frequently asked questions

Is NLWeb an SEO feature?

No. It is a framework for answering natural-language questions over your own data. No search engine documents it as a ranking or citation factor.

Does ChatGPT use MCP to read websites?

ChatGPT is listed among MCP clients that users can connect to servers, but OpenAI does not document using a website's MCP endpoint to build search answers. Its crawlers and fetchers read your pages, as do those of Perplexity, Anthropic and Microsoft.

Do I need schema.org markup for NLWeb?

NLWeb is built around schema.org and RSS data, and its responses use schema.org vocabulary. Without good structured data, it has little precise content to work with.

Is NLWeb production-ready?

The README describes its code as proof-of-concept demonstrations and says CI/CD pipelines are not yet included. Treat it as a reference implementation and plan your own production setup.

What is the difference between llms.txt and an MCP server?

llms.txt is a static Markdown file listing pages for language models to read. An MCP server is a running service that AI applications connect to and call tools on.

Start with what AI search reads today

Before any agent protocol, AI crawlers need to reach your pages and find the content and structured data in the HTML. Run the free check on an important page to see what they receive, with no account needed.

Filed under Strategy and audits. Topics: structured-data, ai-search, strategy, json-ld