Introduction

What AgentXP is, who it’s for, and why it matters.

AgentXP is a drop-in Next.js SDK that detects AI agents visiting your site and serves them clean, token-efficient markdown instead of raw HTML. Human visitors receive your normal site, unchanged. There is no build step to modify and no measurable impact on page load.

Who it’s for

AgentXP is built for Next.js developers who want their sites to work well with AI tools — crawlers, assistants, and agents that read your content programmatically. If you care about how GPT, Claude, Perplexity, or similar tools see your site, AgentXP gives you direct control.

What it does

When an AI agent requests one of your pages, AgentXP intercepts the request in middleware, fetches the page HTML internally, runs it through a four-stage transformation pipeline, and returns markdown. The pipeline expands hidden content (accordions, tabs, modals), strips navigation and boilerplate, converts HTML to clean markdown, and applies any token limit you configure. The result is your actual content as markdown — no markup, styles, or scripts. Token reduction: up to 94% fewer tokens than raw HTML.

Before and after

Without AgentXP, an AI agent requesting your pricing page receives the full HTML document — scripts, styles, navigation, cookie banners, and all. A typical page is 40,000+ characters.

Without AgentXP (raw HTML)
With AgentXP (markdown)

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8" />
  <title>Pricing - Acme Corp</title>
  <link rel="stylesheet" href="/_next/static/css/app.css" />
  <script src="/_next/static/chunks/main.js" defer></script>
  <!-- ... 200 more lines of head tags ... -->
</head>
<body>
  <nav class="fixed top-0 ...">
    <div class="flex items-center ...">
      <!-- 80 lines of navigation markup -->
    </div>
  </nav>
  <main>
    <h1 class="text-4xl font-bold ...">Pricing</h1>
    <!-- actual content buried in div soup -->
  </main>
  <div id="intercom-container"><!-- chat widget --></div>
  <div class="cookie-banner"><!-- cookie consent --></div>
  <script>/* analytics, tracking, intercom ... */</script>
</body>
</html>
<!-- ~45,000 characters / ~11,000 tokens -->

# Pricing

Simple, transparent pricing. No hidden fees.

## Starter — Free

- 5 team members
- 1,000 events/month
- Community support

## Pro — $49/month

- Unlimited team members
- 100,000 events/month
- Priority support
- API access

## Enterprise — Custom

- Unlimited everything
- Dedicated support
- SSO and audit logs
- Custom integrations

All plans include a 14-day free trial.

---

*Source: [https://acme.dev/pricing](https://acme.dev/pricing)*
<!-- ~500 characters / ~125 tokens — 94% reduction -->

Key capabilities

Automatic agent detection — Identifies 20 known AI crawlers by User-Agent and accepts explicit Accept: text/markdown requests from any agent
HTML-to-markdown pipeline — Four-stage transformation: hidden content expansion, content extraction, HTML-to-markdown conversion, cleanup and truncation
/llms.txt endpoint — Auto-generated site index following the llmstxt.org specification
Agent manifest — /.well-known/agent-experience.json advertises your site’s capabilities in a machine-readable format
Content permissions — Declare whether agents may use your content for training, inference, or search indexing
Zero browser overhead — The middleware short-circuits immediately for non-agent requests; human visitors are never affected

Why not just use Readability?

You could parse HTML with Mozilla Readability directly. Here’s what AgentXP adds on top:

Capability	Readability alone	AgentXP
Main content extraction	Yes	Yes (uses Readability internally)
Agent detection (20+ bots)	No — you build it	Automatic, zero config
Hidden content expansion (accordions, tabs, modals)	No	Yes — expands `display:none`, `hidden`, `visibility:hidden`, `max-height:0`
`/llms.txt` site index	No	Auto-generated from config
`/.well-known/agent-experience.json` manifest	No	Auto-generated
Content permission headers (`Content-Signal`)	No	Built in
Site navigation preamble	No	Auto-extracted or manual config
Token limiting with section-aware truncation	No	Built in (`maxTokens`)
Next.js middleware integration	No — you build it	One line: `withAgentXP()`
Works with existing middleware (Clerk, next-intl)	N/A	Pass as second argument

AgentXP wraps Readability and adds detection, hidden content expansion, permission headers, standards compliance, and Next.js integration.

Getting Started

Configuration

Guides

Reference

Introduction

Introduction

Who it’s for

What it does

Before and after

Key capabilities

Why not just use Readability?

Next steps

Quick Start

How It Works

​Introduction

​Who it’s for

​What it does

​Before and after

​Key capabilities

​Why not just use Readability?

​Next steps

Quick Start

How It Works

Introduction

Who it’s for

What it does

Before and after

Key capabilities

Why not just use Readability?

Next steps