> ## Documentation Index
> Fetch the complete documentation index at: https://motherfuckingsideproject.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Introduction
> What AgentXP is, who it's for, and why it matters.
# Introduction
> What AgentXP is, who it's for, and why it matters.
AgentXP is a drop-in Next.js SDK that detects AI agents visiting your site and serves them clean, token-efficient markdown instead of raw HTML. Human visitors receive your normal site, unchanged. There is no build step to modify and no measurable impact on page load.
## Who it's for
AgentXP is built for Next.js developers who want their sites to work well with AI tools — crawlers, assistants, and agents that read your content programmatically. If you care about how GPT, Claude, Perplexity, or similar tools see your site, AgentXP gives you direct control.
## What it does
When an AI agent requests one of your pages, AgentXP intercepts the request in middleware, fetches the page HTML internally, runs it through a four-stage transformation pipeline, and returns markdown. The pipeline expands hidden content (accordions, tabs, modals), strips navigation and boilerplate, converts HTML to clean markdown, and applies any token limit you configure.
The result is your actual content as markdown — no markup, styles, or scripts.
**Token reduction: up to 94% fewer tokens than raw HTML.**
## Before and after
Without AgentXP, an AI agent requesting your pricing page receives the full HTML document — scripts, styles, navigation, cookie banners, and all. A typical page is 40,000+ characters.
```html theme={null}
Pricing - Acme Corp
Pricing
```
```markdown theme={null}
# Pricing
Simple, transparent pricing. No hidden fees.
## Starter — Free
- 5 team members
- 1,000 events/month
- Community support
## Pro — $49/month
- Unlimited team members
- 100,000 events/month
- Priority support
- API access
## Enterprise — Custom
- Unlimited everything
- Dedicated support
- SSO and audit logs
- Custom integrations
All plans include a 14-day free trial.
---
*Source: [https://acme.dev/pricing](https://acme.dev/pricing)*
```
## Key capabilities
* **Automatic agent detection** — Identifies 20 known AI crawlers by User-Agent and accepts explicit `Accept: text/markdown` requests from any agent
* **HTML-to-markdown pipeline** — Four-stage transformation: hidden content expansion, content extraction, HTML-to-markdown conversion, cleanup and truncation
* **`/llms.txt` endpoint** — Auto-generated site index following the [llmstxt.org](https://llmstxt.org) specification
* **Agent manifest** — `/.well-known/agent-experience.json` advertises your site's capabilities in a machine-readable format
* **Content permissions** — Declare whether agents may use your content for training, inference, or search indexing
* **Zero browser overhead** — The middleware short-circuits immediately for non-agent requests; human visitors are never affected
## Why not just use Readability?
You could parse HTML with [Mozilla Readability](https://github.com/mozilla/readability) directly. Here's what AgentXP adds on top:
| Capability | Readability alone | AgentXP |
| --------------------------------------------------- | ----------------- | --------------------------------------------------------------------------- |
| Main content extraction | Yes | Yes (uses Readability internally) |
| Agent detection (20+ bots) | No — you build it | Automatic, zero config |
| Hidden content expansion (accordions, tabs, modals) | No | Yes — expands `display:none`, `hidden`, `visibility:hidden`, `max-height:0` |
| `/llms.txt` site index | No | Auto-generated from config |
| `/.well-known/agent-experience.json` manifest | No | Auto-generated |
| Content permission headers (`Content-Signal`) | No | Built in |
| Site navigation preamble | No | Auto-extracted or manual config |
| Token limiting with section-aware truncation | No | Built in (`maxTokens`) |
| Next.js middleware integration | No — you build it | One line: `withAgentXP()` |
| Works with existing middleware (Clerk, next-intl) | N/A | Pass as second argument |
AgentXP wraps Readability and adds detection, hidden content expansion, permission headers, standards compliance, and Next.js integration.
## Next steps
Install AgentXP and serve your first markdown response in minutes.
Understand agent detection, the transformation pipeline, and special endpoints.