Skip to main content

Documentation Index

Fetch the complete documentation index at: https://motherfuckingsideproject.com/docs/llms.txt

Use this file to discover all available pages before exploring further.

​
Introduction

What AgentXP is, who it’s for, and why it matters.
AgentXP is a drop-in Next.js SDK that detects AI agents visiting your site and serves them clean, token-efficient markdown instead of raw HTML. Human visitors receive your normal site, unchanged. There is no build step to modify and no measurable impact on page load.

​
Who it’s for

AgentXP is built for Next.js developers who want their sites to work well with AI tools — crawlers, assistants, and agents that read your content programmatically. If you care about how GPT, Claude, Perplexity, or similar tools see your site, AgentXP gives you direct control.

​
What it does

When an AI agent requests one of your pages, AgentXP intercepts the request in middleware, fetches the page HTML internally, runs it through a four-stage transformation pipeline, and returns markdown. The pipeline expands hidden content (accordions, tabs, modals), strips navigation and boilerplate, converts HTML to clean markdown, and applies any token limit you configure. The result is your actual content as markdown — no markup, styles, or scripts. Token reduction: up to 94% fewer tokens than raw HTML.

​
Before and after

Without AgentXP, an AI agent requesting your pricing page receives the full HTML document — scripts, styles, navigation, cookie banners, and all. A typical page is 40,000+ characters. 
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8" />
  <title>Pricing - Acme Corp</title>
  <link rel="stylesheet" href="/_next/static/css/app.css" />
  <script src="/_next/static/chunks/main.js" defer></script>
  <!-- ... 200 more lines of head tags ... -->
</head>
<body>
  <nav class="fixed top-0 ...">
    <div class="flex items-center ...">
      <!-- 80 lines of navigation markup -->
    </div>
  </nav>
  <main>
    <h1 class="text-4xl font-bold ...">Pricing</h1>
    <!-- actual content buried in div soup -->
  </main>
  <div id="intercom-container"><!-- chat widget --></div>
  <div class="cookie-banner"><!-- cookie consent --></div>
  <script>/* analytics, tracking, intercom ... */</script>
</body>
</html>
<!-- ~45,000 characters / ~11,000 tokens -->

​
Key capabilities

  • Automatic agent detection — Identifies 20 known AI crawlers by User-Agent and accepts explicit Accept: text/markdown requests from any agent
  • HTML-to-markdown pipeline — Four-stage transformation: hidden content expansion, content extraction, HTML-to-markdown conversion, cleanup and truncation
  • /llms.txt endpoint — Auto-generated site index following the llmstxt.org specification
  • Agent manifest — /.well-known/agent-experience.json advertises your site’s capabilities in a machine-readable format
  • Content permissions — Declare whether agents may use your content for training, inference, or search indexing
  • Zero browser overhead — The middleware short-circuits immediately for non-agent requests; human visitors are never affected

​
Why not just use Readability?

You could parse HTML with Mozilla Readability directly. Here’s what AgentXP adds on top:
CapabilityReadability aloneAgentXP
Main content extractionYesYes (uses Readability internally)
Agent detection (20+ bots)No — you build itAutomatic, zero config
Hidden content expansion (accordions, tabs, modals)NoYes — expands display:none, hidden, visibility:hidden, max-height:0
/llms.txt site indexNoAuto-generated from config
/.well-known/agent-experience.json manifestNoAuto-generated
Content permission headers (Content-Signal)NoBuilt in
Site navigation preambleNoAuto-extracted or manual config
Token limiting with section-aware truncationNoBuilt in (maxTokens)
Next.js middleware integrationNo — you build itOne line: withAgentXP()
Works with existing middleware (Clerk, next-intl)N/APass as second argument
AgentXP wraps Readability and adds detection, hidden content expansion, permission headers, standards compliance, and Next.js integration.

​
Next steps

Quick Start

Install AgentXP and serve your first markdown response in minutes.

How It Works

Understand agent detection, the transformation pipeline, and special endpoints.