Documentation Index
Fetch the complete documentation index at: https://motherfuckingsideproject.com/docs/llms.txt
Use this file to discover all available pages before exploring further.
Core API
Full API reference for @reaganhsu/agentxp-core and @reaganhsu/agentxp-next exports.
@reaganhsu/agentxp-next
@reaganhsu/agentxp-core
withAgentXP(config?)
Next.js middleware factory. Wrap your middleware export with withAgentXP to intercept agent requests and serve optimized markdown automatically.// middleware.ts
import { withAgentXP } from '@reaganhsu/agentxp-next'
export default withAgentXP()
middleware function compatible with the middleware.ts convention.Config
All fields are optional.interface AgentXPMiddlewareConfig {
transform?: {
contentSelector?: string
excludeSelectors?: string[]
handlers?: Record<string, (el: Element) => string>
maxTokens?: number
}
permissions?: {
aiTrain?: boolean
aiInput?: boolean
search?: boolean
}
baseUrl?: string
llmsTxt?: {
siteName?: string
description?: string
pages?: LlmsTxtPage[]
}
siteHeader?: SiteHeaderConfig
overridesDir?: string
}
transform.contentSelector
Type: stringCSS selector for the main content area. When set, AgentXP extracts only the matching element. Auto-detected from main, article, [role="main"], #content, or .content if omitted.Type: string[]Additional CSS selectors for elements to strip before conversion (e.g., cookie banners, chat widgets).Type: numberMaximum token target for the markdown output. Content is truncated by section when exceeded. Defaults to 8000.permissions.aiTrain
Type: booleanPermit AI providers to use content for model training. Reflected in the Content-Signal response header. Defaults to false.Type: booleanPermit AI agents to use content as inference input. Defaults to true.permissions.search
Type: booleanPermit search indexing. Defaults to true.baseUrl
Type: stringBase URL for resolving relative links in the markdown output. Auto-detected from the incoming request when omitted.llmsTxt.siteName
Type: stringSite name written to the # Heading of /llms.txt. Defaults to "Site".llmsTxt.description
Type: stringSite description written as the > blockquote in /llms.txt.llmsTxt.pages
Type: LlmsTxtPage[]Explicit list of pages to include in /llms.txt. See the LlmsTxtPage type under @reaganhsu/agentxp-core.Type: SiteHeaderConfigManual site navigation header prepended to every markdown response. When omitted, AgentXP auto-extracts navigation from the page’s <header> and <nav> elements.overridesDir
Type: stringPath (relative to project root) to a directory of hand-written .md files that take priority over auto-generated markdown. Requires Node.js runtime — use createMarkdownHandler instead if you are on Edge.
AgentExperience
React component. Add it to your root layout to inject agent-optimized metadata with no visible UI.import { AgentExperience } from '@reaganhsu/agentxp-next'
export default function RootLayout({ children }) {
return (
<html>
<body>
{children}
<AgentExperience siteName="My Site" description="Product docs" />
</body>
</html>
)
}
- A JSON-LD
WebSite structured data block
- A
<link rel="alternate" type="text/markdown" href="/llms.txt" /> tag
- A
<meta name="robots"> tag with noai / noimageai directives when training or inference is disabled
Props
siteName
Type: stringSite name written into the JSON-LD WebSite schema. Optional.description
Type: stringSite description written into the JSON-LD WebSite schema. Optional.aiTrain
Type: booleanWhen false, adds the noai directive to the robots meta tag. Defaults to false.Type: booleanWhen false, adds the noimageai directive to the robots meta tag. Defaults to true.
createMarkdownHandler(config)
Import from @reaganhsu/agentxp-next/route-handler. Creates a Next.js Route Handler GET function that serves markdown with file-based override support. Runs in Node.js runtime (not Edge), so it can read files from disk.// app/api/md/[[...path]]/route.ts
import { createMarkdownHandler } from '@reaganhsu/agentxp-next/route-handler'
export const GET = createMarkdownHandler({
overridesDir: 'content/markdown',
siteHeader: { name: 'My Site', links: [{ text: 'Docs', href: '/docs' }] },
})
Config
interface RouteHandlerConfig {
overridesDir: string
baseUrl?: string
siteHeader?: SiteHeaderConfig
bypassHeader?: string
}
overridesDir
Type: string (required)Directory for hand-written markdown overrides, relative to process.cwd(). A request to /pricing will check <overridesDir>/pricing.md first. The root path maps to index.md.baseUrl
Type: stringBase URL for resolving relative links. Defaults to request.nextUrl.origin.Type: SiteHeaderConfigManual site navigation header prepended to every response. When omitted, AgentXP auto-extracts from the page’s HTML.Type: stringInternal header name used to fetch page HTML without triggering the middleware again. Defaults to "x-agentxp-internal".detectAgent(request)
Detect whether an incoming request is from an AI agent.import { detectAgent } from '@reaganhsu/agentxp-core'
const result = detectAgent(request)
if (result.isAgent) {
console.log(result.agent?.name) // e.g. "ClaudeBot"
}
Parameters
request
Type: Request (required)A standard Web API Request object. Inspects the Accept and User-Agent headers.Return type
interface DetectionResult {
isAgent: boolean
agent?: AgentInfo
}
interface AgentInfo {
name: string
confidence: number
method: 'accept-header' | 'user-agent' | 'http-signature' | 'behavioral'
}
isAgent (boolean, required): true when the request is identified as coming from an AI agent.agent (AgentInfo): Present when isAgent is true.
name (string, required): Matched agent name (e.g., "ClaudeBot", "GPTBot"), or "unknown" when detected via Accept: text/markdown with no matching User-Agent.confidence (number, required): Detection confidence score from 0 to 1. 1.0 for accept-header detection; 0.9 for user-agent pattern match.method (string, required): The signal that triggered detection. One of 'accept-header' | 'user-agent' | 'http-signature' | 'behavioral'.
Full HTML-to-markdown transformation pipeline. Processes a raw HTML string through four stages: expand hidden content, extract main content, convert to markdown, clean and finalize.import { transformForAgent } from '@reaganhsu/agentxp-core'
const result = transformForAgent(html, { baseUrl: 'https://example.com' })
console.log(result.markdown)
console.log(result.tokenEstimate)
Parameters
html
Type: string (required)Raw HTML string to transform.options
Type: TransformOptionsOptional transform configuration.
contentSelector (string): CSS selector for the main content area. Auto-detected when omitted.excludeSelectors (string[]): CSS selectors for elements to remove before conversion.baseUrl (string): Base URL for resolving relative links to absolute URLs in the output.maxTokens (number): Maximum token target. Content is truncated by section when exceeded.headingStyle (‘atx’ | ‘setext’): Markdown heading style. 'atx' uses # prefixes (default); 'setext' uses underlines.title (string): Page title override. Auto-extracted from the HTML when omitted.description (string): Page description written into the output metadata.includeNav (boolean | SiteHeaderConfig): Prepend a site navigation preamble to the output. Pass true to auto-extract from the page’s <header>/<nav>; pass a SiteHeaderConfig object to use a manual override. Omit or pass false for no preamble.
Return type
interface TransformResult {
markdown: string
tokenEstimate: number
metadata: {
title: string
sourceUrl?: string
description?: string
}
processingTimeMs: number
}
markdown (string, required): The agent-optimized markdown output.tokenEstimate (number, required): Estimated token count of the markdown output, calculated as Math.ceil(markdown.length / 4).metadata (object, required): Extracted page metadata.
title (string, required): Page title extracted from the HTML or provided via options.title.sourceUrl (string): Source URL, set from options.baseUrl when provided.description (string): Page description extracted from the HTML or provided via options.description.
processingTimeMs (number, required): Wall-clock time in milliseconds for the full pipeline.
generateLlmsTxt(options)
Generate a standards-compliant llms.txt file following the llmstxt.org specification.import { generateLlmsTxt } from '@reaganhsu/agentxp-core'
const content = generateLlmsTxt({
siteName: 'My App',
description: 'Developer documentation for My App.',
baseUrl: 'https://example.com',
pages: [
{ path: '/docs', title: 'Documentation', section: 'Documentation' },
{ path: '/pricing', title: 'Pricing', section: 'Product' },
],
})
Parameters
options
Type: LlmsTxtOptions (required)
siteName (string, required): Site name written as the top-level # Heading of the file.description (string): Site description written as a > blockquote below the heading.baseUrl (string, required): Base URL prepended to each page path to form absolute links.pages (LlmsTxtPage[], required): List of pages to include. Each entry:
interface LlmsTxtPage {
path: string // URL path, e.g. "/pricing"
title: string // Page title
description?: string
section?: string // Section grouping, e.g. "Documentation"
}
maxPages (number): Maximum number of pages to include. Defaults to 15. Pages beyond this limit are silently omitted.
Return type
Returns a string containing the complete llms.txt content.
defineConfig(config)
Type-safe configuration factory for agentxp.config.ts. Returns the config object unchanged — its only purpose is to provide TypeScript type inference and editor autocompletion.// agentxp.config.ts
import { defineConfig } from '@reaganhsu/agentxp-core'
export default defineConfig({
permissions: {
aiTrain: false,
aiInput: true,
search: true,
},
transform: {
maxTokens: 6000,
},
})
Parameters
config
Type: AgentXPConfig (required)Your AgentXP configuration object. All fields are optional.interface AgentXPConfig {
detection?: {
signals?: ('accept-header' | 'user-agent' | 'http-signature')[]
customAgents?: { name: string; pattern: RegExp }[]
}
transform?: {
contentSelector?: string
excludeSelectors?: string[]
handlers?: Record<string, (el: Element) => string>
maxTokens?: number
}
llmsTxt?: {
autoGenerate?: boolean
include?: string[]
exclude?: string[]
description?: string
maxPages?: number
}
analytics?: {
enabled?: boolean
endpoint?: string
}
permissions?: {
aiTrain?: boolean
aiInput?: boolean
search?: boolean
}
}
Return type
Returns the config typed as AgentXPConfig. Use with a default export in agentxp.config.ts.defineConfig is a zero-overhead identity function — it exists purely for TypeScript autocompletion. When using Edge Runtime middleware, pass your config directly to withAgentXP() rather than loading it from a separate file, since dynamic import() and process.cwd() are not available in the Edge Runtime.
