haystack/docs-website/README.md

Haystack Documentation Website

This directory contains the Docusaurus-powered documentation website for Haystack, an open-source framework for building production-ready applications with Large Language Models (LLMs).

• Website URL: https://docs.haystack.deepset.ai

Table of Contents

• About
• Prerequisites
• Quick Start
• Common tasks
• Project Structure
• Technology Stack
• Available Scripts
• Contributing
• CI/CD and Automation
  • Versioning
• Deployment
• llms.txt for AI tools

About

This documentation site is built with Docusaurus 3 and provides comprehensive guides, tutorials, API references, and best practices for using Haystack. The site supports multiple versions and automated API reference generation.

Prerequisites

• Node.js 18 or higher
• npm (included with Node.js) or Yarn

Quick Start

Note

All commands must be run from the haystack/docs-website directory.

# Clone the repository and navigate to docs-website
git clone https://github.com/deepset-ai/haystack.git
cd haystack/docs-website

# Install dependencies
npm install

# Start the development server
npm start

# The site opens at http://localhost:3000 with live reload

Common tasks

• Edit a page: update files under docs/ or versioned_docs/ and preview at http://localhost:3000
• Add to sidebar: update sidebars.js with your doc ID
• Production check: npm run build && npm run serve
• Prose lint (optional): vale --config .vale.ini "docs/**/*.{md,mdx}"
• Full guidance: see CONTRIBUTING.md

Project Structure

docs-website/
├── docs/                          # Main documentation (guides, tutorials, concepts)
│   ├── _templates/               # Authoring templates (excluded from build)
│   ├── concepts/                 # Core Haystack concepts
│   ├── pipeline-components/      # Component documentation
│   └── ...
├── reference/                     # API reference (auto-generated, do not edit manually)
├── versioned_docs/               # Versioned copies of docs/
├── reference_versioned_docs/     # Versioned copies of reference/
├── src/                          # React components and custom code
│   ├── components/              # Custom React components
│   ├── css/                     # Global styles
│   ├── pages/                   # Custom pages
│   ├── remark/                  # Remark plugins
│   └── theme/                   # Docusaurus theme customizations
├── static/                       # Static assets (images, files)
├── scripts/                      # Build and test scripts
│   ├── generate_requirements.py # Generates Python dependencies
│   ├── setup-dev.sh             # Development environment setup
│   └── test_python_snippets.py  # Tests Python code in docs
├── sidebars.js                   # Navigation for docs/
├── reference-sidebars.js         # Navigation for reference/
├── docusaurus.config.js          # Main Docusaurus configuration
├── versions.json                 # Available docs versions
├── reference_versions.json       # Available API reference versions
└── package.json                  # Node.js dependencies and scripts

Technology Stack

Technology	Version	Purpose
Docusaurus	3.8.1	Static site generator
React	19.0.0	UI framework
MDX	3.0.0	Markdown with JSX
Node.js	≥18.0	Runtime environment
Vale	Latest	Prose linting

Key Docusaurus Plugins:

• @docusaurus/plugin-content-docs — Two separate instances of this plugin run simultaneously:

  1. Main docs instance (via the classic preset): serves docs/ at /docs/
  2. Reference instance (explicit plugin): serves reference/ at /reference/

  Each instance has its own sidebar, versioning config (versions.json vs reference_versions.json), and versioned content folders. This allows the API reference and guides to version independently and maintain separate navigation.

• Custom remark plugin (src/remark/versionedReferenceLinks.js) — Automatically rewrites cross-links between docs and reference to include the correct version prefix. For example, if you're viewing docs version 2.19 and click a link to /reference/some-api, the plugin rewrites it to /reference/2.19/some-api so readers stay in the same version context.

When one might need these plugins:

• Broken cross-links after a release: If links between docs and API reference pages break (404s), the remark plugin may need adjustment—especially if version naming conventions change.
• Version dropdown issues: If the version selector shows wrong versions or doesn't switch correctly between docs/reference, check the dual plugin-content-docs configs in docusaurus.config.js.
• Sidebar mismatches: If API reference navigation breaks separately from main docs, remember they use different sidebar files (sidebars.js vs reference-sidebars.js).

Available Scripts

Important: Run these commands from the haystack/docs-website directory:

Command	Description
npm install	Install all dependencies
npm start	Start development server with live reload (http://localhost:3000)
npm run build	Build production-ready static files to build/
npm run serve	Preview production build locally
npm run clear	Clear Docusaurus cache (use if encountering build issues)
npm run docusaurus	Run Docusaurus CLI commands directly
npm run swizzle	Eject and customize Docusaurus theme components

Contributing

We welcome contributions to improve the documentation! See CONTRIBUTING.md for:

• Writing and style guidelines
• How to author new documentation pages
• Setting up your development environment
• Testing requirements
• Pull request process

For code contributions to Haystack itself, see the main repository's contribution guide.

CI/CD and Automation

This site uses automated workflows for prose linting, API reference sync, and preview deployments. See CONTRIBUTING.md for details.

Versioning

Documentation versions are released alongside Haystack releases and are fully automated through GitHub workflows. Contributors do not need to manually create or manage versions.

Automated Workflows:

• promote_unstable_docs.yml - Automatically triggered during Haystack releases
• minor_version_release.yml - Creates new version directories and updates version configuration

These workflows automatically create versioned documentation snapshots and pull requests during the release process.

Deployment

The documentation site is automatically deployed to https://docs.haystack.deepset.ai when changes are merged to the main branch.

llms.txt for AI tools

This docs site exposes a concatenated view of the documentation for AI tools with an llms.txt file, generated by the docusaurus-plugin-generate-llms-txt plugin.

• What it is: A single, generated text file that concatenates the docs content to make it easier for LLMs and other tools to consume.

• Where to find it (deployed): At the site root https://docs.haystack.deepset.ai/llms.txt.

• How it’s generated:

  • Automatically when you run:

    • npm run start
    • npm run build

  • Manually with:

    npm run generate-llms-txt
    

• Configuration:

  • The plugin is wired in docusaurus.config.js under the plugins array as 'docusaurus-plugin-generate-llms-txt' with outputFile: 'llms.txt'.
  • A local plugin (plugins/txtLoaderPlugin.js) configures Webpack to treat .txt files (including llms.txt) as text assets so they don’t cause build-time parse errors.*