AI & Software Industry Standards

This page outlines emerging and established standards relevant to the intersection of Artificial Intelligence (AI), Large Language Models (LLMs), and web development. These standards aim to improve interoperability, guide AI behavior, and structure information effectively.

/llms.txt

/llms.txt is a newly proposed standard designed to help websites provide Large Language Models (LLMs) and AI agents with a clear, structured, and machine-readable summary of their content.

Purpose & Motivation:

• For LLMs: Web pages can be complex for LLMs to parse due to HTML structure, JavaScript, and navigation elements. /llms.txt offers a concise, context-rich overview, making it easier for AI models to accurately retrieve, summarize, and utilize the site's information.
• For Website Owners: It provides a mechanism to direct AI models towards the most important content, summaries, or documentation. This improves the quality of AI-generated responses related to the site and helps reduce the likelihood of AI misinterpretation or hallucination.

Location:

The file is intended to be placed at the root directory of a website, similar to robots.txt or sitemap.xml. For example: https://example.com/llms.txt.

How it Works (Format & Structure):

/llms.txt uses Markdown for its simplicity and readability for both humans and machines. The basic structure includes:

• Required: An H1 header (#) specifying the site or project name.
• Required: A blockquote (>) containing a brief summary of the website or project.
• Optional: Additional Markdown text (paragraphs, lists, etc., but excluding further headings) to provide more context or instructions.
• Optional: H2 headers (##) to introduce lists of links. These links should point to important pages or documentation (ideally Markdown versions if available), each accompanied by a short description.
• Optional: A specific section titled ## Optional can be used for listing less critical resources.