Simplifying llms.txt Implementation for WordPress Sites

Key Highlights

• Llms.txt and llms-full.txt is a brand new web standard that is proposed to help Large Language Models (LLMs) like ChatGPT and Claude better understand and navigate your site’s content.

• Unlike robots.txt and sitemap.xml, which are made for search engines, llms.txt is optimized for GenAI reasoning engines.

• Various tools and plugins are available to facilitate the setup of llms.txt on WordPress and non-WordPress sites.

• Using llms.txt on WordPress sites may lead to improved AI visibility, interaction and enhanced SEO opportunities, but this hasn’t been proven yet.

• There are security considerations related to llms.txt, with steps to mitigate potential risks.

• Several real-world examples illustrate the implementation and impact of llms.txt on site traffic and organic keywords.

• The usefulness of llms.txt and llms-full.txt is debatable because it ignores rich sources of content like images and videos.

What is the llms.txt file and why is it important?

Llms.txt is a new web standard by Jeremy Howard, an Australian data scientist and co-founder of Answer.ai. It guides LLMs like ChatGPT. Unlike SEO for search engines, llms.txt helps LLMs skip non-essential elements like navigation, ads, JavaScript. This allows them to access website content easily. By providing info in a suitable format, it enhances AI systems’ understanding of large sites.LLMs depend more on site data now but struggle due to small context windows. Converting complex HTML into plain text for LLMs is tough. Websites cater to humans and LLMs, with the latter benefiting from precise data in one place. This is crucial for tasks like coding environments where quick access to details is vital for LLMs.

Llms.txt is proposed text file standard in markdown format with a clearly defined structure. It similar to robots.txt in that it provides a directive for LLM crawlers. The standard contains two distinct files: /llms.txt and /llms-full.txt. The /llms.txt file gives AI systems a brief overview of your documentation, assisting them in comprehending your website’s layout swiftly. It begins with an H1 project name, followed by a blockquote summary, and then uses H2 headers to organize documentation links. An /llms-full.txt is a comprehensive file containing all your site’s documentation, positioned conveniently in one place. This approach is particularly useful for AI tools that can directly load content from a link.

Is llms.txt the new SEO file you need to know?

Do you really need these new files? I think the jury is out as of this writing. Llms.txt and llms-full.txt DO NOT include information from images and other forms of more complex media that LLMs are voraciously consuming for training purposes. So I think it’s difficult to say if this proposed standard will stand the test of time. I’d personally like to see it implement some form of simplifying other forms of media for LLMs, but that remains for the data scientists. If you’re interested in contributing to the standard, you can find the project on GitHub at https://github.com/AnswerDotAI/llms-txt/.

To implement llms.txt or not to implement, that is the question. It’s up to you and how much time you have to spare. If it’s easy, what have you got to lose. If it’s difficult or time-consuming, maybe wait to see how things shake out. You can watch the directories at llmstxt.site and directory.llmstxt.cloud to see all the platforms and websites that have adopted this proposed standard. As of this writing, the list includes Anthropic, Perplexity, WordLift, Hugging Face, Zapier, and many more. I’ll show you how it is really easy in WordPress below.

How llms.txt differs from robots.txt and sitemap.xml

Like robots.txt and sitemap.xml, llms.txt and llms-full.txt live in a website’s root directory. While /sitemap.xml lists all indexable pages and /robots.txt suggests crawler access, both do not assist in processing the content or help with understanding it. Llms.txt solves AI-related challenges, focusing on overcoming context window limitations, removing non-essential markup and JS scripts, and structuring content for AI processing. Here’s how they compare:

	llms.txt	robots.txt	sitemap.xml
Purpose	Guides AI	Directs crawlers	Guides SEO
File Info	Website content layout	Site areas crawlers can/can’t access, (really “should/shouldn’t” 😉 )	List of site URLs
Content Processing	Enabled	Not enabled	Not Enabled

How do I implement llms.txt in WordPress?

Implementing llms.txt in WordPress has become really easy with the right plugins. These include ‘LLMs.txt Generator’, an ultimate solution that optimally guides AI models like ChatGPT, Gemini, and Claude to discover and interact with your WordPress site. Handling automatic llms.txt file updates, providing intuitive interface for AI optimization settings, these tools significantly simplify llms.txt implementation process, making your website more AI-friendly.

Note: As of today, only one of these plugins appear to be able to create llms-full.txt. Hope the others add that ability soon.

LLMs.txt Plugins Currently Available for in the WordPress Repository

1) Website llms.txt by Ryan Howard

This plugin currently has 4000+ installs and a rating of 5-stars with 2 reviews. The author claims full integration with Yoast SEO and RankMath for sitemap generation and cache management. However, when I installed it, it was not working. It was updated to version 5 very recently. The author has been made aware of the problem. You can also choose how often the llms.txt file is updated.

2) Advanced LLMs.txt Generator by Taha Büyüktaş

This plugin currently has 10+ installs and a rating of 5-stars with 3 reviews. I installed this one on one of my sites and it worked instantly. The only downside is all of the instructions and buttons are in Turkish. I hope he adds the option for other languages. For now, I’m just using Google Translate. The LLMs.txt file came out just fine in English. This plugin has options for adding directives for different AI crawlers, among other features.

3) LLMs.txt Generator by Pedro Ladeira

This plugin currently has 60+ installs and a rating of 5-stars with 1 review. The free version has manual file updates, a basic caching system, and monthly automatic updates. The premium version available starting at $49/year, and is the most fully-featured of this set of WordPress plugins. Real question is, do you actually need it yet?

4) LLMs-Full.txt and LLMs.txt Generator by rankth

This plugin currently has less than 10 installs and a rating of 5-stars with 1 review. I installed this one on one of my sites and had to look for the settings. They are hiding in the “Settings” menu in the WordPress sidebar. It’s very simple and works immediately. This is the plugin I’m going to stick with, for now, simply because it also implements llms-full.txt.

Here’s a simple step-by-step process to add llms.txt to your WordPress site:

1. Install the plugin of your choice: Visit ‘Plugins > Add New’ in your WordPress admin area, search for “LLMs.txt,” and install/activate it.

2. Access the Plugin’s Settings: Go to the sidebar of your WordPress dashboard and click on its settings, or find it under the “Settings” menu.

3. Configure AI Optimization Settings: Customize the settings as you see fit, like basic llms.txt file generation for pages, automatic updates, and caching system if the plugin includes those capabilities.

4. Generate Your llms.txt File: You can create your llms.txt file using the standard template provided by the plugin or customize based on your needs.

5. Review and Save Changes: Review all changes and then save your llms.txt file. Check if it’s correctly placed in the root directory of your website.

6. Update Regularly: Ensure your llms.txt file is updated regularly in line with any changes you make to your website. Some plugins allow you to schedule updates.

How do I add llms.txt to a non-WordPress website?

Implementing llms.txt on a non-WordPress website requires more manual work, but follows the similar principles. First, create an llms.txt file adhering to the established formatting guidelines with a markdown editor or plain text editor. Make sure the file has a clear H1 project name, followed by blockquote summary and subsequent sections with H2 headers. Place this file in the root directory of your website. Ensure it’s accessible at https://yourwebsite.com/llms.txt. Regularly update this file to keep up with any changes in your website content. Remember, non-WordPress sites may not have plugins to automate this process, so manual upkeep is necessary.

Tools and Plugins That Help You Set Up LLMs.txt on Mon-WordPress sites

There are numerous tools and plugins that assist in setting up llms.txt:

• Mintlify: This tool auto-generates both /llms.txt and /llms-full.txt for documentation hosted on its platform.

• llmstxt by dotenv: Mot, the creator of dotenvx, has developed a tool that generates llms.txt from your site’s sitemap.xml.

• llmstxt by Firecrawl: This tool by Firecrawl’s founder, Eric Ciarla, scrapes your website using Firecrawl to generate the llms.txt file for free. You can also get a free API key with 500 credits to generate the llms-full.txt of your site. There are three paid plans.

• WordLift: A free tool, but as of today it shows Hugging Face’s logo and never finishes loading.

What are the Benefits of Using llms.txt on my WordPress Site?

I remain skeptical of the benefits of the proposed llms.txt standard. There is SO MUCH content on a page that it leaves out that I don’t know if it will ever be universally useful.

That said, llms.txt enhances the relationship between your webite and Generative AI models like ChatGPT, Gemini, and Claude. This theoretically translates into better AI visibility and improved interaction with your site content. Additionally, llms.txt purportedly ensures that your website content is accurately indexed, leading to higher SEO rankings and more website traffic. This is not definitive at this point, so please consider if you have the time or resources to implement this proposed standard.

It also remains to be seen how we measure the benefits of this standard. 🤷‍♀️

Enhanced SEO Opportunities

Alongside increasing AI visibility, llms.txt may also enhance your SEO (or GEO/AEO/REO/CEO/VEO or whatever the acronym of the moment is. 😉 By representing your website’s structure, llms.txt and llms-full.txt become an resource for AI-powered search engines. Accurate information representation, unrestricted by context window limitations, means that your whole content (sans non-text media) can be taken into consideration while determining page importance and rank. This may drive more organic traffic to your website and improves ranking in search results. Like I said earlier, it remains to be seen how we test and measure the potential benefits.

What are the Security Considerations for llms.txt?

Despite its numerous benefits, llms.txt and llms-full.txt also bring some security considerations. Any additional plugin adds a security concern to your WordPress site Unauthorized access or modifications could misrepresent your content or cause security vulnerabilities. Risks include prompt injection vulnerabilities and data integrity concerns. With effective website security practices, including regular off-site backups, these risks can be minimized.

The primary security concern around using llms.txt is prompt injection vulnerabilities. Malicious actors could potentially insert harmful instructions into the file, manipulating LLM behavior, causing them to ignore safe instructions, or produce unintended outputs. Additionally, unauthorized changes to your llms.txt file could mislead both SEO performance and user trust, threatening data integrity.

What are some real-world examples of llms.txt implementation?

Implementation of llms.txt is already happening across the Web, with several leading sites adopting it. By integrating llms.txt into their websites, they’re trying to optimize their content for better visibility and navigability by AI engines. However, it’s debatable whether the implementation of the proposed standard has led to an increase in site visibility and improved interaction with AI models.

Case Studies from Leading WordPress Sites

Anthropic’s Documentation Website

Notable examples of current llms.txt integration on WordPress includes Anthropic’s documentation site, and WordLift. WordLift.io implemented the llms.txt standard into its website to enhance AI system accessibility while doc.Anthropic.com utilizes a comprehensive llms-full.txt file to provide a single, extensive markdown file containing all its technical documentation, including some code examples. They each implemented llms.txt on their sites in November 2024, Anthropic on or before 11/15/24, and WordLift on or before 11/21/24.

I looked at Semrush’s data for both sites around November and found that doc.anthropic.com did see an inflection point in organic keywords about this time (see figure below). There was a very noisy period in October and November. However, traffic did not really see such an inflection point, but kept going up and to the right steadily.

WordLift.io Website

Meanwhile, Semrush data indicates that not much happened at WordLift.io during this time period. If anything, the number of organic keywords went down.

What Does This Mean?

Do results mean that the smaller file, llms.txt, doesn’t do all that much, while llms-full.txt provides a boost? Not at all. Anthropic had three announcements in late November, including their Model Context Protocol (MCP), which could have driven the ranking of keywords on the developer’s site. Wordlift only created a “news” page on or before October 6, 2024, according to the Wayback Machine. So, I think the jury is very much out on the benefits of llms.txt and llms-full.txt. These are simply the very early days of a newly proposed standard, and Google search still dominates.

Furthermore, if you look at only the SERP Features graph (I’ll let you do this) where Semrush lumps in AI Overviews, neither website shows much of a change from the trajectory it was already on.

Other WordPress websites using LLMs.txt

Perplexity’s documentation website uses WordPress similarly to Anthropic’s. Implemented on or before November 18, 2024, Perplexity.ai saw no real change around this time, but did have an inflection point in SERP features in September 2024.

Examples of Other Websites using LLMs.txt

As I mentioned earlier you can watch the directories at llmstxt.site and directory.llmstxt.cloud to see all the platforms and websites that have adopted this proposed standard. As of this writing, the non-WordPress list includes, Hugging Face‘s documentation site, and Zapier‘s documentation site.

Another moral of this story is that the sites that are adopting the proposed standard are those that are text rich. It’s typically organization’s documentation sites that have joined the llms.txt club.

Analysis of Impact Post-Implementation

The impact of llms.txt implementation is evident in improved AI interactions, accurate indexing, and enhanced SEO outcomes. Sites have observed that AI systems can efficiently access and process their most relevant information, thus allowing for better comprehension and navigation. Consequently, these websites have bolstered their visibility on AI-driven search platforms, driving more traffic and achieving higher rankings.

What’s the future of llms.txt?

Answer? Who knows?

While a good idea, I think llms.txt and llms-full.txt have some big hurdles to overcome for wide-spread adoption simply because it doesn’t include provisions for high-value, non-text digital assets on a website, like images and videos. Adoption of this protocol seems to be increasing among major AI-related websites, but a brief survey of results finds little to no affect of providing these files on generating traffic. This may simply be a function of our ability to measure traffic from AI-powered search engines, or the level of use of AI-enabled search overall compared to Google’s continuing dominance.

As AI-first documentation becomes a regular industry practice, similar to how SEO became crucial for search visibility, having AI-readable content may also become increasingly important because of limiting factors like a site’s JavaScript, ads and navigation. Tools and best practices developed around the llms.txt proposal will continue to evolve and adapt as more sites adopt this standard.

Trends and Predictions

The growing adoption of llms.txt appears to forecast a shift toward an AI-first approach in documentation. Just as SEO is critical for search visibility, ensuring that AI-readable content exists may become integral for tools and documentation.

Current trend suggests that website owners believe that llms.txt provides a practical and effective solution to help AI systems better understand, utilize web content, especially that pertaining to technical documentation and APIs. I’m going to step out on a limb and predict that the lack of provision for handling images and other non-text media will either be added or render the proposed standard of relatively little use.

The standard is also proposed because of the limited context windows of LLMs. I think it’s reasonable to think this is a temporary situation as more and more GPUs and TPUs come online and models improve at breakneck speed. Context windows are getting larger all the time, and coupled with the voracious nature of LLMs for data, the lack of non-text media coverage may render the proposal moot. Time will tell.

Frequently Asked Questions

What are the primary benefits of implementing llms.txt on my WordPress site?

One of the major benefits of implementing llms.txt on your WordPress site may be improved AI visibility and interaction. Additionally, it may ensure accurate indexing of your website content which can lead to enhanced outcomes in AI-driven search/answer/reasoning engines, driving brand recognition and more revenue from search results.

How often should I update my llms.txt file?

Your llms.txt or llms-full.txt file should be updated regularly to keep up with any changes you make to your website content, just like your website’s sitemap. Regular updates will ensure that AI systems have access to the most accurate and recent information about your site for optimal ingestion.

Are there any specific WordPress plugins recommended for managing llms.txt?

WordPress plugins such as ‘LLMs.txt Generator’ are highly recommended for easing the process of creating, managing, and automatically updating your llms.txt files. These plugins significantly simplify the llms.txt implementation process on WordPress sites, supposedly making them more AI-friendly.