How To Stop ChatGPT From Accessing My Website (With 2 Lines Of Code) – Updated To Include GPTBot

Whether you’re for or against AI. ChatGPT is here to stay. If you’re against or just a little cautious did you know that ChatGPT can be blocked from your website? ChatGPT respects robot.txt, a standard file all websites have that dictates what bot is allowed to crawl, scan and index your website. So let’s talk about how to Stop ChatGPT and GPTBot From Accessing My Website.

What Is A.I.

We will keep this brief as most people will have some understanding. AI is a language model that is capable of deep learning to answer questions. 

IE – It’s a big computer and it’s really, really clever.

ChatGPT Limitations 

Until a few months ago ChatGPT had no internet access and a limited knowledge, it knows very little past September 2019

To test this we asked ChatGPT “Can you give me some news headlines from last week?”

It came up with articles from April 2019

1. CDC recommends pausing use of Johnson & Johnson vaccine due to blood clot concerns
2. Derek Chauvin found guilty of murder and manslaughter in the death of George Floyd
3. US imposes new sanctions on Russia over election interference and cyberattacks (This happens quite regularly so who really knows when this happened)
4. India sees record-breaking surge in COVID-19 cases and deaths

That’s now all changed, whereas previously OpenAI allowed their bot to, for want of a better word, index / learn from the internet. 

ChatGPT will now only surf the web if you ask, even then it’s quite selective about what it can and will do.

Here we asked what is Jamie Oliver’s latest cookbook

EG it only knows Jamie Oliver has a new cookbook because we asked it to go to his website. Let’s be honest he releases so many we could just guess new ones out.

“please summarise I.T. Recipes blog post How To Submit A Site To Bing, Yahoo and DuckDuckGo In 5 Easy Steps” 

Bing’s SafeSearch and ChatGPT.

If you didn’t already know, Microsoft paid OpenAI a lot of money, £800 million, to help fund AI research because of this Microsoft is fully integrating AI into its search platform Bing.

With this ChatGPT can ONLY visit websites available through Bing’s SafeSearch, let’s see if we are categorised as ‘Safe’ by Microsoft.

• Preheat your computer or mobile device and connect it to the internet.
• Open a browser and go to Bing.com.
• Look for the icon in the upper right corner of the Bing.com window.
• Click on Settings, and then select More.
• Choose your desired SafeSearch preference: Strict, Moderate, or Off.
• Click Save at the bottom of the menu.
• Serve and enjoy a safer browsing experience on Bing!

Why block ChatGPT, it’s the future

No matter who we have spoken to, AI is very much a marmite topic. You either love it or hate it. You’re either in “its the future” group or believe the terminator storyline is coming true.

No matter what group we are in, we may make the decision that our content, that we worked so hard to create, shouldn’t be available to it.

Arguments For AI – It’s coming no matter what we do so we might as well embrace it.

You can’t stop progress, until AI does something terrible it’s here to stay. So we need to find a way to stay relevant in these changing times. It’s surely better to appear in the top result in an ai search than not appear at all, right?

Arguments Against AI – It’s going to kill search as we know it. 

Microsoft’s version of AI adds sources and links to sites that have provided it with the answer. But this is besides the point, if a user has the answer they don’t need to navigate to our site so they don’t see our ads and can’t click affiliate links which will lower site revenue.

Robots.txt

OpenAI will behave like any other bot and respect the robots protocol. This is a simple text file that sits on our server and dictates who cannot scan our site.

If we don’t say no, bots (or user agents) will assume they are allowed to index our website. You may have heard of GoogleBot or BingBot. These are the automated robots that scan and index our site so that we are available in a traditional search. 

OpenAI uses the same system, meaning we can say no to OpenAI but yes to Google and Bing. Though OpenAI uses the same system they don’t index the internet like a search engine. ChatGPT will only scan a site if you input a URL.

How To Block ChatGPT From Our Site

Updating our robots.txt is easy, to block ChatGPT and its plugins we can add the below

User-agent: ChatGPT-User

Disallow: /

That’s it, 2 lines, nice and easy. But how do you edit your robots.txt?

How To Block GPTBot From Our Site

Just add the below 2 lines of code to your robots.txt file

User-agent: GPTBot

Disallow: /

Tweet

Struggling with the above recipe? Hire a chef to do it for you