The open web hasn’t given in to chatbots yet


The BBC is just one of many companies and organizations that block chatbots like ChatGPT from accessing their content.

The UK news organization plans to work with tech companies and regulators to safely develop generative AI, with a focus on maintaining trust in the news industry. The BBC’s three guiding principles include acting in the public interest, prioritizing talent and creativity, and being transparent about AI-generated content.

However, the BBC has blocked web crawlers from OpenAI and Common Crawl from accessing its websites to protect copyrighted material. It joins other news organizations such as CNN and The New York Times.

ChatGPT can’t access nearly half of the top 100 websites

OpenAI re-launched ChatGPT Browsing with Bing in late September, but the chatbot can’t browse 47% of the top 100 websites, including Facebook, Google, and Amazon, according to a recent study by



The study found three reasons for this: being blocked by the site’s Robots.txt, internal limitations of OpenAI, and technical difficulties. In one case, ChatGPT didn’t respect a site’s Robots.txt block. To enable ChatGPT Browse with Bing, users must be ChatGPT Plus subscribers.

Since the beginning of August, more and more websites are blocking ChatGpt and other chatbots from using their content. | Image:

Since the relaunch, OpenAI’s ChatGPT Browse feature only provides 100-word summaries of web content to avoid copyright issues, but these summaries can be unreliable and inaccurate. The AI tool sometimes misses the point or omits important aspects of the content, so you need to check the source to be certain. This makes ChatGPT Browsing less useful than traditional search engines, which return relevant snippets directly from the source without errors or inaccuracies creeping in through rephrasing.

Chatbot as search replacement seems less likely now compared to earlier in the year

Between copyright issues, hallucinations, and websites blocking chatbots, a near-future scenario in which chatbots replace traditional search seems a lot further away than it did earlier this year, when Microsoft CEO Satya Nadella openly stated that he wanted a larger piece of Google’s search market share.

Nadella recently backtracked, telling a court that he had overstated Bing Chat’s potential to take market share from Google. While that statement in Google’s antitrust trial was strategic, Microsoft hasn’t actually taken market share from Google, even though it offers GPT-4 for free. It costs $20 per month if you get it with OpenAI’s ChatGPT.

While chatbots may be able to compete with search engines in the future, they currently fall short in terms of reliability and copyright compliance. Microsoft rushed to release its Bing chatbot in the face of criticism that it was spreading false information, even about elections. It’s prioritizing being faster than Google with its search AI products over safety.


Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top