Sean Walsh
Sean is a Director at Intelligency heading up our digital marketing and client services operations. Sean has 15+ years experiencing working both in-house and agency with brands including Lloyds, Alstom, Hitachi, Lufthansa, Viaplay, DFDS Seaways and Mercedes-Benz.
Reddit has made a significant change to its website settings, blocking many search engines, including Bing, from accessing its content. This change took effect on July 1, 2024, when Reddit updated its “robots.txt” file, which guides search engines on how to interact with the site. While Google is still allowed to crawl Reddit, most other search engines have been blocked.
Bing’s Response and Content Disappearance
Shortly after the update, it was noticed that Reddit results were disappearing from Bing’s search index. This was confirmed by Mark Williams-Cook and several media outlets. The reason behind this is Reddit’s use of IP detection to show different versions of the robots.txt file to search engines and humans, making it clear that Bing’s crawlers were indeed blocked. As a result, Bing is no longer indexing new content from Reddit.
Microsoft and Reddit Confirmations
A Microsoft representative confirmed that Bing respects the robots.txt standards and follows the instructions provided by websites. Once Reddit updated their robots.txt file to prohibit crawling, Bing complied and stopped indexing Reddit. Tim Rathschmidt, a spokesperson for Reddit, clarified that this change is unrelated to Reddit’s recent partnership with Google. Reddit has been negotiating with various search engines and has blocked those that could not agree on specific terms, especially regarding the use of Reddit content for AI purposes. More details on this confirmation can be found on Search Engine Land.
Google and Reddit Partnership
In a strategic move, Reddit partnered with Google in a $60 million deal to enhance Google’s search results using Reddit’s user-generated content. This collaboration aims to integrate Reddit’s vast, real-time insights into Google’s search algorithms, improving search relevance and depth. For Reddit, this deal increases visibility and user engagement across its communities, aligning with its broader business goals and future growth plans.
Potential Issues with Content Accuracy
While the partnership with Google has increased Reddit’s visibility in search results, there are concerns about the accuracy and reliability of the information presented. According to Amsive, Reddit’s rapid SEO growth has raised questions about the quality and helpfulness of its user-generated content. Although Reddit provides diverse perspectives, the accuracy of these posts can vary greatly, potentially leading to misinformation or incomplete answers, especially for sensitive topics.
The Amsive article highlights several key issues:
- Varying Quality: Reddit content is user-generated and can range from highly informative to completely inaccurate. This inconsistency poses a risk when search engines present Reddit threads as definitive answers.
- Context and Misinterpretation: Information pulled from Reddit can be taken out of context, leading to misunderstandings. Discussions on Reddit often contain jokes, sarcasm, or incomplete thoughts that can be misinterpreted when isolated.
- Moderation and Verification: Unlike professional content, Reddit posts are not typically vetted by experts. The community-driven moderation system can help flag incorrect information, but it is not foolproof.
Why It Matters
Reddit’s new licensing deal with Google has put it in a strong position, allowing the site to demand more control over how its content is used by other search engines and AI tools. By blocking most other search engines, Reddit is ensuring that only Google drives traffic to its site, which has been substantial. Google is even testing special features in its search results to enhance visibility for Reddit content.
Conclusion
In summary, Reddit’s update to its robots.txt file has effectively stopped Bing and several other search engines from accessing its site, while its partnership with Google continues to thrive. However, the increased visibility on Google also brings potential challenges related to the accuracy and reliability of the user-generated content featured prominently in search results.
