User-agent: Barkrowler/0.9 Disallow: / User-agent: MJ12bot Disallow: / User-agent: barkrowler Disallow: / User-agent: * Disallow: /members/* Disallow: /*/postcomment disallow: /fetch/getsearch User-agent: NPBot Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: fast Disallow: / User-agent: * Disallow: *.axd$ User-agent: * Disallow: /WebResource.axd User-agent: BLEXBot Disallow: / User-agent: coccocbot-web Disallow: / User-agent: bingbot Crawl-delay: 10 User-agent: msnbot Crawl-delay: 10 User-agent: serpstatbot Disallow: / User-agent: grapeshot Disallow: / User-agent: Httrack Disallow: / User-agent: SemrushBot Disallow: / # https://megaindex.com/crawler User-agent: MegaIndex.ru Disallow: / User-agent: megaindex.com Disallow: / User-agent: weborama-fetcher Disallow: / User-agent: weborama-fetcher Disallow: * User-agent: weborama Disallow: / User-agent: Amazonbot Disallow: / User-agent: PetalBot Disallow: / User-agent: Linespider Disallow: / User-agent: CriteoBot/0.1 Disallow: / User-agent: adsbot Disallow: / User-agent: proximic Disallow: / User-agent: dataprovider Disallow: / User-agent: dotbot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: SpiderLing Disallow: / User-agent: SeekportBot Disallow: / User-agent: AwarioRssBot User-agent: AwarioSmartBot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / # Still want to ban the thieving Not For Profit arseholes. User-agent: CCBot Disallow: / # No to Siri # https://support.apple.com/en-gb/119829 User-agent: Applebot-Extended Disallow: / # Test to see if allowing AI bot will increase visitor numbers. User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / # User-agent: Google-Extended # Disallow: / User-agent: PerplexityBot Disallow: / Sitemap: https://www.misterdeveloper.com/sitemaps/mdblogarticles.xml Sitemap: https://www.misterdeveloper.com/sitemaps/mdmain.xml Sitemap: https://www.misterdeveloper.com/sitemaps/mdtaglist.xml