User-agent: Yandexbot Disallow: / User-agent: MojeekBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: Linespider Disallow: / User-agent: * Crawl-delay: 30 Disallow: Disallow: /tag/ Allow: /wp-admin/admin-ajax.php Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /search/ Disallow: /wp-login.php Disallow: /activate/ Disallow: /?s= Sitemap: https://www.thewrap.com/sitemap.xml