User-agent: * Disallow: /library/ User-agent: Mediapartners-Google* Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: Urlpulse Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: wget Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: NPBot Disallow: / User-agent: WebReaper Disallow: / User-agent: EdisterBot Disallow: / User-agent: swebot Disallow: / User-agent: SWEBot/1.0 Disallow: / User-agent: ccbot Disallow: / User-agent: turnitinbot Disallow: / User-agent: discobot Disallow: / User-agent: linkdex Disallow: / User-agent: linkdex.com Disallow: / User-agent: linkdex.com/v2.0 Disallow: / User-agent: PiplBot Disallow: / User-agent: oBot Disallow: / User-agent: ICCrawler - iCjobs Disallow: / User-agent: Yandex Disallow: / User-agent: OpenindexSpider Disallow: / User-agent: Ezooms Disallow: / User-agent: Ezooms/1.0 Disallow: / User-agent: Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com) Disallow: / User-agent: Yeti Disallow: / User-agent: Yeti/1.0 Disallow: / User-agent: Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) Disallow: / User-agent: findlinks Disallow: / User-agent: findlinks/2.0.2 Disallow: / User-agent: findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/) Disallow: / User-agent: brandwatch Disallow: / User-agent: magpie Disallow: / User-agent: magpie-crawler Disallow: / User-agent: magpie-crawler/1.1 Disallow: / User-agent: magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) Disallow: / User-agent: Unister Disallow: / User-agent: unister Disallow: / User-agent: unisterbot Disallow: / User-agent: UnisterBot Disallow: / User-agent: UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de) Disallow: / User-agent: pixray Disallow: / User-agent: Pixray Disallow: / User-agent: pixray-seeker Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: Pixray-Seeker/1.1 Disallow: / User-agent: Pixray-Seeker/1.1 (Pixray-Seeker; crawler@pixray.com) Disallow: / User-agent: Screaming Frog SEO Spider/1,80 Disallow: / User-agent: CatchBot Disallow: / User-agent: Dropcatcher.de Disallow: / User-agent: CligooRobot/1.0 Disallow: / User-agent: cityreview Disallow: / User-agent: sistrix Disallow: / User-agent: spbot Disallow: / User-agent: dotbot Disallow: / User-agent: SurveyBot Disallow: / User-agent: Eurobot Disallow: / User-agent: MJ12bot Disallow: / User-agent: WebmasterCoffee Disallow: / User-agent: AboutUsBot Disallow: / User-agent: BizInformation Disallow: / User-agent: 008 Disallow: / User-agent: WebImages 0.3 Disallow: / User-agent: panscient.com Disallow: / User-agent: CazoodleBot Disallow: / User-agent: Python-urllib/2.1 Disallow: / User-agent: ia_archiver Disallow: / User-agent: VadixBot Disallow: / User-agent: Jim Disallow: / User-agent: Gigamega.bot/1.0 Disallow: / User-agent: EnaBot/1.2 Disallow: / User-agent: AhrefsBot Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: PagesInventory Disallow: / User-agent: * Crawl-Delay: 10 Disallow: /membership-services/health-a-safety/public-has-resources-a-faqs Disallow: /about-afpe Disallow: /professional-leadership/afpe-national-conference Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /components/user/login/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ Disallow: /xmlrpc/ Disallow: /email_templates/ Disallow: /logs Disallow: /templates Disallow: /reports