DatayzeBot is the website crawler for our website analysis tool suite. If you see DatayzeBot in your access logs it is because someone has requested your site be analyzed. Don't want your site crawled in whole or in part? DatayzeBot respects the robots exclusion standard.
Example: To prevent the spider from crawling any URLs on your site simply add the following to your robots.txt file:
There are currently 2 apps that do not follow the robots exclusion standard:
Datayze does not store, cache or index pages crawled. Setting the noindex meta field will have no effect on the datayze spider.