Crawler
A web crawler is a bot that systematically browses the World Wide Web, typically for the purpose of web indexing. A web crawler may also be called a web spider, an ant, or a web robot. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a … Read more