A class library for crawling websites and taking actions when different urls are encountered. This can be used for: downloadng sites/images/movies, collecting pages for testing, computing statistics about websites, or anything else that involves crawling the wev.