WebClient Slow In Crawling Or Web Scraping A Website In C#
Here's a tip i got from stack overflow on webclient slow on web crawling. 1 2 3 ServicePointManager.DefaultConnectionLimit = int .MaxValue; ServicePointManager.MaxServicePoints = int .MaxValue; ServicePointManager.MaxServicePointIdleTime = 0; I simply changed the default connection limit and max service points to a numeric value. Then, the crawling starts to speed up. Greg