Sunday, May 5, 2013

WebClient slow in crawling (C#)

Here's a tip i got from stack overflow on webclient slow on web crawling.
1
2
3
ServicePointManager.DefaultConnectionLimit = int.MaxValue;  
 ServicePointManager.MaxServicePoints = int.MaxValue;  
 ServicePointManager.MaxServicePointIdleTime = 0; 
I simply changed the default connection limit and max service points
to a numeric value. Then, the crawling starts to speed up.

Greg

0 comments:

Post a Comment