.NET GENES

Here's a tip i got from stack overflow on webclient slow on web crawling. 1 2 3 ServicePointManager.DefaultConnectionLimit = int .MaxValue; ServicePointManager.MaxServicePoints = int .MaxValue; ServicePointManager.MaxServicePointIdleTime = 0; I simply changed the default connection limit and max service points to a numeric value. Then, the crawling starts to speed up. Greg

In our case when we encountered this error using WebClient, the solution was to set the user agent and add it as header to the WebCclient object. newWebClient.Headers.Add( "user-agent" , "Mozilla/5.0 (Windows NT 6.1; rv:2.0) Gecko/20100101 Firefox/4.0" );

In a scenario where you want to download xml data using webclient, the response from firefox or any browser will display the xml file correctly. However, using a web request/webclient.downloadstring to download the xml file, the response is somewhat corrupted or in hashed form. The response headers are the following: 1. Content-Type - application/xml 2. Content-Encoding - gzip As you can see, the content encoding is of type gzip . The solution is to override the web request method to something like this: class DecompressGzipResponse : WebClient { protected override WebRequest GetWebRequest (Uri address) { HttpWebRequest request = (HttpWebRequest) base .GetWebRequest(address); request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate; return request; } } //to use this in the program (main function) DecompressGzipResponse client = new DecompressGzipResponse (); Listi...

Here is a snippet to use WebProxy. This is useful for scraping or other webrequest activities. Dim webclient As New System.Net.WebClient Try 'true for bypassOnLocal webclient .Proxy = New System.Net.WebProxy( "180.95.129.232:80" , true ) wc.OpenRead( "http://yoursite.com" ) lblStatus.Text = "ok" Catch lblStatus.Text = "error proxy" End Try

Using downloadstring of webclient returns an error as stated by the post title. The solution was to add user agent header to the webclient headers. The value of user agent can be found in the mozilla firebug. Solution: string URL = string .Format(your_url); newWebClient.Headers.Add( "user-agent" , "Mozilla/5.0 (Windows NT 6.1; rv:2.0) Gecko/20100101 Firefox/4.0 "); string pageSource = newWebClient.DownloadString(URL); pageSource = System.Web.HttpUtility.HtmlDecode(pageSource);

I experimented a snippet to get my ISP provided IP address using whatismyip. Using tutorials or code snippets from links, the web request just returned a 404 error. After reading the whatismyip API again,they have changed their automation URL. Below is the code snippet. //previous URL present on most tutorials which is not working //string whatIsMyIp = "http://www.whatismyip.com/automation/n09230945.asp"; //updated URL from whatismyip API string whatIsMyIp = "http://automation.whatismyip.com/n09230945.asp" ; WebClient wc = new WebClient(); UTF8Encoding utf8 = new UTF8Encoding(); string requestHtml = "" ; requestHtml = utf8.GetString(wc.DownloadData(whatIsMyIp));