C# Fiddler网站web请求模拟屏幕抓取_C#_C# 4.0_Cookies_Httpwebrequest

C# Fiddler网站web请求模拟屏幕抓取

c# c#-4.0 cookies

C# Fiddler网站web请求模拟屏幕抓取,c#,c#-4.0,cookies,httpwebrequest,C#,C# 4.0,Cookies,Httpwebrequest,我正在尝试登录一个网站，通过我的帐户下载数据。这是登录后表单的原始Fiddler请求 POST login/login.jsp HTTP/1.1 Host: server.com Connection: keep-alive Content-Length: 73 Cache-Control: max-age=0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Origin: https://serve

我正在尝试登录一个网站，通过我的帐户下载数据。这是登录后表单的原始Fiddler请求

POST login/login.jsp HTTP/1.1
Host: server.com
Connection: keep-alive
Content-Length: 73
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: https://server.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: https://server.com/login/login.jsp
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: __utma=109610308.114257620.1370889472.1373479499.1371761934.3; __utmc=109613338; __utmz=109610308.1373249472.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _bcvm_vid_424161365915852877=4393500994580715020; _bcvm_vrid_424161365915852877=4393492275825713189; WT_FPC=id=199.234.233.42-2645888112.30303753:lv=1371356395815:ss=1371758333825; JSESSIONID=RGJGy4yQ2WCXRPbnhxCTKGb2rZh39b67d8g8PktTQLqfsBQTlTlYLTD!1154156211; BIGipServeresuite_prod_pool=295635768.2713643.0000

然后，它会作出以下回应：

HTTP/1.1 302 Moved Temporarily
Date: Fri, 21 Jun 2013 12:39:46 GMT
Location: https://server.com/login/redirect.jsp?APPLICATION=0
Content-Type: text/html
Set-Cookie: SECURITY_SESSION_ID=383826514*198399234219875960; domain=.server.com; path=/
Connection: Close
Set-Cookie: BIGipServeresuite_prod_pool=294168768.27163.0000; expires=Fri, 21-Jun-2013 13:09:47 GMT; path=/
Content-Length: 3669

该安全会话ID是在站点上执行任何操作所需的

为了模仿它，我编写了以下代码：

   //GET the Login page - I preform a quick get to pick up the first two important cookies

          HttpWebRequest GETLoginRequest = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
          GETLoginRequest.Method = "GET";
          GETLoginRequest.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
          GETLoginRequest.AllowAutoRedirect = false;
          GETLoginRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)";
          GETLoginRequest.CookieContainer = cookieJar;  

          HttpWebResponse GETLoginResponse = (HttpWebResponse)GETLoginRequest.GetResponse(); //Gets the JSession and BIGipServer cookies
          Console.Write(" \n 3rd count after GETLoginResponse : " + cookieJar.Count + "\n");

 //POST Login

          HttpWebRequest POSTLoginRequest = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
          POSTLoginRequest.Method = "POST";
          WebHeaderCollection myWebHeaderCollection = POSTLoginRequest.Headers;
          POSTLoginRequest.AllowAutoRedirect = true;
          byte[] bytes = Encoding.ASCII.GetBytes(formParams);

       ///Cache
          POSTLoginRequest.Headers.Add(HttpRequestHeader.CacheControl, "max-age=0"); 

        //Client
          POSTLoginRequest.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
          POSTLoginRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate,sdch"); 
          myWebHeaderCollection.Add("Accept-Language:en-US");
          POSTLoginRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)";
        //Cookies/Login
          POSTLoginRequest.CookieContainer = cookieJar; //The cookie jar contains mainly added
        //Entity
          POSTLoginRequest.ContentLength = bytes.Length;
          POSTLoginRequest.ContentType = "Content-Type: application/x-www-form-urlencoded";
        //Miscellanous
          POSTLoginRequest.Headers.Add("Origin: https://server.com");
          POSTLoginRequest.Referer = "https://server.com/login/login.jsp";
        //Transport

              //Fix I found to allow Connection: Keep-Alive
          var sp = POSTLoginRequest.ServicePoint;
          var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
          prop.SetValue(sp, (byte)0, null);

          ServicePointManager.Expect100Continue = false;

          POSTLoginRequest.Host = "server.com";

          using (Stream os = POSTLoginRequest.GetRequestStream())
          {
              os.Write(bytes, 0, bytes.Length);
          }
          HttpWebResponse POSTLoginResponse = (HttpWebResponse)POSTLoginRequest.GetResponse();
          Console.Write(" \n 4th count after POSTLoginResponse : " + cookieJar.Count + "\n");

最后，我在fiddler中的请求如下：

 POST /login/login.jsp HTTP/1.1
Cache-Control: max-age=0
Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)
Content-Type: Content-Type: application/x-www-form-urlencoded
Origin: https://server.com
Referer: https://server.com/login/login.jsp
Host: esuite.pjm.com
Cookie: __utma=1096103034.235016339.1371048460.1371048460.1371048460.1; __utmz=109610428.1371048460.1.1.utmcsr=bing|utmccn=(organic)|utmcmd=organic|utmctr=pjm; _bcvm_vrid_424161365915852877=4393493862784729423; WT_FPC=id=199.234.233.42-3603288592.30304123:lv=1371044861062:ss=1371044859892; JSESSIONID=RGbYQd7JnPdNkTvtGCzQ9NLyFgfBnnyLFzbvKPg2Y0gLnhL2hp8F!-1770592471; BIGipServeresuite_prod_pool=327723200.27163.0000
Content-Length: 73
Connection: Keep-Alive

这几乎是一样的，但我得到的回应是：

HTTP/1.1 200 OK
Date: Fri, 21 Jun 2013 13:58:17 GMT
Content-Length: 3356
Content-Type: text/html
Set-Cookie: BIGipServeresuite_prod_pool=327723200.27163.0000; expires=Fri, 21-Jun-2013 14:28:17 GMT; path=/

我想有一件事与浏览器中的答案有关，那就是暂时移动了302，但我真的不知道。两个重要的cookie似乎是JSession和BIGIp，因为这些都是由网站设置的。我手动添加的其他cookies似乎是谷歌分析cookies，我认为它们并不重要。不管怎么说，标题几乎是空的

相同，但它仍然没有响应我正在寻找的安全会话ID。有人知道我做错了什么吗

您的“接受”标题不同，提交的内容类型部分错误：

POSTLoginRequest.ContentType = "Content-Type: application/x-www-form-urlencoded";

应该是

POSTLoginRequest.ContentType = "application/x-www-form-urlencoded";

确保post参数与fiddler示例中的相同：

因此，在将它们转换为字节之前，请先执行此操作

HttpUtility.Urlencode(forumparameters)

看看这是否有效

因为您在ContentLength中也缺少计数，所以这一定是原因。

谢谢您的回复，我更改了它。关于内容类型，这是一个多么愚蠢的错误啊不管怎样，它仍然不能用正确的Cookie进行回复。您是否尝试过使用CURLLib？由于我多次遇到从https页面获取cookies的问题。如果您想继续使用HTTPWEBREQUEST，只需将cookies复制到容器中即可。如果您需要示例代码，请告诉我。这很简单。谢谢，我没有使用CURLLib，我只是使用fiddler，并正确地遵循了它。我读到你应该总是把AllowAutoRedirect=false。。。。情况并非总是如此。有时候你应该把它设为真。这取决于网站。我上面代码中的代码片段中的所有内容都设置为false（这与AllowAutoRedirect有关）。然后在请求中，将其设置为true并运行后：Dcurlib不像fiddler，而是httpwebrequest，它甚至不需要复制我写过的所有请求。我只需将url更改为get、post，它工作得非常好。您还应该检查一下，它内部还支持多线程。您可以发送一个url数组，带或不带post参数。