Warning: file_get_contents(/data/phpspider/zhask/data//catemap/6/rest/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
C# 在浏览器中查看源内容与在c中查看httpwebrequest的源内容不同#_C#_Asp.net_Web Scraping - Fatal编程技术网

C# 在浏览器中查看源内容与在c中查看httpwebrequest的源内容不同#

C# 在浏览器中查看源内容与在c中查看httpwebrequest的源内容不同#,c#,asp.net,web-scraping,C#,Asp.net,Web Scraping,我正在尝试从该页面中删除内容: 我遇到的问题是,在浏览器中打开该url时,我得到了需要删除的所有内容,但删除代码中的同一链接时,在价格和卖家信息下方缺少两个(重要)部分,即评论编号和评级。 以下是c#中内部web客户端的屏幕截图: 以下是我试图获取内容的代码: public string navGet(string inURL, CookieContainer inCookieContainer, bool GZip, string proxyAddress, int proxyPort

我正在尝试从该页面中删除内容:

我遇到的问题是,在浏览器中打开该url时,我得到了需要删除的所有内容,但删除代码中的同一链接时,在价格和卖家信息下方缺少两个(重要)部分,即评论编号和评级。 以下是c#中内部web客户端的屏幕截图:

以下是我试图获取内容的代码:

    public string navGet(string inURL, CookieContainer inCookieContainer, bool GZip, string proxyAddress, int proxyPort,string proxyUserName, string proxyPassword)
    {
        try
        {
        this.currentUrl = inURL;           
        HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(inURL);
    webRequest.Timeout = this.TimeOutSetting;
    webRequest.CookieContainer = inCookieContainer;

    if (proxyAddress == "0" || proxyPort == 0)
    { }
    else
    {
        webRequest.Proxy = new WebProxy(proxyAddress, proxyPort);
        // Use login credentials to access proxy
        NetworkCredential networkCredential = new NetworkCredential(proxyUserName, proxyPassword);
        webRequest.Proxy.Credentials = networkCredential;
    }

    Uri destination = webRequest.Address;
    webRequest.KeepAlive = true;
    webRequest.Method = "GET";
    webRequest.Accept = "*/*";
    webRequest.Headers.Add("Accept-Language", "en-us");
    if (GZip)
    {
        webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
    }
        webRequest.AllowAutoRedirect = true;
    webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";

        webRequest.ContentType = "text/xml";         
        //webRequest.CookieContainer.Add(inCookieContainer.GetCookies(destination));

    try
    {
        string strSessionID = inCookieContainer.GetCookies(destination)["PHPSESSID"].Value;
        webRequest.Headers.Add("Cookie", "USER_OK=1;PHPSESSID=" + strSessionID);
    }
    catch (Exception ex2)
    {

    }
        HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
        if (webRequest.HaveResponse)
        {
            // First handle cookies
            foreach(Cookie retCookie in webResponse.Cookies)
            {
                bool cookieFound = false;
                foreach(Cookie oldCookie in inCookieContainer.GetCookies(destination))
                {
                    if (retCookie.Name.Equals(oldCookie.Name))
                    {
                        oldCookie.Value = retCookie.Value;
                        cookieFound = true;
                    }
                }
                if (!cookieFound)
                    inCookieContainer.Add(retCookie);
            }                


            // Read response
        Stream responseStream = responseStream = webResponse.GetResponseStream();



        if (webResponse.ContentEncoding.ToLower().Contains("gzip"))
        {
            responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
        }
        else if (webResponse.ContentEncoding.ToLower().Contains("deflate"))
        {
            responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);
        }

        StreamReader stream = new StreamReader(responseStream, System.Text.Encoding.Default);

        string responseString = stream.ReadToEnd();
            stream.Close();
            this.currentUrl = webResponse.ResponseUri.ToString();
            this.currentAddress = webRequest.Address.ToString();               
            setViewState(responseString);
            return responseString;
        }
        throw new Exception("No response received from host.");
        return "An error was encountered";
        }
        catch(Exception ex)
        {
            //MessageBox.Show("NavGet:" + ex.Message);
            return ex.Message;
        }
    }

看起来这是因为评论数量和评分是使用Java脚本(可能是AJAX或其他东西)动态生成的。在这种情况下,您需要分析在浏览器中加载页面时发生的其他流量,并找到这些数据的传输位置,或者分析JavaScript代码以查看其生成方式