Warning: file_get_contents(/data/phpspider/zhask/data//catemap/8/http/4.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Http Curl和Golang中的不同请求状态(200和403)_Http_Go_Ssl_Curl - Fatal编程技术网

Http Curl和Golang中的不同请求状态(200和403)

Http Curl和Golang中的不同请求状态(200和403),http,go,ssl,curl,Http,Go,Ssl,Curl,我在使用本地MacBookAir上的标准golang http库从网站()获取数据时遇到问题-我的状态为403禁止。从浏览器(使用禁用的JavaScript)打开该站点没有问题,使用标准Curl函数(Curl-v)获取数据也没有问题—始终获取状态200 有一个Golang代码: package main import ( "crypto/tls" "crypto/x509" "flag" "f

我在使用本地MacBookAir上的标准golang http库从网站()获取数据时遇到问题-我的状态为403禁止。从浏览器(使用禁用的JavaScript)打开该站点没有问题,使用标准Curl函数(Curl-v)获取数据也没有问题—始终获取状态200

有一个Golang代码:

package main

import (
    "crypto/tls"
    "crypto/x509"
    "flag"
    "fmt"
    "io/ioutil"
    "net/http/httputil"
    "time”
    "database/sql"
    "log"
    "net/http"

    "golang.org/x/net/http2”
    "github.com/PuerkitoBio/goquery"
)


func main() {

    request, err := http.NewRequest("GET", "https://www.avito.ru/moscow", nil)
    if err != nil {
        log.Fatal(err)
    }
    request.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36")
    request.Header.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
    request.Header.Add("Accept-Language", "en-us")
    request.Header.Add("Connection", "keep-alive")
    request.Header.Add("Host", "www.avito.ru")
    fmt.Println(request.Header)

    // Do something with the request
    client := &http.Client{}
    caCert, err := ioutil.ReadFile("/etc/ssl/cert.pem")
    if err != nil {
        log.Fatalf("Reading server certificate: %s", err)
    }
    caCertPool := x509.NewCertPool()
    caCertPool.AppendCertsFromPEM(caCert)

    var httpVersion = flag.Int("version", 2, "HTTP version")

    fmt.Println(string(*httpVersion))
    client.Transport = &http2.Transport{
        TLSClientConfig: &tls.Config{
            RootCAs:            caCertPool,
            // InsecureSkipVerify: true,
        },
    }

    requestDump, err := httputil.DumpRequest(request, true)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Println(string(requestDump))

    response, err := client.Do(request)
    fmt.Println(string(response.Proto))

    if err != nil {
        log.Fatal(err)
    }

    defer response.Body.Close()

    fmt.Println(response.Status)

    doc, err := goquery.NewDocumentFromReader(response.Body)
    if err != nil {
        fmt.Println("here2")
        log.Fatal(err)
    }
    pageTitle := doc.Find("title").Contents().Text()

    fmt.Println(pageTitle)
}
有两张照片:



GET / HTTP/1.1
Host: www.avito.ru
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Connection: keep-alive
Accept-Encoding: gzip

HTTP/2.0
map[Content-Type:[text/html; charset=utf-8] Date:[Sat, 14 Nov 2020 13:14:10 GMT] Server:[nginx] X-Content-Type-Options:[nosniff] X-Envoy-Upstream-Service-Time:[14] X-Xss-Protection:[1; mode=block]]
403 Forbidden
here3
Доступ с вашего IP-адреса временно ограничен — Авито
here4
Доступ с вашего IP-адреса временно ограничен — Авито
有Curl-v的指纹:


Alexanders-MacBook-Air:~ alexchistyakov$ curl -v https://www.avito.ru/moscow
*   Trying 185.89.12.132...
* TCP_NODELAY set
* Connected to www.avito.ru (185.89.12.132) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=RU; ST=Moscow; L=Moscow; O=Limited Liability Company KEH eCommerce; CN=*.avito.ru
*  start date: Feb  4 09:48:54 2019 GMT
*  expire date: Feb  4 09:48:54 2021 GMT
*  subjectAltName: host "www.avito.ru" matched cert's "*.avito.ru"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation CA - SHA256 - G2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f7feb00ba00)
> GET /moscow HTTP/2
> Host: www.avito.ru
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Sat, 14 Nov 2020 13:28:28 GMT
< content-type: text/html; charset=UTF-8
< set-cookie: u=2ke25u1o.n9oms3.9cgwbvpc4sw0; path=/; expires=Tue, 10-Nov-37 13:28:28 GMT; HttpOnly; Max-Age=536112000; secure; domain=.avito.ru
< set-cookie: v=1605360508; path=/; expires=Sat, 14-Nov-20 13:58:28 GMT; HttpOnly; Max-Age=1800; secure; domain=.avito.ru; SameSite=Lax
< cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< set-cookie: buyer_location_id=621540; expires=Sun, 14-Nov-2021 13:28:28 GMT; Max-Age=31536000; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: luri=rossiya; expires=Sun, 15-Nov-2020 13:28:28 GMT; Max-Age=86400; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: buyer_selected_search_radius4=0_general; expires=Thu, 18-Mar-3019 13:28:28 GMT; Max-Age=31504464000; path=/; domain=.avito.ru; secure; SameSite=Lax
< set-cookie: tmpBannerHash=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: tmpSideBlockHash=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: tmpSearchBottomHash=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: sx=H4sIAAAAAAACAw3EwQqAIAwA0H%2FZucOiJcu%2FyYiRA4UWDhL%2Fvd7hdVjSu0tpT0lVSQ5CY%2FJ%2Fh9ihQYQr31osaMskfpCIV0UUFkOuajDBCXEOuBKFDXmMD9eve2RUAAAA; expires=Sat, 21-Nov-2020 13:28:28 GMT; Max-Age=604800; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: so=1605360508; expires=Sat, 14-Nov-2020 13:43:28 GMT; Max-Age=900; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: dfp_group=52; expires=Sat, 14-Nov-2020 13:38:28 GMT; Max-Age=600; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: buyer_laas_tooltip=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: tmpSrcFromBp=ijrkns6kvj4gwc4ggwok00g8gs08oks; expires=Sat, 14-Nov-2020 14:28:28 GMT; Max-Age=3600; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: sessid=e33049edd828b91d73300175d802c306.1605360508; expires=Sun, 15-Nov-2020 13:28:28 GMT; Max-Age=86400; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: buyer_from_page=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=Lax
< x-request-url: /moscow
< x-envoy-upstream-service-time: 532
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< 

<!DOCTYPE html>
   
<html> <head> <script>


Alexanders MacBook Air:~alexchistyakov$curl-vhttps://www.avito.ru/moscow
*正在尝试185.89.12.132。。。
*TCP_节点集
*连接到www.avito.ru(185.89.12.132)端口443(#0)
*阿尔卑斯山,提供h2
*ALPN,提供http/1.1
*已成功设置证书验证位置:
*CAfile:/etc/ssl/cert.pem
卡帕斯:没有
*TLSv1.2(输出),TLS握手,客户端问候(1):
*TLSv1.2(IN)、TLS握手、服务器hello(2):
*TLSv1.2(IN),TLS握手,证书(11):
*TLSv1.2(IN)、TLS握手、服务器密钥交换(12):
*TLSv1.2(IN),TLS握手,服务器完成(14):
*TLSv1.2(输出)、TLS握手、客户端密钥交换(16):
*TLSv1.2(OUT),TLS更改密码,更改密码规范(1):
*TLSv1.2(输出),TLS握手,完成(20):
*TLSv1.2(IN),TLS更改密码,更改密码规范(1):
*TLSv1.2(IN),TLS握手,完成(20):
*使用TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256的SSL连接
*ALPN,服务器接受使用h2
*服务器证书:
*主题:C=RU;ST=莫斯科;L=莫斯科;O=有限责任公司KEH电子商务;CN=*.avito.ru
*开始日期:2月4日09:48:54格林尼治标准时间2019
*过期日期:2月4日09:48:54格林威治标准时间2021
*subjectAltName:主机“www.avito.ru”匹配证书“*.avito.ru”
*发行人:C=BE;O=全球信号内华达州;CN=全球签名组织验证CA-SHA256-G2
*SSL证书验证正常。
*使用HTTP2,服务器支持多用途
*连接状态已更改(HTTP/2已确认)
*升级后正在将流缓冲区中的HTTP/2数据复制到连接缓冲区:len=0
*使用流ID:1(易处理0x7f7feb00ba00)
>获取/HTTP/2
>主持人:www.avito.ru
>用户代理:curl/7.64.1
>接受:*/*
> 
*连接状态已更改(最大并发流==128)!
要想像Curl一样在Golang获得200状态,我需要做些什么?
谢谢

以下是工作示例:

package main
import "net/http"

func main() {
   r, e := http.Head("https://www.avito.ru/moscow")
   if e != nil {
      panic(e)
   }
   println(r.StatusCode == 200)
}

请给出一个简单的例子。@Volker代码简化错误消息说“从您的IP地址访问暂时受限-Avito”(以防万一)