Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/python/290.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Javascript 发现每个产品url_Javascript_Python_Web Scraping_Beautifulsoup_Scrapy - Fatal编程技术网

Javascript 发现每个产品url

Javascript 发现每个产品url,javascript,python,web-scraping,beautifulsoup,scrapy,Javascript,Python,Web Scraping,Beautifulsoup,Scrapy,需要提取所有产品URL,请尝试此操作。他们中的大多数人都找不到 url=”https://alcohaul.sg/" var x=document.querySelectorAll(“a”); var myarray=[] 对于(var i=0;i当你说“所有产品URL”时,你指的是主页上可见的产品,是吗 我在浏览器中访问了该页面并记录了我的网络流量。浏览器向一些REST api发出了各种HTTP GET请求。其中一个具有端点api/newarrival,其响应包含主页上产品的所有产品信息(JS

需要提取所有产品URL,请尝试此操作。他们中的大多数人都找不到

url=”https://alcohaul.sg/"

var x=document.querySelectorAll(“a”);
var myarray=[]
对于(var i=0;i当你说“所有产品URL”时,你指的是主页上可见的产品,是吗

我在浏览器中访问了该页面并记录了我的网络流量。浏览器向一些REST api发出了各种HTTP GET请求。其中一个具有端点
api/newarrival
,其响应包含主页上产品的所有产品信息(JSON)。您所要做的就是模拟该HTTP GET请求。这API似乎相当宽松-不需要请求头,甚至不需要任何查询字符串参数。然而,这也意味着,随着新的“新到达”的到来,此API的响应会随着时间而变化

def main():

    import requests

    url = "https://alcohaul.sg/api/newarrival"

    response = requests.get(url)
    response.raise_for_status()

    base_url = "https://alcohaul.sg/products/{}"

    for product in response.json():
        print(base_url.format(product["slug"]))

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())
输出:

https://alcohaul.sg/products/absolut-vodka-blue-70cl
https://alcohaul.sg/products/campari-70cl
https://alcohaul.sg/products/botanist-gin-70cl
https://alcohaul.sg/products/strangelove-premium-light-tonic-pack-4-x-180ml
https://alcohaul.sg/products/strangelove-premium-indian-tonic-no-8-pack-4-x-180ml
https://alcohaul.sg/products/strangelove-premium-coastal-tonic-pack-4-x-180ml
https://alcohaul.sg/products/strangelove-premium-lo-cal-double-ginger-beer-pack-4-x-300ml
...

滚动页面时是否动态加载产品?
https://alcohaul.sg/products/absolut-vodka-blue-70cl
https://alcohaul.sg/products/campari-70cl
https://alcohaul.sg/products/botanist-gin-70cl
https://alcohaul.sg/products/strangelove-premium-light-tonic-pack-4-x-180ml
https://alcohaul.sg/products/strangelove-premium-indian-tonic-no-8-pack-4-x-180ml
https://alcohaul.sg/products/strangelove-premium-coastal-tonic-pack-4-x-180ml
https://alcohaul.sg/products/strangelove-premium-lo-cal-double-ginger-beer-pack-4-x-300ml
...