网页抓取：下一页是用Javascript呈现的，我如何使用Scrapy获得它

我一直在尝试用Scrapy刮这个网站。我可以获取页面上的内容，但无法转到下一页，因为它似乎是用javascript呈现的。如何做到这一点？

我不确定如何实现使用scrapy所做的工作，但看起来javascript正在从后端API获取所有结果。您可以通过浏览器开发工具找到后端URL和AJAX请求的详细信息。看起来都像下面的代码，试试看。您可以直接从他们的API中提取您要查找的信息

import requests

payload = {
    "count":24,
    "offset":0,
    "sorts":{
        "0":{
            "listingContractDate":"desc"
        }
    },
    "terms":{
        "place":{
            "lat":40.70668199998021,
            "lon":-73.97795499996471,
            "city":"New York",
            "state":"NY",
            "placename":"New York, NY",
            "placeType":"city",
            "placeId":"3651000",
            "areaSquareMiles":308.12
        },"locationRect":{
            "minLat":40.3400891972592,
            "maxLat":41.07193158068027,
            "minLon":-74.24986662109377,
            "maxLon":-73.70604337890627
        },
        "bPropertyType":[
            "Single Family",
            "Condo/Townhome",
            "Mobile Home",
            "Multi-Family",
            "Rental","Farm",
            "Land"],
        "bStatus":[
            "For Sale",
            "Under Contract"
        ],
        "city":[
            "New York"
        ],
        "State":[
            "NY"
        ]
    },
    "listingLoadLevel":"Search"
}


r = requests.post("https://public-api-gateway-prod.kube.remax.booj.io/listings/search/run/", json=payload)
print(r.json())