Python 如何刮取JSON网页

嘿，所以我有一些刮html的经验，但从来没有json，所以我需要使用scrapy刮下面的网页，我在网上找到了一个教程，它使用scrapy和jmspath从web上刮json数据。我得到的教程工作，但我试图改变它与我的网站工作，以没有运气。没有错误，但不返回任何数据。任何帮助都将不胜感激

items.py

import scrapy


class NameItem(scrapy.Item):
    """User item definition for jsonplaceholder /LoginSpider endpoint."""
    name = scrapy.Field()
    condition = scrapy.Field()
    price = scrapy.Field()
    rarity = scrapy.Field()

罗根斯皮德酒店

import scrapy
import json
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import NameItem
from scrapy.loader import ItemLoader
from scrapy.loader.processors import Join, MapCompose, SelectJmes


class UserSpider(scrapy.Spider):
    """Spider to scrape `http://www.starcitygames.com/buylist/search?search-type=category&id=5061`."""
    name = 'LoginSpider'
    allowed_domains = ['http://www.starcitygames.com/buylist/search?search-type=category&id=5061']
    start_urls = ['http://www.starcitygames.com/buylist/search?search-type=category&id=5061']
    # dictionary to map UserItem fields to Jmes query paths
    jmes_paths = {
            'name': 'name',
            'condition': 'condition',
            'price': 'price',
            'rarity': 'rarity',
            }

    def parse(self, response):
        jsonresponse = json.loads(response.body_as_unicode())
        for user in jsonresponse:
            loader = ItemLoader(item=NameItem())  # create an ItemLoader to populate a NameItem
            loader.default_input_processor = MapCompose(str)  # apply str conversion on each value
            loader.default_output_processor = Join(' ')
            for (field, path) in self.jmes_paths.items():
                loader.add_value(field, SelectJmes(path)(user))
            yield loader.load_item()

此url的响应分为三个级别：

“好的”

“搜索”

“结果”##这包含数据

和结果键有多个值，您应该迭代这些值。 值中包含数据。 试试这个代码，希望你能帮忙

这是模块items.py

class SoResponseItem(scrapy.Item):
        name = scrapy.Field()
        condition = scrapy.Field()
        price = scrapy.Field()
        rarity = scrapy.Field()

这是蜘蛛

import scrapy
import json
from SO_response.items import SoResponseItem

class LoginspiderSpider(scrapy.Spider):
    name = 'LoginSpider'
    allowed_domains = ['www.starcitygames.com']
    url = 'http://www.starcitygames.com/'

    def start_requests(self):
        yield scrapy.Request(url=self.url, callback=self.parse)

    def parse(self, response):
        url = response.urljoin('buylist/search?search-type=category&id=5061')
        yield scrapy.Request(url=url, callback=self.parse_data)

    def parse_data(self, response):
        jsonreponse = json.loads(response.body)
        for result in jsonreponse['results']:
            for index in range(len(result)):
                items = SoResponseItem()
                items['name'] = result[index]['name']
                items['condition'] = result[index]['condition']
                items['price'] = result[index]['price']
                items['rarity'] = result[index]['rarity']
                yield items

在您的shell中尝试： scrapy crawl-o jmes.json