Web scraping 将scrapy导入json时遇到问题

Web scraping 将scrapy导入json时遇到问题,web-scraping,scrapy,scrapy-spider,Web Scraping,Scrapy,Scrapy Spider,我正试图从Craigslist中提取一些信息并将其存储在JSON文件中,但信息的存储有点错误。我得到的不是一个[title,link,location,time]数组,而是一个包含所有标题的数组,一个包含所有链接的数组,等等。我的标题是错误的还是for循环本身是错误的 from scrapy.spiders import BaseSpider from scrapy.selector import HtmlXPathSelector from scrapy.selector import Sel

我正试图从Craigslist中提取一些信息并将其存储在JSON文件中,但信息的存储有点错误。我得到的不是一个[title,link,location,time]数组,而是一个包含所有标题的数组,一个包含所有链接的数组,等等。我的标题是错误的还是for循环本身是错误的

from scrapy.spiders import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.selector import Selector
from craigslist_sample.items import CraigslistSampleItem

class MySpider(BaseSpider):
 name = "craig"
 allowed_domains = ["craigslist.org"]
 start_urls = ["https://pittsburgh.craigslist.org/search/ccc"]

 def parse(self, response):
    titles = response.selector.xpath("//p[@class='row']")
    items = []
    for titles in titles:
        item = CraigslistSampleItem()
        item["title"] = titles.xpath("//span[@id='titletextonly']").extract()
        item["link"] = titles.xpath("a/@href").extract()
        item["location"] = titles.xpath("//small").extract()
        item["time"] = titles.xpath('//time').extract()
        items.append(item)
    return items

这是因为您的内部XPath匹配从树的根开始的元素。相反,您需要通过在每个项目的上下文中预先添加一个点来强制它们工作:

for title in titles:
    item = CraigslistSampleItem()
    item["title"] = title.xpath(".//span[@id='titletextonly']").extract()
    item["link"] = title.xpath("a/@href").extract()
    item["location"] = title.xpath(".//small").extract()
    item["time"] = title.xpath('.//time').extract()
    yield item