Warning: file_get_contents(/data/phpspider/zhask/data//catemap/8/mysql/70.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Python 如何将结果直接导出到MySQL数据库_Python_Mysql_Scrapy - Fatal编程技术网

Python 如何将结果直接导出到MySQL数据库

Python 如何将结果直接导出到MySQL数据库,python,mysql,scrapy,Python,Mysql,Scrapy,这是我的代码,可以根据不同国家的输入要求,将结果导出到csv文件中,然后手动上传到MySQL,但我想知道是否有办法自动完成。我已经设置了MySQL数据库。Err,你的意思是说?编写自己的自定义项目管道,如以下示例所示: from scrapy.spider import BaseSpider from project.items import QualificationItem from scrapy.selector import HtmlXPathSelector from scrapy.h

这是我的代码,可以根据不同国家的输入要求,将结果导出到csv文件中,然后手动上传到MySQL,但我想知道是否有办法自动完成。我已经设置了MySQL数据库。

Err,你的意思是说?编写自己的自定义项目管道,如以下示例所示:
from scrapy.spider import BaseSpider
from project.items import QualificationItem
from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
from urlparse import urljoin

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0'

class recursiveSpider(BaseSpider):
name = 'bristol'
allowed_domains = ['bristol.ac.uk/']
start_urls = ['http://www.bristol.ac.uk/international/countries/']

def parse(self, response):
    hxs = HtmlXPathSelector(response)

    xpath = '//*[@id="all-countries"]/li/ul/li/a/@href'
    a_of_the_link = '//*[@id="all-countries"]/li/ul/li/a/text()'
    for text, link in zip(hxs.select(a_of_the_link).extract(), hxs.select(xpath).extract()):
        yield Request(urljoin(response.url, link),
        meta={'a_of_the_link': text},
        headers={'User-Agent': USER_AGENT},
        callback=self.parse_linkpage,
        dont_filter=True)

def parse_linkpage(self, response):
    hxs = HtmlXPathSelector(response)
    item = QualificationItem()
    xpath = """
            //h2[normalize-space(.)="Entry requirements for undergraduate courses"]
             /following-sibling::p[not(preceding-sibling::h2[normalize-space(.)!="Entry requirements for undergraduate courses"])]
            """
    item['Qualification'] = hxs.select(xpath).extract()[1:]
    item['Country'] = response.meta['a_of_the_link']
    return item