Python 如何将结果直接导出到MySQL数据库
这是我的代码,可以根据不同国家的输入要求,将结果导出到csv文件中,然后手动上传到MySQL,但我想知道是否有办法自动完成。我已经设置了MySQL数据库。Err,你的意思是说?编写自己的自定义项目管道,如以下示例所示:Python 如何将结果直接导出到MySQL数据库,python,mysql,scrapy,Python,Mysql,Scrapy,这是我的代码,可以根据不同国家的输入要求,将结果导出到csv文件中,然后手动上传到MySQL,但我想知道是否有办法自动完成。我已经设置了MySQL数据库。Err,你的意思是说?编写自己的自定义项目管道,如以下示例所示: from scrapy.spider import BaseSpider from project.items import QualificationItem from scrapy.selector import HtmlXPathSelector from scrapy.h
from scrapy.spider import BaseSpider
from project.items import QualificationItem
from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
from urlparse import urljoin
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0'
class recursiveSpider(BaseSpider):
name = 'bristol'
allowed_domains = ['bristol.ac.uk/']
start_urls = ['http://www.bristol.ac.uk/international/countries/']
def parse(self, response):
hxs = HtmlXPathSelector(response)
xpath = '//*[@id="all-countries"]/li/ul/li/a/@href'
a_of_the_link = '//*[@id="all-countries"]/li/ul/li/a/text()'
for text, link in zip(hxs.select(a_of_the_link).extract(), hxs.select(xpath).extract()):
yield Request(urljoin(response.url, link),
meta={'a_of_the_link': text},
headers={'User-Agent': USER_AGENT},
callback=self.parse_linkpage,
dont_filter=True)
def parse_linkpage(self, response):
hxs = HtmlXPathSelector(response)
item = QualificationItem()
xpath = """
//h2[normalize-space(.)="Entry requirements for undergraduate courses"]
/following-sibling::p[not(preceding-sibling::h2[normalize-space(.)!="Entry requirements for undergraduate courses"])]
"""
item['Qualification'] = hxs.select(xpath).extract()[1:]
item['Country'] = response.meta['a_of_the_link']
return item