Python 如何从不同的URL获取xpath,由start\u requests方法返回
这是我的密码:Python 如何从不同的URL获取xpath,由start\u requests方法返回,python,xpath,scrapy,web-crawler,Python,Xpath,Scrapy,Web Crawler,这是我的密码: import scrapy from scrapy.spider import BaseSpider from scrapy.selector import Selector import MySQLdb class AmazonSpider(BaseSpider): name = "amazon" allowed_domains = ["amazon.com"] start_urls = [] def parse(self, respons
import scrapy
from scrapy.spider import BaseSpider
from scrapy.selector import Selector
import MySQLdb
class AmazonSpider(BaseSpider):
name = "amazon"
allowed_domains = ["amazon.com"]
start_urls = []
def parse(self, response):
print self.start_urls
def start_requests(self):
conn = MySQLdb.connect(user='root',passwd='root',db='mydb',host='localhost')
cursor = conn.cursor()
cursor.execute(
'SELECT url FROM products;'
)
rows = cursor.fetchall()
for row in rows:
yield self.make_requests_from_url(row[0])
conn.close()
如何获取start\u请求
函数返回的URL的xpath
注意:URL属于不同的域,不尽相同。
yield
使start\u请求成为生成器。使用for
循环获取从中返回的每个结果
像这样:
...
my_spider = AmazonSpider()
for my_url in my_spider.start_requests():
print 'we get URL: %s' % str(my_url)
...