Python Scrapy中间件设置
我正在尝试使用scrapy访问公共代理以获取一些数据。当我尝试运行代码时,出现以下错误: ImportError:加载对象“craiglist.Middleware.ProxyMiddleware”时出错:没有名为Middleware的模块 我已经用以下代码创建了Python Scrapy中间件设置,python,web-scraping,web-crawler,scrapy,Python,Web Scraping,Web Crawler,Scrapy,我正在尝试使用scrapy访问公共代理以获取一些数据。当我尝试运行代码时,出现以下错误: ImportError:加载对象“craiglist.Middleware.ProxyMiddleware”时出错:没有名为Middleware的模块 我已经用以下代码创建了middleware.py文件: import base64 # Start your middleware class class ProxyMiddleware(object): # overwrite process re
middleware.py
文件:
import base64
# Start your middleware class
class ProxyMiddleware(object):
# overwrite process request
def process_request(self, request, spider):
# Set the location of the proxy
request.meta['proxy'] = "http://124.200.36.150:8118"
settings.py:
DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
'craiglist.middlewares.ProxyMiddleware': 100,
}
欢迎任何帮助。使用此设置,您需要将
middleware.py
升级到craiglist
包中。请显示您的目录结构。好的,middleware.py位于C:\Users\Downloads\craiglist\craiglist\spider和settings.py中,位于C:\Users\Downloads\craiglist\craiglist中