Python 3.x 在python中通过抓取子URL下载文件
我正在尝试从大量的web链接下载文档(主要是pdf格式),如下所示: 但是,这些链接无法直接访问pdf文件。您需要单击子URL来访问PDF。有没有办法抓取子URL并从中下载所有相关文件?我正在用下面的代码进行尝试,但到目前为止还没有任何成功,特别是对于这里列出的这些URL 如果您需要进一步澄清,请告诉我。我很乐意这样做。多谢各位Python 3.x 在python中通过抓取子URL下载文件,python-3.x,web-scraping,python-requests,scrapy,web-crawler,Python 3.x,Web Scraping,Python Requests,Scrapy,Web Crawler,我正在尝试从大量的web链接下载文档(主要是pdf格式),如下所示: 但是,这些链接无法直接访问pdf文件。您需要单击子URL来访问PDF。有没有办法抓取子URL并从中下载所有相关文件?我正在用下面的代码进行尝试,但到目前为止还没有任何成功,特别是对于这里列出的这些URL 如果您需要进一步澄清,请告诉我。我很乐意这样做。多谢各位 from simplified_scrapy import Spider, SimplifiedDoc, SimplifiedMain, utils class
from simplified_scrapy import Spider, SimplifiedDoc, SimplifiedMain, utils
class MySpider(Spider):
name = 'download_pdf'
allowed_domains = ["www.worldbank.org"]
start_urls = [
"https://projects.worldbank.org/en/projects-operations/document-detail/P167897?type=projects",
"https://projects.worldbank.org/en/projects-operations/document-detail/P173997?type=projects",
"https://projects.worldbank.org/en/projects-operations/document-detail/P166309?type=projects"
] # Entry page
def afterResponse(self, response, url, error=None, extra=None):
if not extra:
print ("The version of library simplified_scrapy is too old, please update.")
SimplifiedMain.setRunFlag(False)
return
try:
path = './pdfs'
# create folder start
srcUrl = extra.get('srcUrl')
if srcUrl:
index = srcUrl.find('year/')
year = ''
if index > 0:
year = srcUrl[index + 5:]
index = year.find('?')
if index>0:
path = path + year[:index]
utils.createDir(path)
# create folder end
path = path + url[url.rindex('/'):]
index = path.find('?')
if index > 0: path = path[:index]
flag = utils.saveResponseAsFile(response, path, fileType="pdf")
if flag:
return None
else: # If it's not a pdf, leave it to the frame
return Spider.afterResponse(self, response, url, error, extra)
except Exception as err:
print(err)
def extract(self, url, html, models, modelNames):
doc = SimplifiedDoc(html)
lst = doc.selects('div.list >a').contains("documents/", attr="href")
if not lst:
lst = doc.selects('div.hidden-md hidden-lg >a')
urls = []
for a in lst:
a["url"] = utils.absoluteUrl(url.url, a["href"])
# Set root url start
a["srcUrl"] = url.get('srcUrl')
if not a['srcUrl']:
a["srcUrl"] = url.url
# Set root url end
urls.append(a)
return {"Urls": urls}
# Download again by resetting the URL. Called when you want to download again.
def resetUrl(self):
Spider.clearUrl(self)
Spider.resetUrlsTest(self)
SimplifiedMain.startThread(MySpider()) # Start download
有一个API端点包含您在网站上看到的整个响应以及。。。文档的URL
pdf
:D
因此,您可以查询API,获取URL,最后获取文档
以下是方法:
导入请求
pids=[“P167897”、“P173997”、“P166309”]
对于pid中的pid:
终点=f“https://search.worldbank.org/api/v2/wds?" \
f“format=json&includepublicdocs=1&”\
f“fl=docna,lang,docty,repnb,docdt,doc\u authr,可在&”\
f“os=0&rows=20&proid={pid}&apilang=en”
documents=requests.get(end_point).json()[“documents”]
对于documents.values()中的document_数据:
尝试:
pdf\u url=文件\u数据[“pdfurl”]
打印(f“获取:{pdf_url}”)
以pdf格式打开(pdf_url.rsplit(“/”[-1],“wb”):
write(requests.get(pdf_url.content)
除KeyError外:
持续
输出:(完全下载的.pdf文件)
谢谢这个@baduker。我以前试过使用他们的API,但问题是下载会因请求大量文档而中断。是否有一种方法可以从中断发生的地方重新启动,否则所有内容都需要从头开始重新下载?当然有。它被称为重试逻辑,但您必须自己实现。提示:使用while循环和
try execpt
块,在请求之间暂停。
Fetching: http://documents.worldbank.org/curated/en/106981614570591392/pdf/Official-Documents-Grant-Agreement-for-Additional-Financing-Grant-TF0B4694.pdf
Fetching: http://documents.worldbank.org/curated/en/331341614570579132/pdf/Official-Documents-First-Restatement-to-the-Disbursement-Letter-for-Grant-D6810-SL-and-for-Additional-Financing-Grant-TF0B4694.pdf
Fetching: http://documents.worldbank.org/curated/en/387211614570564353/pdf/Official-Documents-Amendment-to-the-Financing-Agreement-for-Grant-D6810-SL.pdf
Fetching: http://documents.worldbank.org/curated/en/799541612993594209/pdf/Sierra-Leone-AFRICA-WEST-P167897-Sierra-Leone-Free-Education-Project-Procurement-Plan.pdf
Fetching: http://documents.worldbank.org/curated/en/310641612199201329/pdf/Disclosable-Version-of-the-ISR-Sierra-Leone-Free-Education-Project-P167897-Sequence-No-02.pdf
and more ...