Python 3.x 在python中使用BeautifulSoup从网站上抓取报告

我正在尝试从公司网站下载报告。最后，我想下载所有可用的报告

我在网络垃圾方面几乎没有经验，所以我在定义正确的搜索模式时遇到了一些困难。以前我需要删除所有包含PDF的链接，即我可以使用soup.select（'div[id=“id name”]a[data type=“PDF”]”）。但是对于这个网站，没有列出链接的数据类型。如何选择“报告和演示文稿”下的所有链接？以下是我尝试过的，但它返回一个空列表：

from bs4 import BeautifulSoup
import requests

url = "https://www.investorab.com/investors-media/reports-presentations/"
response = requests.get(url)
soup = BeautifulSoup(response.text, 'lxml')

# Select all reports, publication_dates
reports = soup.select('div[class="html not-front not-logged-in no-sidebars page-events-archive i18n-en"] a[href]')
pub_dates = soup.select('div[class="html not-front not-logged-in no-sidebars page-events-archive i18n-en"] div[class="field-content"]')

---

我还想选择所有出版物的日期，但也以空列表结束。感谢您在正确方向上提供的任何帮助。

您需要做的是遍历页面，或者我所做的只是遍历year参数。一旦你得到了年度清单，就可以得到每个报告的链接，然后在每个链接中找到pdf链接。然后，您将使用该pdf链接写入文件：

from bs4 import BeautifulSoup
import requests
import os

# Gets all the links
linkList = []
url = 'https://vp053.alertir.com/v3/en/events-archive?'
for year in range(1917,2021):

    query = 'type%5B%5D=report&type%5B%5D=annual_report&type%5B%5D=cmd&type%5B%5D=misc&year%5Bvalue%5D%5Byear%5D=' + str(year)

    response = requests.get(url + query )
    soup = BeautifulSoup(response.text, 'html.parser')

    links = soup.find_all('a', href=True)
    linkList += [link['href'] for link in links if 'v3' in link['href']]
    print ('Gathered links for year %s.' %year)

# Go to each link and get the pdsf within them
print ('Downloading PDFs...')
for link in linkList:
    url = 'https://vp053.alertir.com' + link
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')

    for pdflink in soup.select("a[href$='.pdf']"):
        folder_location = 'C:/test/pdfDownloads/'
        if not os.path.exists(folder_location):
            os.mkdir(folder_location)

        try:
            filename = os.path.join(folder_location,pdflink['href'].split('/')[-1])
            with open(filename, 'wb') as f:
                f.write(requests.get('https://vp053.alertir.com' + pdflink['href']).content)
                print ('Saved: %s' %pdflink['href'].split('/')[-1])
        except Exception as ex:
             print('%s not saved. %s' %(pdflink['href'],ex))

---

谢谢，这很好地回答了我的变化。非常感谢。