Python 索引器：使用BeautifulSoup从网页中提取表

我试图从中提取包含航班到达信息的表（包含列-航班、承运人、始发地、日期、计划、估计、状态），但我得到以下错误：

  IndexError Traceback (most recent call last)
  <ipython-input-39-2f7369a95ba9> in <module>()
        6  for cl in cols:
        7  dv = cl.findAll('div')
  ----> 8  if 'col-xs-12 col-sm-6' in dv[0]['class']:
        9  flight, carrier, origin, date, scheduled, estimated, status = [c.text for c in dv]
       10 print(flight, carrier, origin, date, scheduled, estimated, status)
  IndexError: list index out of range

---

感谢您的贡献。

问题是

tr

下的第一个

td

没有

div

，这就是

dv

将返回空的原因。将代码更改为：

# import libraries
import requests
from bs4 import BeautifulSoup

# query the website and return the html to the variable ‘page’
page = requests.get("https://www.aucklandairport.co.nz/flights").text

soup = BeautifulSoup(page)
tbody = soup.find('tbody')
rows = tbody.findAll('tr',{'class':'flight-toggle'}) #find tr whose class = flight-toggle
for tr in rows:
    cols = tr.findAll('td',class_=lambda x: x != 'logo') # find td whose class!=logo (exclude the first td)
    dv0 = cols[0].find('div').findAll('div') #flight, carrier, origin under second td
    flight, carrier, origin = [c.text.strip() for c in dv0]
    dv1 = cols[1].find('div').findAll('div') #date, schedule under third td
    date, scheduled = [c.text.strip() for c in dv1]
    dv2 = cols[2].find('div').findAll('div') #estimated, statusunder fouth td
    estimated, status = [c.text.strip() for c in dv2[1:]] # exclude the first div 
    print(flight, carrier, origin, date, scheduled, estimated, status)

这将打印出：

(u'EK406', u'', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'QF8762', u'EK406', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'EK434', u'', u'Dubai / Brisbane', u'18 Nov', u'02:45pm', u'02:49pm', u'Processing')
...

---

问题是

tr

下的第一个

td

没有

div

，这就是

dv

将返回空的原因。将代码更改为：

# import libraries
import requests
from bs4 import BeautifulSoup

# query the website and return the html to the variable ‘page’
page = requests.get("https://www.aucklandairport.co.nz/flights").text

soup = BeautifulSoup(page)
tbody = soup.find('tbody')
rows = tbody.findAll('tr',{'class':'flight-toggle'}) #find tr whose class = flight-toggle
for tr in rows:
    cols = tr.findAll('td',class_=lambda x: x != 'logo') # find td whose class!=logo (exclude the first td)
    dv0 = cols[0].find('div').findAll('div') #flight, carrier, origin under second td
    flight, carrier, origin = [c.text.strip() for c in dv0]
    dv1 = cols[1].find('div').findAll('div') #date, schedule under third td
    date, scheduled = [c.text.strip() for c in dv1]
    dv2 = cols[2].find('div').findAll('div') #estimated, statusunder fouth td
    estimated, status = [c.text.strip() for c in dv2[1:]] # exclude the first div 
    print(flight, carrier, origin, date, scheduled, estimated, status)

这将打印出：

(u'EK406', u'', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'QF8762', u'EK406', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'EK434', u'', u'Dubai / Brisbane', u'18 Nov', u'02:45pm', u'02:49pm', u'Processing')
...

---

你可以用不同的方法剥同一个苹果。这里有另一种方法可以达到同样的效果

import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.aucklandairport.co.nz/flights")
soup = BeautifulSoup(response.text,"lxml")
table = soup.select(".flights-table")[0]
for items in table.select("tr.flight-toggle"):
    data = ' '.join([' '.join(item.text.split()) for item in items.select("td")])
    print(data.strip())

部分结果：

QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
AA7377 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
BA7421 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
CZ7575 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing

---

你可以用不同的方法剥同一个苹果。这里有另一种方法可以达到同样的效果

import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.aucklandairport.co.nz/flights")
soup = BeautifulSoup(response.text,"lxml")
table = soup.select(".flights-table")[0]
for items in table.select("tr.flight-toggle"):
    data = ' '.join([' '.join(item.text.split()) for item in items.select("td")])
    print(data.strip())

部分结果：

QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
AA7377 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
BA7421 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
CZ7575 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing