使用python（通过请求或其他方式）检索下拉列表选择的html表

我试图从以下站点获取历史天气数据：

我需要选择两个下拉菜单来加载任何特定日期的表格：一个用于月/年，另一个用于当天

我一直试图通过以下方式使用请求获取数据：

from bs4 import BeautifulSoup as soup
import requests
import pandas as pd

website = "https://www.timeanddate.com/weather/usa/boston/historic"
payload = {"month": '2018-06',"wt-his-select":"20180608"}
page_response = requests.get(website, data = payload, timeout=5).text
sp = soup(page_response,'lxml')
My_table = sp.find('table',id='wt-his')
table_rows = My_table.find_all('tr')
rows = []
for tr in table_rows:
    td = tr.find_all(['th','td'])
    rows.append([i.text for i in td])
pd.DataFrame(rows)

但是运行这段代码，我只得到今天的值，而不是2018/06/08的值-我做错了什么

谢谢

---

编辑：如果您不能对请求执行此操作，那么如何执行此操作？

此页面的数据通过

JSON

动态加载。更改日期时，您可以在网络选项卡中找到此

AJAX

请求。但是

JSON

的问题是它的格式不正确，因此当您使用

response.JSON（）

时，它会给您一个错误。因此，您必须将数据转换为正确的格式

import re
import requests

url = 'https://www.timeanddate.com/scripts/cityajax.php?n=usa/boston&mode=historic&hd=20180608&month=6&year=2018&json=1'

headers = {
'Accept': '*/*', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate, br',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36',
}
response  = requests.get(url, headers=headers)

raw_json = eval(response.text.replace(',{c:', '""",{c:').replace('{c:', '"""{c:')[::-1].replace(']', ']"""', 1)[::-1])
for i in raw_json:
    json_data = eval(i.replace('{c:', '{"c":').replace('{h:', '{"h":').replace('{s:', '{"s":').replace(',h:', ',"h":').replace('="', '=\'').replace('">', '\'>').replace('" ', '\' '))

    visibility = json_data['c'][8]['h'].replace(' ', '')
    barometer = json_data['c'][7]['h']
    humidity = json_data['c'][6]['h']
    wind = json_data['c'][4]['h'].replace('\/', '/')
    weather = json_data['c'][3]['h']
    temp = json_data['c'][2]['h'].replace(' ', '')
    time = re.findall(r'\d{2}:\d{2}', json_data['c'][0]['h'])[0]

    print(time, temp, weather, wind, humidity, barometer, visibility, end='    |   ')

输出：

00:54 14°C Overcast. 15 km/h 78% 1020 mbar 16km    |   01:54 14°C Overcast. 13 km/h 78% 1020 mbar 16km    |   02:54 13°C Passing clouds. 15 km/h 84% 1020 mbar 16km    |   03:54 13°C Passing clouds. 15 km/h 84% 1020 mbar 16km    |   04:54 13°C Clear. 15 km/h 87% 1020 mbar 16km    |   05:54 13°C Passing clouds. 20 km/h 90% 1020 mbar 16km    |   06:54 14°C Passing clouds. 13 km/h 81% 1020 mbar 16km    |   07:54 17°C Passing clouds. 11 km/h 73% 1020 mbar 16km    |   08:54 19°C Passing clouds. 7 km/h 66% 1020 mbar 16km    |   09:54 21°C Scattered clouds. 13 km/h 66% 1020 mbar 16km    |   10:54 24°C Passing clouds. 7 km/h 48% 1020 mbar 16km    |   11:54 25°C Passing clouds. 13 km/h 47% 1019 mbar 16km    |   12:54 26°C Broken clouds. 7 km/h 42% 1019 mbar 16km    |   13:54 27°C Partly sunny. 9 km/h 42% 1018 mbar 16km    |   15:54 21°C Partly sunny. 13 km/h 70% 1018 mbar 16km    |   16:54 22°C Broken clouds. 15 km/h 63% 1018 mbar 16km    |   17:54 21°C Scattered clouds. 11 km/h 64% 1017 mbar 16km    |   18:54 21°C Scattered clouds. 9 km/h 64% 1018 mbar 16km    |   19:54 22°C Scattered clouds. 11 km/h 57% 1018 mbar 16km    |   20:54 22°C Passing clouds. 6 km/h 59% 1018 mbar 16km    |   21:54 21°C Passing clouds. No wind 66% 1019 mbar 16km    |   22:54 22°C Passing clouds. 6 km/h 57% 1019 mbar 16km    |   23:54 21°C Clear. 6 km/h 59% 1018 mbar 16km

---

我需要选择两个下拉菜单来加载任何特定日期的表格：一个用于月/年，另一个用于当天。那么你可能无法使用请求，除非所有数据都已经在页面上，并且下拉列表只显示/隐藏它。你的石头！非常感谢。