Scrapy Python Web抓取JSON
我正在努力找出如何使用Scrapy Python刮取JSON响应。我能够在同一站点的不同页面上成功地获取JSON。我将感谢任何帮助 我如何在“tournamentGroup”(即id、姓名)以及年份、头衔等中获取值 部分代码:Scrapy Python Web抓取JSON,python,json,web,web-scraping,scrapy,Python,Json,Web,Web Scraping,Scrapy,我正在努力找出如何使用Scrapy Python刮取JSON响应。我能够在同一站点的不同页面上成功地获取JSON。我将感谢任何帮助 我如何在“tournamentGroup”(即id、姓名)以及年份、头衔等中获取值 部分代码: start_url = 'https://api.wtatennis.com/tennis/tournaments/?page=0&pageSize=100&excludeLevels=ITF&from=2020-09-01&to=2020
start_url = 'https://api.wtatennis.com/tennis/tournaments/?page=0&pageSize=100&excludeLevels=ITF&from=2020-09-01&to=2020-09-30'
with urllib.request.urlopen(start_url) as start_url:
json_obj = start_url.read()
rank_list = json.loads(json_obj)
for item in rank_list:
rank_data = []
tourney_id = item['content']['id']
tourney_year = item['year']
rank_data = [tourney_id, tourney_year]
cur.execute("""insert into wta_rankings(tourney_id, tourney_year)
values(%s, %s)
ON CONFLICT DO NOTHING"""
,(rank_data))
conn.commit()
cur.close()
JSON:
{
"pageInfo":{
"page":0,
"numPages":0,
"pageSize":100,
"numEntries":10
},
"content":[
{
"tournamentGroup":{
"id":2023,
"name":"Prague 125K",
"level":"125K",
"metadata":null
},
"year":2020,
"title":"Prague Open",
"startDate":"2020-08-29",
"endDate":"2020-09-06",
"surface":"Clay",
"inOutdoor":"O",
"city":"PRAGUE",
"country":"Czech Republic",
"singlesDrawSize":128,
"doublesDrawSize":32,
"prizeMoney":3125000,
"prizeMoneyCurrency":"USD",
"liveScoringId":"2023"
},
URL示例:尝试以下操作:
import requests
url = "https://api.wtatennis.com/tennis/tournaments/?page=0&pageSize=100&excludeLevels=ITF&from=2020-09-01&to=2020-09-30"
response = requests.get(url).json()
for item in response["content"]:
print(f"{item['tournamentGroup']['name']} - {item['year']} - {item['title']}")
这将为您提供(这只是一个示例,您可以获得您想要的任何字段):
如果您难以通过JSON进行“导航”,只需将响应内容复制到联机文档中,单击
扳手
图标进行修复,然后单击格式化/美化
非常感谢!这回答了我的问题,让我开心!我很高兴它做到了。如果你觉得答案有用,请接受它。
Prague 125K - 2020 - Prague Open
US OPEN - 2020 - US Open - New York, United States, NY
WARSAW - 2020 - BNP Paribas Warsaw Open - Warsaw, Poland
ISTANBUL - 2020 - TEB BNP Paribas Tennis Championship Istanbul - Istanbul, Turkey
MADRID - 2020 - Mutua Madrid Open - Madrid, Spain
HIROSHIMA - 2020 - Hana-cupid Japan Women's Open - Hiroshima, Japan
ROME - 2020 - Internazionali BNL d'Italia - Rome, Italy
STRASBOURG - 2020 - Internationaux de Strasbourg - Strasbourg, France
ROLAND GARROS - 2020 - Roland Garros - Paris, France
TASHKENT - 2020 - Tashkent Open - Tashkent, Uzbekistan