Python 我怎样才能找到有靓汤的残疾人？_Python_Css_Web Scraping_Beautifulsoup_Tags

Python 我怎样才能找到有靓汤的残疾人？

python css web-scraping tags

Python 我怎样才能找到有靓汤的残疾人？,python,css,web-scraping,beautifulsoup,tags,Python,Css,Web Scraping,Beautifulsoup,Tags,我在网页上找不到禁用：及我正在解析来自woot.com的信息此页面使用JavaScript设置禁用的元素，但请求/美化组无法运行JavaScript。您可能需要控制可以运行JavaScript的web浏览器或者您可以尝试获取JavaScript（AJAX/XHR）用于从服务器获取数据的url 但是这个页面没有使用AJAX/XHR来加载信息。在HTML中，我看到带有 var offerItems = [...] 他们可能有所有的信息此脚本获取所有第6行，并截断第4行，其中包含变量of

我在网页上找不到禁用：

及

我正在解析来自woot.com的信息

此页面使用

JavaScript

设置禁用的元素，但

请求

美化组

无法运行

JavaScript

。您可能需要控制可以运行

JavaScript

的web浏览器

或者您可以尝试获取

JavaScript

（

AJAX

XHR

）用于从服务器获取数据的url

但是这个页面没有使用

AJAX/XHR

来加载信息。在HTML中，我看到带有

var offerItems = [...]

他们可能有所有的信息

此脚本获取所有第6行

，并截断第4行，其中包含

变量offerItems=

（我手动找到此位置检查代码）

接下来，它只切断

[]

中包含数据的文本，并使用模块

json

将其转换为Python的列表/字典

稍后，它可以很容易地从该列表/字典中获取值

它只有信息

大小

和

数量

，因此您必须添加代码来计算缺少的大小，以便禁用它们。我跳过了这一部分——我甚至都没有试着计算它

顺便说一句：一些代码我写了很多行，使它更可读，但你可以写在一行

import requests
from bs4 import BeautifulSoup
import json

response = requests.get("https://sport.woot.com/offers/asics-mens-clearance-calculator?ref=w_cnt_lnd_cat_sprt_18_1")
soup = BeautifulSoup(response.content, 'lxml')

all_scripts = soup.find_all('script')
#for i, item in enumerate(all_scripts):
#    print('---', i, '---')
#    print(item.text[:200])

text = all_scripts[5].text
lines = text.split('\n')
line = lines[3].strip()
line = line[len("var offerItems = "):-1]

data = json.loads(line)

#sizes = ["6.5", "8", "8.5", "9.5", "10", "10.5", "11", "12", "13", "14", "12 Wide US", "13 (4E)"]

print(data[0].keys())
for item in data:
    print('------------------------------------')
    print('Key:', item['Key'])
    print('Quantity:', item['Quantity'])
    for x in item['Attributes']:
        print('> ', x['Key'], ':', x['Value'])
    #for key, value in item.items():
    #    print('>', key, ':', value)

结果:

dict_keys(['Id', 'Asin', 'AmazonPrice', 'Attributes', 'FormattedDiscount', 'FormattedListPrice', 'FormattedSalePrice', 'FormattedWarrantyPrice', 'Key', 'ListPrice', 'Quantity', 'SalePrice', 'SKU', 'TotalReviews', 'WarrantyPrice', 'WarrantyTerm', 'WarrantyUrl'])

------------------------------------
Key: 6.5|Gel-Nimbus 19 - Black/Onyx/Silver
Quantity: 10
>  Size : 6.5
>  Model : Gel-Nimbus 19 - Black/Onyx/Silver
------------------------------------
Key: 8|Gel-Excite 4 - Silver/Black/Imperial
Quantity: 6
>  Size : 8
>  Model : Gel-Excite 4 - Silver/Black/Imperial
------------------------------------
Key: 8|GT-2000 7 - Illusion Blue/Black
Quantity: 5
>  Size : 8
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 8|GT-2000 6 - Everglade/ Black
Quantity: 10
>  Size : 8
>  Model : GT-2000 6 - Everglade/ Black
------------------------------------
Key: 8.5|Frequent Trail - Black/ Birch
Quantity: 0
>  Size : 8.5
>  Model : Frequent Trail - Black/ Birch
------------------------------------
Key: 8.5|GT-2000 7 - Illusion Blue/Black
Quantity: 0
>  Size : 8.5
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 9.5|Gel-Venture 6 - Frost Grey/ Phantom/ Black
Quantity: 0
>  Size : 9.5
>  Model : Gel-Venture 6 - Frost Grey/ Phantom/ Black
------------------------------------
Key: 9.5|GT-2000 7 - Illusion Blue/Black
Quantity: 3
>  Size : 9.5
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 9.5|Gel-Nimbus 20 - Island Blue/White/Black
Quantity: 10
>  Size : 9.5
>  Model : Gel-Nimbus 20 - Island Blue/White/Black
------------------------------------
Key: 10|GT-2000 6 - Everglade/ Black
Quantity: 4
>  Size : 10
>  Model : GT-2000 6 - Everglade/ Black
------------------------------------
Key: 9.5|GT-2000 7 - Black/White
Quantity: 0
>  Size : 9.5
>  Model : GT-2000 7 - Black/White
------------------------------------
Key: 9.5|GT-2000 6 - Dark Grey/Black/Fiery Red
Quantity: 9
>  Size : 9.5
>  Model : GT-2000 6 - Dark Grey/Black/Fiery Red
------------------------------------
Key: 10|GT-2000 8 - Piedmont Grey/Black
Quantity: 3
>  Size : 10
>  Model : GT-2000 8 - Piedmont Grey/Black
------------------------------------
Key: 10|Gel-Contend 4 - Carbon/ Classic Red/ Black
Quantity: 6
>  Size : 10
>  Model : Gel-Contend 4 - Carbon/ Classic Red/ Black
------------------------------------
Key: 10.5|Gel-Excite 4 - Silver/Black/Imperial
Quantity: 10
>  Size : 10.5
>  Model : Gel-Excite 4 - Silver/Black/Imperial
------------------------------------
Key: 10.5|Gel-Excite 4 - Carbon/ Silver/ Black
Quantity: 0
>  Size : 10.5
>  Model : Gel-Excite 4 - Carbon/ Silver/ Black
------------------------------------
Key: 10.5|Gel-Nimbus 21 - Black/Classic Red
Quantity: 1
>  Size : 10.5
>  Model : Gel-Nimbus 21 - Black/Classic Red
------------------------------------
Key: 11|Gel-Nimbus 20 - Island Blue/White/Black
Quantity: 5
>  Size : 11
>  Model : Gel-Nimbus 20 - Island Blue/White/Black
------------------------------------
Key: 12|Gel-Nimbus 19 - Glacier Grey/Silver/White
Quantity: 0
>  Size : 12
>  Model : Gel-Nimbus 19 - Glacier Grey/Silver/White
------------------------------------
Key: 13|GT-1000 7 - Grand Shark/ Peacoat
Quantity: 0
>  Size : 13
>  Model : GT-1000 7 - Grand Shark/ Peacoat
------------------------------------
Key: 14|Gel-Cumulus 20 - Race Blue/ Peacoat
Quantity: 1
>  Size : 14
>  Model : Gel-Cumulus 20 - Race Blue/ Peacoat
------------------------------------
Key: 14|Gel-Nimbus 20 - Indigo blue/fiery red
Quantity: 0
>  Size : 14
>  Model : Gel-Nimbus 20 - Indigo blue/fiery red
------------------------------------
Key: 12 Wide US|Gel-Nimbus 20 - Black/White/Carbon
Quantity: 3
>  Size : 12 Wide US
>  Model : Gel-Nimbus 20 - Black/White/Carbon
------------------------------------
Key: Gel-Venture 6 - Frost Grey/ Phantom/ Black|13 (4E)
Quantity: 0
>  Model : Gel-Venture 6 - Frost Grey/ Phantom/ Black
>  Size : 13 (4E)

请提供。此页面使用JavaScript设置为禁用-但

请求

美化组

无法运行JavaScript。您可能需要控制可以运行JavaScript的web浏览器。顺便说一句：如果您在web浏览器中关闭JavaScript并加载url，那么您将看到没有禁用的选项。这就是BeautifulSoup.BTW的功能：此页面不使用AJAX/XHR加载信息，但在HTML中，我看到的脚本可能包含所有信息。

var offerItems = [...]

import requests
from bs4 import BeautifulSoup
import json

response = requests.get("https://sport.woot.com/offers/asics-mens-clearance-calculator?ref=w_cnt_lnd_cat_sprt_18_1")
soup = BeautifulSoup(response.content, 'lxml')

all_scripts = soup.find_all('script')
#for i, item in enumerate(all_scripts):
#    print('---', i, '---')
#    print(item.text[:200])

text = all_scripts[5].text
lines = text.split('\n')
line = lines[3].strip()
line = line[len("var offerItems = "):-1]

data = json.loads(line)

#sizes = ["6.5", "8", "8.5", "9.5", "10", "10.5", "11", "12", "13", "14", "12 Wide US", "13 (4E)"]

print(data[0].keys())
for item in data:
    print('------------------------------------')
    print('Key:', item['Key'])
    print('Quantity:', item['Quantity'])
    for x in item['Attributes']:
        print('> ', x['Key'], ':', x['Value'])
    #for key, value in item.items():
    #    print('>', key, ':', value)

dict_keys(['Id', 'Asin', 'AmazonPrice', 'Attributes', 'FormattedDiscount', 'FormattedListPrice', 'FormattedSalePrice', 'FormattedWarrantyPrice', 'Key', 'ListPrice', 'Quantity', 'SalePrice', 'SKU', 'TotalReviews', 'WarrantyPrice', 'WarrantyTerm', 'WarrantyUrl'])

------------------------------------
Key: 6.5|Gel-Nimbus 19 - Black/Onyx/Silver
Quantity: 10
>  Size : 6.5
>  Model : Gel-Nimbus 19 - Black/Onyx/Silver
------------------------------------
Key: 8|Gel-Excite 4 - Silver/Black/Imperial
Quantity: 6
>  Size : 8
>  Model : Gel-Excite 4 - Silver/Black/Imperial
------------------------------------
Key: 8|GT-2000 7 - Illusion Blue/Black
Quantity: 5
>  Size : 8
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 8|GT-2000 6 - Everglade/ Black
Quantity: 10
>  Size : 8
>  Model : GT-2000 6 - Everglade/ Black
------------------------------------
Key: 8.5|Frequent Trail - Black/ Birch
Quantity: 0
>  Size : 8.5
>  Model : Frequent Trail - Black/ Birch
------------------------------------
Key: 8.5|GT-2000 7 - Illusion Blue/Black
Quantity: 0
>  Size : 8.5
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 9.5|Gel-Venture 6 - Frost Grey/ Phantom/ Black
Quantity: 0
>  Size : 9.5
>  Model : Gel-Venture 6 - Frost Grey/ Phantom/ Black
------------------------------------
Key: 9.5|GT-2000 7 - Illusion Blue/Black
Quantity: 3
>  Size : 9.5
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 9.5|Gel-Nimbus 20 - Island Blue/White/Black
Quantity: 10
>  Size : 9.5
>  Model : Gel-Nimbus 20 - Island Blue/White/Black
------------------------------------
Key: 10|GT-2000 6 - Everglade/ Black
Quantity: 4
>  Size : 10
>  Model : GT-2000 6 - Everglade/ Black
------------------------------------
Key: 9.5|GT-2000 7 - Black/White
Quantity: 0
>  Size : 9.5
>  Model : GT-2000 7 - Black/White
------------------------------------
Key: 9.5|GT-2000 6 - Dark Grey/Black/Fiery Red
Quantity: 9
>  Size : 9.5
>  Model : GT-2000 6 - Dark Grey/Black/Fiery Red
------------------------------------
Key: 10|GT-2000 8 - Piedmont Grey/Black
Quantity: 3
>  Size : 10
>  Model : GT-2000 8 - Piedmont Grey/Black
------------------------------------
Key: 10|Gel-Contend 4 - Carbon/ Classic Red/ Black
Quantity: 6
>  Size : 10
>  Model : Gel-Contend 4 - Carbon/ Classic Red/ Black
------------------------------------
Key: 10.5|Gel-Excite 4 - Silver/Black/Imperial
Quantity: 10
>  Size : 10.5
>  Model : Gel-Excite 4 - Silver/Black/Imperial
------------------------------------
Key: 10.5|Gel-Excite 4 - Carbon/ Silver/ Black
Quantity: 0
>  Size : 10.5
>  Model : Gel-Excite 4 - Carbon/ Silver/ Black
------------------------------------
Key: 10.5|Gel-Nimbus 21 - Black/Classic Red
Quantity: 1
>  Size : 10.5
>  Model : Gel-Nimbus 21 - Black/Classic Red
------------------------------------
Key: 11|Gel-Nimbus 20 - Island Blue/White/Black
Quantity: 5
>  Size : 11
>  Model : Gel-Nimbus 20 - Island Blue/White/Black
------------------------------------
Key: 12|Gel-Nimbus 19 - Glacier Grey/Silver/White
Quantity: 0
>  Size : 12
>  Model : Gel-Nimbus 19 - Glacier Grey/Silver/White
------------------------------------
Key: 13|GT-1000 7 - Grand Shark/ Peacoat
Quantity: 0
>  Size : 13
>  Model : GT-1000 7 - Grand Shark/ Peacoat
------------------------------------
Key: 14|Gel-Cumulus 20 - Race Blue/ Peacoat
Quantity: 1
>  Size : 14
>  Model : Gel-Cumulus 20 - Race Blue/ Peacoat
------------------------------------
Key: 14|Gel-Nimbus 20 - Indigo blue/fiery red
Quantity: 0
>  Size : 14
>  Model : Gel-Nimbus 20 - Indigo blue/fiery red
------------------------------------
Key: 12 Wide US|Gel-Nimbus 20 - Black/White/Carbon
Quantity: 3
>  Size : 12 Wide US
>  Model : Gel-Nimbus 20 - Black/White/Carbon
------------------------------------
Key: Gel-Venture 6 - Frost Grey/ Phantom/ Black|13 (4E)
Quantity: 0
>  Model : Gel-Venture 6 - Frost Grey/ Phantom/ Black
>  Size : 13 (4E)