Fatal编程技术网
  • python-3.x/
  • Python 3.x 如何使用Ajax方法获取网站数据

Python 3.x 如何使用Ajax方法获取网站数据

python-3.x ajax

Python 3.x 如何使用Ajax方法获取网站数据,python-3.x,ajax,beautifulsoup,Python 3.x,Ajax,Beautifulsoup,我正在尝试获取此url中的所有商店位置 我目前使用这种方法 def getAllStoreLocation(): session = requests.Session() url = "https://www.walmart.com/store/finder?location=Pennsylvani&distance=100" try: stateStoreSoup = BeautifulSoup(session.get(url).text, 'html

我正在尝试获取此url中的所有商店位置

我目前使用这种方法

def getAllStoreLocation():

session = requests.Session()
url = "https://www.walmart.com/store/finder?location=Pennsylvani&distance=100"

try:
    stateStoreSoup = BeautifulSoup(session.get(url).text, 'html.parser')
    stateTagId = find_tag(stateStoreSoup)
    stateScript = stateStoreSoup.find("script", {"id":stateTagId})
    stateData = json.loads(stateScript.get_text(strip=True))

except Exception as e:
    print(e)
tagId函数

def find_tag(soup):
    script = soup.find('script', {'type': 'application/json', 'id':re.compile(r'^((?!tb-djs).)*$')})
return script['id']
这是可行的，但会产生太多不需要的数据。我想用这个答案中的方法

获取所有存储数据的列表

如何使用答案中的Ajax方法获取存储位置数据。请告诉我，我只熟悉beautifulsoup，不熟悉Ajax，所以我非常感谢您对
的良好解释
代码

我注意到您使用另一个python库请求给出的答案

如果尚未安装库，请尝试
pip安装请求

并尝试像这样使用这些URL来获取数据

我注意到您使用另一个python库请求给出的答案

如果尚未安装库，请尝试
pip安装请求

并尝试像这样使用这些URL来获取数据

导入请求
进口稀土
def主（url）：
r=请求。获取（url）
match=[x.group（1）代表re.finditer（r“address\”：（{.*？}）”，r.text中的x]
对于匹配的项目：
项目=评估（项目）
打印（“城市：{：
导入请求
进口稀土
def主（url）：
r=请求。获取（url）
match=[x.group（1）代表re.finditer（r“address\”：（{.*？}）”，r.text中的x]
对于匹配的项目：
项目=评估（项目）

印刷（“城市：{：您是如何找到URL的？您需要尝试使用一些工具分析网络请求。在Chorme中，您可以按f12，您将看到“开发工具”和“查找网络”选项卡，尝试在页面中更改参数，您将发现页面发出了一些ajax请求，当您更改参数时，显示您是否能够找到URL？您需要尝试分析网络请求通过一些工具。在Chorme中，您可以按f12键，您将看到“开发工具”和“查找网络”选项卡，尝试在页面中更改参数，您将发现页面发出了一些ajax请求，当您更改参数时，请您解释一下页面中的情况match@learner101我使用了
re
，这是我匹配
地址的
Regex
page source中的模式。好的，谢谢，如何将每个存储id包括在抓取中每个存储id都有以#开头的存储id，并存储在class=“result element store number”中@learner101下次你必须在你的问题中包含这样的细节，不管怎样，我已经更新了我的答案。请你解释一下这里面发生了什么match@learner101我使用了
re
，这是
Regex
，我在页面源代码中匹配了
address
模式。好的，谢谢你，每个存储如何id将包含在抓取中每个存储区都有以#开头的storeid，并存储在class=“result element store number”@learner101中。下次您必须将此类详细信息包含在您的问题中，以便查询。无论如何，我已经更新了我的答案。

City: Montoursville                 , Address: 1015 N Loyalsock Ave
City: Mill Hall                     , Address: 167 Hogan Blvd
City: Lewisburg                     , Address: 120 Ajk Blvd
City: Selinsgrove                   , Address: 980 N Susquehanna Trl
City: Bloomsburg                    , Address: 100 Lunger Dr
City: Mansfield                     , Address: 1169 S Main St
City: State College                 , Address: 373 Benner Pike
City: State College                 , Address: 1665 N Atherton St
City: Coal Township                 , Address: 9300 State Route 61
City: Lewistown                     , Address: 10180 Us Highway 522 S
City: Elizabethville                , Address: 200 Kocher Ln
City: Saint Clair                   , Address: 500 Terry Rich Blvd
City: Sayre                         , Address: 1887 Elmira St
City: Hazle Township                , Address: 761 Airport Rd
City: Clearfield                    , Address: 100 Supercenter Dr
City: Painted Post                  , Address: 3217 Silverback Ln
City: Huntingdon                    , Address: 6716 Towne Center Blvd
City: Mechanicsburg                 , Address: 6520 Carlisle Pike Ste 550
City: Tamaqua                       , Address: 35 Plaza Dr
City: Horseheads                    , Address: 1400 County Rd 64
City: Tunkhannock                   , Address: 808 Hunter Hwy
City: Harrisburg                    , Address: 6535 Grayson Rd
City: Camp Hill                     , Address: 3400 Hartzdale Drive
City: Palmyra                       , Address: 100 No Londonderry Square
City: Wilkes Barre                  , Address: 2150 Wilkes Barre Twnsp Mktpl
City: Carlisle                      , Address: 60 Noble Blvd
City: Saint Marys                   , Address: 1102 Million Dollar Hwy
City: Lebanon                       , Address: 1355 E Lehman St
City: Pittston                      , Address: 390 Route 315 Hwy
City: Etters                        , Address: 50 Newberry Pkwy
City: Hamburg                       , Address: 1800 Tilden Ridge
City: Taylor                        , Address: 1325 Main
City: Du Bois                       , Address: 20 Industrial Dr
City: Shippensburg                  , Address: 100 S Conestoga Dr
City: Lehighton                     , Address: 1731 Blakeslee Boulevard Dr E
City: Altoona                       , Address: 2600 Plank Road Commons
City: Dickson City                  , Address: 900 Commerce Blvd
City: Hornell                       , Address: 1000 State Route 36
City: Watkins Glen                  , Address: 515 E 4th St
City: Temple                        , Address: 5370 Allentown Pike
City: Duncansville                  , Address: 200 Commerce Dr
City: Vestal                        , Address: 2405 Vestal Pkwy E
City: Wyomissing                    , Address: 1135 Berkshire Blvd
City: York                          , Address: 1000 Town Center Drive
City: York                          , Address: 2801 E Market St Bldg B
City: Olean                         , Address: 1869 Plaza Dr
City: Ephrata                       , Address: 890 E Main St
City: Johnson City                  , Address: 2 Gannett Dr
City: Lancaster                     , Address: 2030 Fruitville Pike
City: Bradford                      , Address: 50 Foster Brook Blvd



import requests
import re


def main(url):
    r = requests.get(url)
    match = [x.group(1, 2) for x in re.finditer(
        r',"id":(.*?),.*?address\":({.*?})', r.text)]
    for a, b in match:
        item = eval(b)
        idf = a
        print("ID: {}, City: {:<30}, Address: {}".format(idf,
                                                         item['city'], item['address']))


main("https://www.walmart.com/store/finder?location=Pennsylvani&distance=100")



ID: 1887, City: Montoursville                 , Address: 1015 N Loyalsock Ave
ID: 2528, City: Mill Hall                     , Address: 167 Hogan Blvd
ID: 1644, City: Lewisburg                     , Address: 120 Ajk Blvd
ID: 2185, City: Selinsgrove                   , Address: 980 N Susquehanna Trl
ID: 1794, City: Bloomsburg                    , Address: 100 Lunger Dr
ID: 1945, City: Mansfield                     , Address: 1169 S Main St
ID: 2230, City: State College                 , Address: 373 Benner Pike
ID: 1640, City: State College                 , Address: 1665 N Atherton St
ID: 2481, City: Coal Township                 , Address: 9300 State Route 61
ID: 1607, City: Lewistown                     , Address: 10180 Us Highway 522 S
ID: 3412, City: Elizabethville                , Address: 200 Kocher Ln
ID: 2535, City: Saint Clair                   , Address: 500 Terry Rich Blvd
ID: 2208, City: Sayre                         , Address: 1887 Elmira St
ID: 2255, City: Hazle Township                , Address: 761 Airport Rd
ID: 2129, City: Clearfield                    , Address: 100 Supercenter Dr
ID: 2992, City: Painted Post                  , Address: 3217 Silverback Ln
ID: 5470, City: Huntingdon                    , Address: 6716 Towne Center Blvd
ID: 1886, City: Mechanicsburg                 , Address: 6520 Carlisle Pike Ste 550
ID: 3634, City: Tamaqua                       , Address: 35 Plaza Dr
ID: 1976, City: Horseheads                    , Address: 1400 County Rd 64
ID: 2024, City: Tunkhannock                   , Address: 808 Hunter Hwy
ID: 1591, City: Harrisburg                    , Address: 6535 Grayson Rd
ID: 5888, City: Camp Hill                     , Address: 3400 Hartzdale Drive
ID: 2888, City: Palmyra                       , Address: 100 No Londonderry Square       
ID: 1623, City: Wilkes Barre                  , Address: 2150 Wilkes Barre Twnsp Mktpl   
ID: 2574, City: Carlisle                      , Address: 60 Noble Blvd
ID: 1792, City: Saint Marys                   , Address: 1102 Million Dollar Hwy
ID: 2023, City: Lebanon                       , Address: 1355 E Lehman St
ID: 2543, City: Pittston                      , Address: 390 Route 315 Hwy
ID: 4404, City: Etters                        , Address: 50 Newberry Pkwy
ID: 4612, City: Hamburg                       , Address: 1800 Tilden Ridge
ID: 4276, City: Taylor                        , Address: 1325 Main
ID: 1769, City: Du Bois                       , Address: 20 Industrial Dr
ID: 5358, City: Shippensburg                  , Address: 100 S Conestoga Dr
ID: 2169, City: Lehighton                     , Address: 1731 Blakeslee Boulevard Dr E   
ID: 2049, City: Altoona                       , Address: 2600 Plank Road Commons
ID: 1884, City: Dickson City                  , Address: 900 Commerce Blvd
ID: 2326, City: Hornell                       , Address: 1000 State Route 36
ID: 3221, City: Watkins Glen                  , Address: 515 E 4th St
ID: 2614, City: Temple                        , Address: 5370 Allentown Pike
ID: 4348, City: Duncansville                  , Address: 200 Commerce Dr
ID: 1835, City: Vestal                        , Address: 2405 Vestal Pkwy E
ID: 1670, City: Wyomissing                    , Address: 1135 Berkshire Blvd
ID: 2205, City: York                          , Address: 1000 Town Center Drive
ID: 1529, City: York                          , Address: 2801 E Market St Bldg B
ID: 2159, City: Olean                         , Address: 1869 Plaza Dr
ID: 2340, City: Ephrata                       , Address: 890 E Main St
ID: 5732, City: Johnson City                  , Address: 2 Gannett Dr
ID: 5200, City: Lancaster                     , Address: 2030 Fruitville Pike
ID: 3514, City: Bradford                      , Address: 50 Foster Brook Blvd