Selenium Can'；t使用美丽的汤和硒来访问整个页面

我一直在尝试使用beautiful soup和selenium收集数据，但无法获得超过表中前20个结果（总共超过2000个）。我看到一些相关的问题和答案，建议尝试不同的解析器，所以我尝试了lxml、html.parser和html5lib，但没有一个有效。我还看到一些建议使用selenium和webdriver的答案，但这两种方法都无法获得整个页面。这是我现在的代码

import requests, time
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import urllib.request
#import pandas as pd
from bs4 import BeautifulSoup
import requests, time
import lxml

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('https://coinmarketcap.com/exchanges/uniswap-v2/')
#supposed to scroll to the bottom of the page
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

time.sleep(5)

page = driver.page_source
soup = BeautifulSoup(''.join(page), 'lxml')

pairs = soup.find_all('div', attrs = {"class" : "hmd6df-0 kCRNNr"})
data = [i.find_all('a')[0] for i in pairs]

---

任何帮助都会很棒。谢谢。

我想出了一个纯硒解决方案——下面的python脚本成功地从表中删除了所有数据。我只是以这样的方式打印作为一个示例，您将需要了解如何组织数据

更新：为了展开和刮取所有行，我滚动到列表的底部，然后单击“展开更多”按钮，这将再展开100行。我对它进行了足够的迭代，以遍历1850行数据

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import time

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('https://coinmarketcap.com/exchanges/uniswap-v2/')

total_height = int(driver.execute_script("return document.body.scrollHeight"))
for x in range(20):

    for i in range(1, total_height, 130):
        driver.execute_script("window.scrollTo(0, {});".format(i))
    if x == 0:
        driver.find_element_by_css_selector('div.cmc-cookie-policy-banner__close').click()

    driver.find_element_by_xpath('//button[text() = "Load More"]').click()
time.sleep(2)
first_column = driver.find_elements_by_css_selector('td.cmc-table__cell.cmc-table__cell--sticky.cmc-table__cell--sortable.cmc-table__cell--left.cmc-table__cell--sort-by__rank > div')
second_column = driver.find_elements_by_css_selector('div.cwwgik-0.bCvAgC')
third_column = driver.find_elements_by_css_selector('div.hmd6df-0.kCRNNr')
fourth_column = driver.find_elements_by_css_selector('div.cmc-table__column-market-pair-volume-24h')
fifth_column = driver.find_elements_by_css_selector('div.cmc-table__column-market-pair-volume-percent')
sixth_column = driver.find_elements_by_css_selector('td.cmc-table__cell.cmc-table__cell--sortable.cmc-table__cell--right.cmc-table__cell--sort-by__quote-usd-effective-liquidity > div')
seventh_column = driver.find_elements_by_css_selector('td.cmc-table__cell.cmc-table__cell--sortable.cmc-table__cell--right.cmc-table__cell--sort-by__fee-type > div')
eighth_column = driver.find_elements_by_css_selector('div.ghkc60-0.fLaXDt')
for i in range(len(second_column)):
    print(str(first_column[i].get_attribute("innerText")) + ' ' + str(second_column[i].get_attribute("innerText")) + '  ' + str(third_column[i].get_attribute("innerText")) + '  ' + str(fourth_column[i].get_attribute("innerText")) + '  ' + str(fifth_column[i].get_attribute("innerText")) + '  ' + str(sixth_column[i].get_attribute("innerText")) + '  ' + str(seventh_column[i].get_attribute("innerText")) + '  ' + str(eighth_column[i].get_attribute("innerText")))

输出：

1 USD Coin USDC/WETH $250,651,977 20.81% 935 Percentage Recently
2 Fei Protocol FEI/WETH $157,435,972 13.07% - Percentage Recently
3 WETH WETH/USDT $153,706,066 12.76% 915 Percentage Recently
4 Dai DAI/WETH $58,995,907 4.90% 850 Percentage Recently
5 Tendies TEND/WETH $41,429,102 3.44% - Percentage 778 hours ago
6 Wrapped Bitcoin WBTC/WETH $37,838,161 3.14% 901 Percentage Recently
7 SHIBA INU SHIB/WETH $30,416,126 2.53% 612 Percentage Recently
...

1770 yplutus  yPLT/WETH  $?  0.00%  -  Percentage  671 hours ago
1771 Deflect  DEFLCT/RFI  $?  0.00%  -  Percentage  96 days ago
1772 Blaze DeFi  BNFI/WETH  $?  0.00%  -  Percentage  781 hours ago
1773 Buy-Sell  BSE/WETH  $?  0.00%  -  Percentage  Recently
1774 PIRANHAS  $PIR/WETH  $?  0.00%  -  Percentage  Recently
1775 HLand Token  HLAND/USDT  $?  0.00%  -  Percentage  Recently
1776 Hub - Human Trust Protocol  HUB/WETH  $?  0.00%  -  Percentage  126 days ago
1777 WETH  WETH/YVS  $?  0.00%  -  Percentage  Recently
1778 Reflector.Finance  RFCTR/WETH  $?  0.00%  -  Percentage  Recently
1779 WETH  WETH/R34P  $?  0.00%  -  Percentage  Recently
1780 WETH  WETH/RFR  $?  0.00%  265  Percentage  Recently
1781 Golden Ratio Per Liquidity  GRPL/WETH  $?  0.00%  -  Percentage  Recently
1782 xETH-G  xETH-G/WETH  $?  0.00%  -  Percentage  75 days ago
1783 3XT TOKEN  3XT/WETH  $?  0.00%  -  Percentage  62 days ago
1784 Diffract Finance  DFR/WETH  $?  0.00%  -  Percentage  Recently
1785 Bitpower  BPP/WETH  $?  0.00%  -  Percentage  Recently
1786 IDL Token  IDL/ELYX  $?  0.00%  -  Percentage  77 days ago
1787 IDL Token  IDL/WETH  $?  0.00%  -  Percentage  81 days ago
1788 Dai  DAI/BCC  $?  0.00%  -  Percentage  42 days ago
1789 Stand Share  SAS/USDT  $?  0.00%  -  Percentage  123 days ago
1790 Definex  Dswap/USDT  $?  0.00%  -  Percentage  314 hours ago
1791 Vaultz  VAULTZ/WETH  $?  0.00%  -  Percentage  781 hours ago
1792 Fission Cash  FCX/WETH  $?  0.00%  -  Percentage  Recently
1793 DeltaHub Community  DHC/USDT  $?  0.00%  -  Percentage  821 hours ago
1794 Dai  DAI/DST  $?  0.00%  -  Percentage  111 days ago
1795 AGAr  AGAr/WETH  $?  0.00%  -  Percentage  311 hours ago
1796 Basis Cash  BAC/Mars  $?  0.00%  -  Percentage  809 hours ago
1797 Tether  USDT/NOW  $?  0.00%  -  Percentage  781 hours ago
1798 USD Coin  USDC/wCUSD  $?  0.00%  -  Percentage  Recently
1799 zzz.finance v2  ZZZV2/WETH  $?  0.00%  -  Percentage  781 hours ago
1800 Rigel Finance  RIGEL/WETH  $?  0.00%  -  Percentage  Recently
1801 Bitbot Protocol  BBP/WETH  $?  0.00%  -  Percentage  Recently
1802 HeroSwap  HERO/WETH  $?  0.00%  -  Percentage  108 days ago
1803 XUSD Stable  XUSD/LINK  $?  0.00%  -  Percentage  69 days ago
1804 XUSD Stable  XUSD/DAI  $?  0.00%  -  Percentage  96 days ago
1805 CURE Farm  CURE/WETH  $?  0.00%  -  Percentage  Recently
1806 WETH  WETH/stETH  $?  0.00%  518  Percentage  Recently
1807 Xstable.Protocol  XST/WETH  $?  0.00%  -  Percentage  Recently
1808 ZCore  WZCR/WETH  $?  0.00%  -  Percentage  94 days ago
1811 Wrapped BIND  wBIND/WETH  $?  0.00%  -  Percentage  Recently
1812 USDFreeLiquidity  USDFL/USDT  $?  0.00%  -  Percentage  Recently
1813 USDFreeLiquidity  USDFL/DAI  $?  0.00%  -  Percentage  113 hours ago
1814 USDFreeLiquidity  USDFL/USDN  $?  0.00%  -  Percentage  Recently
1815 WETH  WETH/MCX  $?  0.00%  -  Percentage  Recently
1816 Mythic Finance  MYTHIC/WETH  $?  0.00%  -  Percentage  781 hours ago
1817 Polkabase  PBASE/WETH  $?  0.00%  -  Percentage  Recently
1818 WETH  WETH/RAC  $?  0.00%  -  Percentage  Recently
1819 MiraQle  MQL/WETH  $?  0.00%  -  Percentage  68 days ago
1820 MiraQle  MQL/USDT  $?  0.00%  -  Percentage  68 days ago
1821 Parsiq Boost  PRQBOOST/WETH  $?  0.00%  -  Percentage  Recently
1822 WETH  WETH/PUX  $?  0.00%  -  Percentage  Recently
1823 Previse  PRVS/WETH  $?  0.00%  -  Percentage  Recently
1824 SIMBA Storage  SIMBA/USDT  $?  0.00%  -  Percentage  781 hours ago
1825 CryptoPing  PING/USDT  $?  0.00%  -  Percentage  Recently
1826 Wrapped Bitcoin  WBTC/INSTAR  $?  0.00%  -  Percentage  Recently
1827 Vow  VOW/WETH  $?  0.00%  -  Percentage  Recently
1828 Value Set Dollar  VSD/DAI  $?  0.00%  -  Percentage  471 hours ago
1829 Value Set Dollar  VSD/USDT  $?  0.00%  -  Percentage  110 hours ago
1830 Value Set Dollar  VSD/WETH  $?  0.00%  -  Percentage  44 days ago
1831 WETH  WETH/DEGENS  $?  0.00%  -  Percentage  Recently
1832 Xriba  XRA/WETH  $?  0.00%  -  Percentage  630 hours ago
1833 Tower token  TOWER/LYM  $?  0.00%  -  Percentage  469 hours ago
1834 Shadetech  SHD/WETH  $?  0.00%  -  Percentage  Recently
1835 Wrapped Bitcoin  WBTC/DGCL  $?  0.00%  -  Percentage  333 hours ago
1836 Rare Pepe  rPepe/WETH  $?  0.00%  -  Percentage  Recently
1837 xSigma  SIG/USDT  $?  0.00%  -  Percentage  52 days ago
1838 Dollar Protocol  USDf/USDC  $?  0.00%  -  Percentage  156 hours ago
1839 MYFinance  MYFI/WETH  $?  0.00%  -  Percentage  Recently
1840 Delta  DELTA/WETH  $?  0.00%  -  Percentage  Recently
1841 Kambria Yield Tuning Engine  KYTE/WETH  $?  0.00%  -  Percentage  Recently
1842 ClinTex CTi  CTI/WETH  $?  0.00%  -  Percentage  Recently
1843 BasenjiDAO  BSJ/WETH  $?  0.00%  -  Percentage  Recently
1844 EURxb  EURxb/USDT  $?  0.00%  -  Percentage  Recently
1845 Landbox  LAND/WETH  $?  0.00%  -  Percentage  442 hours ago
1846 Folder Protocol  FOL/WETH  $?  0.00%  -  Percentage  Recently
1847 Databroker  DTX/USDT  $?  0.00%  -  Percentage  Recently
1848 STATERA  STA/WSTA  $?  0.00%  -  Percentage  Recently
1849 Delta Exchange Token  DETO/WETH  $?  0.00%  -  Percentage  Recently
1850 Elongate Deluxe  ELongD/WETH  $?  0.00%  -  Percentage  Recently

---

我相信有2000多个条目，selenium只能检索到前400个？哦！我刚才看到按钮上的“加载更多”按钮。我将尝试编写一些代码来扩展整个内容并收集所有数据。非常感谢，这非常有用。这是我第一次浏览网页，我似乎不知道如何通过前20个左右的条目。@ElijahFox查看我答案的最新更新，以获得扩展和浏览表格中所有1850行的解决方案。非常感谢！