如何一次加载无限滚动中的所有条目以解析python中的HTML

我正试图从中提取信息。页面一次加载10项，我需要滚动加载所有条目（总共100项）。我能够解析HTML并获得前10个条目所需的信息，但我希望在解析HTML之前完全加载所有条目

我正在使用python、请求和BeautifulSoup。当页面加载前10个条目时，我解析页面的方式如下：

from bs4 import BeautifulSoup
import requests
s = requests.Session()
r = s.get('https://medium.com/top-100/december-2013')
page = BeautifulSoup(r.text)

但这只加载前10个条目。所以我查看了页面，得到了用于加载后续条目的AJAX请求，我得到了一个响应，但它是在一个时髦的JSON中，我宁愿使用HTML解析器，而不是解析JSON。代码如下：

from bs4 import BeautifulSoup
import requests
import json
s = requests.Session()
url = 'https://medium.com/top-100/december-2013/load-more'
payload = {"count":100}
r = s.post(url, data=payload)
page = json.loads(r.text[16:]) #skip some chars that throw json off

---

这给了我数据，但它是在一个非常长且复杂的JSON中，我更愿意加载页面上的所有数据并简单地解析HTML。此外，呈现的HTML提供了比JSON响应更多的信息（即作者的姓名而不是模糊的用户ID等）。有一个类似的问题，但没有相关的答案。理想情况下，我希望进行POST调用，然后请求HTML并对其进行解析，但我无法做到这一点。

这是您无法处理请求和美化组的，因为您要从中提取信息的页面在向下滚动时通过JS加载其余条目。您可以使用打开真正的浏览器并以编程方式传递页面向下按键事件来完成此操作。观看此视频以了解动作

下面是使用selenium提取所有100篇文章标题的脚本

import time

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser = webdriver.Chrome()

browser.get("https://medium.com/top-100/december-2013")
time.sleep(1)

elem = browser.find_element_by_tag_name("body")

no_of_pagedowns = 20

while no_of_pagedowns:
    elem.send_keys(Keys.PAGE_DOWN)
    time.sleep(0.2)
    no_of_pagedowns-=1

post_elems = browser.find_elements_by_class_name("post-item-title")

for post in post_elems:
    print post.text

输出：

When Your Mother Says She’s Fat
When “Life Hacking” Is Really White Privilege
As tendências culturais dos anos 2000 adiantadas pelo É o Tchan na década de 90
Coming Out as Biracial
Como ganhar discussões com seus parentes de direita neste Natal
How to save local bookstores in two easy steps
Welcome to Dinovember
How to Piss Off Your Barista
The boy whose brain could unlock autism
CrossFit’s Dirty Little Secret
Welcome to Medium
Here’s How the Military Wasted Your Money in 2013
Why I Wear Nail Polish
The day of High School I’ll never forget
7 Reasons Buffalonians Shouldn’t Hate Snow
Dear Guy Who Just Made My Burrito:
Is the Mona Lisa Priceless?
Please stop live tweeting people’s private conversations
Your Friends and Rapists
Eight things you can live without
The Value of Content
40 Ways To Make Life Simple Again
Manila-Beijing-Washington:
Things I Wish Someone Had Told Me When I Was Learning How to Code
Dear Ticketmaster,
Steve Jobs Danced To My Song
11 Things I Wish I Knew When I Started My Business
Bullish: Benevolent Sexism and “That Guy” Who Makes Everything Awkward
Advice to a College Music Student
Silver Gyninen joutui sotaan
Imagining the Post-Antibiotics Future
Which side are you on?
Put it away, junior. 
Casual Predation
The sad little iPhone commercial
How Node.js is Going to Replace JavaScript
Why you should have your heart broken into a million little pieces. 
How to Write Emails Like a CEO
Designing Products That Scale
How radioactive poison became the assassin’s weapon of choice
Why do people hate CrossFit?
We (Still) Need Feminism
10 Advanced Hearthstone Arena Tips
Let It Full-Bleed
What Medium Is For
How a Small Force of Finnish Ski Troops Fought Off a Massive Soviet Army
An Introvert’s Guide to Better Presentations
Mandela The Terrorist
Why You Should have a Messy Desk
Why I’m Not a TEDx Speaker
Fonts have feelings too
You Don’t Want Your Thanksgiving to Go Like This
What I’ve Learned in My First Month as a VC
Why Quantity Should be Your Priority
My Airbnb story
I Wanna Date You Like An Animal
The GIF Guide to Getting Paid
How We Discovered the Underground Chinese App Market
First Images of a Heart Injected with Liquid Metal 
Beyonce Broke the Music Business
“View mode” approach to responsive web design
Sometimes You Will Forget Your Mom Has Cancer
Darkness Ray Beams Invisibility From A Distance
Why Work As We Know It May Be Immoral
Staying Ahead of the Curve
The Geekiest Game Ever Made Has Been Released In Germany 
The Dirty Secret Behind the Salesforce $1M Hackathon
I’m a really good impostor
Mathematical Model of Zombie Epidemics Reveals Two Types of Living-Dead Infections
The Heartbreak Kid
200 Things
I’m Not Racist But—
Duel of the Superbattleships
23 and You
The Seattle NO
I’m a vaccine refuser. There, I said it. 
The Year We Broke Everything
How to make a DIY home alarm system with a raspberry pi and a webcam
Strike While the App is Hot
How to Fall In (and Out) of Love:
Why did Google make an ad for promoting “Search” in India where it has over 97% market share?
A Holiday Message From Jesus
Revealed: The Soviet Union’s $1 Billion ‘Psychotronic’ Arms Race with the US
Postmortem of a Venture-backed Startup
The 1.x Crore Myth
The “Getting Shit Done” Sleep Cycle 
Is the F-35 Joint Strike Fighter the New F-4?
Can the F-35 Win a Dogfight?
Responsive Photosets
Fightball: Millennials vs Boomers
The iconicity of “peaceful resistance”
How We Make Chocolate
Five Ships of the Chinese Navy You Really Ought to Know About
Glassholes and Black Rock City
Bad News for U.S. Warplane Pilots: Russia’s New Dogfighting Missile Can’t Miss
How Antisec Died
10 ways you’ll probably f**k up your startup
UPDATED: Finding the unjustly homeless, and teaching them to code.
Technology hasn’t Changed Us.
What I’ve learned from fatherhood 

---

您可以尝试以下方法：

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.support.ui import WebDriverWait

pause = 10
driver = webdriver.PhantomJS(executable_path='phantomjs.exe')
driver.get("your_url")
#This code will scroll down to the end
while True:
     try:
        # Action scroll down
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
     break
 except: 
     pass

---

或者，你也可以试试安德鲁·谢恩（Andrew Che in）给出的答案。这是一个3年前的问题，页面不再存在，但如果您尝试类似的滚动结果页面（例如，我搜索了“顶部”），您可以找到可以调用的端点以获取结果列表中的不同页面，在本例中是“”。

看起来解析JSON是最简单的方法go@zenpoy谢谢我已经编写了JSON解析器，它满足了我的需要。然而，我已经更新了我的问题，以强调呈现的HTML页面比JSON页面有更多有用的信息这一事实。我假设它在某个时候被翻译并注入HTML。我使用JSoup（使用java）来解析网页，我也面临着同样的问题，即只得到很少的搜索结果。实现滚动的指针，请点击@praveen！那看起来像是我要找的。然而，看起来elem.send_键（keys.PAGE_DOWN）并不像在视频中那样触发页面向下。可能是我的网速太慢了吗？编辑：看来要让按键在下一页工作，窗口必须在前面。我找不到自动对焦Chrome窗口的方法，所以当窗口弹出时，我似乎必须手动点击窗口。谢谢你的帮助@preveen！是 啊我忘记告诉你了。chrome无法聚焦于窗口，无论我试图聚焦于哪个元素。我必须手动点击窗口我必须下载chromedriver并指定路径…Browser=webdriver.Chrome（r'chromedriverpath'）伟大的答案@praveen！我还查看了你的YT视频并为其评分。我一直在寻找这样的解决方案，从类似的网站获取一些数据。不要链接其他人的答案来回答问题。如何获取将由bs4解析的html内容？