Html 使用Beauty Soup缺少网站文本_Html_Web Scraping_Beautifulsoup

Html 使用Beauty Soup缺少网站文本

html web-scraping

Html 使用Beauty Soup缺少网站文本,html,web-scraping,beautifulsoup,Html,Web Scraping,Beautifulsoup,我正在尝试使用美丽的汤来获得旅行时间： https://betaplanner.trimet.org/map/#/?fromPlace=10255%20SW%20CANYON%20RD%3A%3A45.493227%2C-122.782138&toPlace=4809%20N%20KERBY%20AVE%3A%3A45.557817%2C-122.673216&date=2019-08-14&time=16%3A56&arriveBy=false&mode

我正在尝试使用美丽的汤来获得旅行时间：

https://betaplanner.trimet.org/map/#/?fromPlace=10255%20SW%20CANYON%20RD%3A%3A45.493227%2C-122.782138&toPlace=4809%20N%20KERBY%20AVE%3A%3A45.557817%2C-122.673216&date=2019-08-14&time=16%3A56&arriveBy=false&mode=BUS%2CTRAM%2CRAIL%2CGONDOLA%2CCAR_RENT&showIntermediateStops=true&optimize=QUICK&ignoreRealtimeUpdates=true&companies=NaN&minTransitDistance=50%25&searchTimeout=10000&onlyTransitTrips=true&ui_activeItinerary=0

这个问题似乎与javascript呈现有关，但是，当我尝试执行html请求时，出现了一个错误

对于Bs4：

 r = session.get(*linkfromabove*)
    soup = BeautifulSoup(r.content, 'html.parser')
    soup.select

对于HTML_请求：

import requests
from requests_html import HTMLSession
session = HTMLSession()
r = session.get(linkfromabove)
r.html.render(wait=8, sleep=8)

使用BS4的实际结果：

<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="ie=edge" http-equiv="x-ua-compatible"/>
<title>TriMet MOD OTP Demo</title>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="https://fonts.googleapis.com/css? 
family=Hind:300,400,500,600,700" 
rel="stylesheet"/>
<link href="main.css" rel="stylesheet"/></head>
<body>
<div id="main"></div>
<script src="bundle.js" type="text/javascript"></script></body>
</html>

若您检查网络选项卡，您将看到一个返回json和您的信息的请求

import requests

r = requests.get('https://maps.trimet.org/otp_mod/plan?fromPlace=10255%20SW%20CANYON%20RD%3A%3A45.493227%2C-122.782138&toPlace=4809%20N%20KERBY%20AVE%3A%3A45.557817%2C-122.673216&date=2019-08-14&time=16%3A56&arriveBy=false&mode=BUS%2CTRAM%2CRAIL%2CGONDOLA%2CCAR_RENT%2CWALK&showIntermediateStops=true&optimize=QUICK&ignoreRealtimeUpdates=true&companies=NaN&minTransitDistance=50%25&searchTimeout=10000&onlyTransitTrips=true').json()

bs4和请求都不进行javascript呈现。除了硒，我什么都不知道。你照原样复制了吗？似乎仍然适用于我。请查看在使用F5刷新页面时是否可以在“网络”选项卡中找到相同的url。感谢您的快速响应。当我尝试这样做时，我使用原始链接得到了以下错误代码：jsondecoderror：期望值：第1行第1列（char 0）。你在哪里找到这个新链接的？在“网络”选项卡中。我在网络选项卡F12中打开了原始url和viewd，然后按F5刷新页面。我在网络流量中搜索了一个我希望在结果中找到的值。请考虑接受一个答案。看到和

import requests

r = requests.get('https://maps.trimet.org/otp_mod/plan?fromPlace=10255%20SW%20CANYON%20RD%3A%3A45.493227%2C-122.782138&toPlace=4809%20N%20KERBY%20AVE%3A%3A45.557817%2C-122.673216&date=2019-08-14&time=16%3A56&arriveBy=false&mode=BUS%2CTRAM%2CRAIL%2CGONDOLA%2CCAR_RENT%2CWALK&showIntermediateStops=true&optimize=QUICK&ignoreRealtimeUpdates=true&companies=NaN&minTransitDistance=50%25&searchTimeout=10000&onlyTransitTrips=true').json()