Python Web抓取脚本返回并为空列表
我正试图为一个测试网站编写我的第一个网页刮板。它涉及到登录,我遵循了如何处理这种情况的教程Python Web抓取脚本返回并为空列表,python,web-scraping,python-requests,Python,Web Scraping,Python Requests,我正试图为一个测试网站编写我的第一个网页刮板。它涉及到登录,我遵循了如何处理这种情况的教程 import requests from lxml import html payload = { "email": "test_test@test.com", "password": "123qweasd", "_token": "3ow4dl7COwnRHa8a6nvNGp4eLkF3wQapT3otGXjR" } rs = requests.session() login_url = '
import requests
from lxml import html
payload = {
"email": "test_test@test.com",
"password": "123qweasd",
"_token": "3ow4dl7COwnRHa8a6nvNGp4eLkF3wQapT3otGXjR"
}
rs = requests.session()
login_url = 'https://cloud.webscraper.io/login'
log_page = rs.get(login_url)
tree = html.fromstring(log_page.content)
auth_token = list(set(tree.xpath("//input[@name='_token']/@value")))[0]
login = rs.post(login_url,data=payload, headers=dict(referer=login_url))
url = "https://cloud.webscraper.io/sitemaps"
result = rs.get(url, headers=dict(referer=url))
tree = html.fromstring(result.text)
sidebar_cat = tree.xpath('//*[@id="main-menu-inner"]/ul')
print(sidebar_cat)
我想让这个脚本从侧边栏中列出类别。似乎每次脚本都返回空列表。
电流输出为
"[]
Process finished with exit code 0"
尝试打印“结果”,并在打印后暂时注释掉所有内容。这样,您就知道这是请求部分的错误,还是响应处理的问题。如果响应按预期打印,则对最后三行代码进行故障排除。如果没有,则对代码的请求部分进行故障排除 您提取了
\u令牌
值,但使用了硬编码值。尝试将提取的值传递给有效负载:
import requests
from lxml import html
rs = requests.session()
login_url = 'https://cloud.webscraper.io/login'
log_page = rs.get(login_url)
tree = html.fromstring(log_page.content)
auth_token = tree.xpath("//input[@name='_token']/@value")[0]
payload = {
"email": "test_test@test.com",
"password": "123qweasd",
"_token": auth_token
}
login = rs.post(login_url,data=payload, headers=dict(referer=login_url))
url = "https://cloud.webscraper.io/sitemaps"
result = rs.get(url, headers=dict(referer=url))
tree = html.fromstring(result.text)
sidebar_cat = tree.xpath('//*[@id="main-menu-inner"]/ul')
print(sidebar_cat)