Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/python/365.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Python Web抓取脚本返回并为空列表_Python_Web Scraping_Python Requests - Fatal编程技术网

Python Web抓取脚本返回并为空列表

Python Web抓取脚本返回并为空列表,python,web-scraping,python-requests,Python,Web Scraping,Python Requests,我正试图为一个测试网站编写我的第一个网页刮板。它涉及到登录,我遵循了如何处理这种情况的教程 import requests from lxml import html payload = { "email": "test_test@test.com", "password": "123qweasd", "_token": "3ow4dl7COwnRHa8a6nvNGp4eLkF3wQapT3otGXjR" } rs = requests.session() login_url = '

我正试图为一个测试网站编写我的第一个网页刮板。它涉及到登录,我遵循了如何处理这种情况的教程

import requests
from lxml import html



payload = {
"email": "test_test@test.com",
"password": "123qweasd",
"_token": "3ow4dl7COwnRHa8a6nvNGp4eLkF3wQapT3otGXjR"
 }

rs = requests.session()

login_url = 'https://cloud.webscraper.io/login'
log_page = rs.get(login_url)

tree = html.fromstring(log_page.content)
auth_token = list(set(tree.xpath("//input[@name='_token']/@value")))[0]

login = rs.post(login_url,data=payload, headers=dict(referer=login_url))

url = "https://cloud.webscraper.io/sitemaps"
result = rs.get(url, headers=dict(referer=url))

tree = html.fromstring(result.text)
sidebar_cat = tree.xpath('//*[@id="main-menu-inner"]/ul')

print(sidebar_cat)
我想让这个脚本从侧边栏中列出类别。似乎每次脚本都返回空列表。 电流输出为

"[] 
Process finished with exit code 0"

尝试打印“结果”,并在打印后暂时注释掉所有内容。这样,您就知道这是请求部分的错误,还是响应处理的问题。如果响应按预期打印,则对最后三行代码进行故障排除。如果没有,则对代码的请求部分进行故障排除

您提取了
\u令牌
值,但使用了硬编码值。尝试将提取的值传递给有效负载

import requests
from lxml import html

rs = requests.session()

login_url = 'https://cloud.webscraper.io/login'
log_page = rs.get(login_url)

tree = html.fromstring(log_page.content)
auth_token = tree.xpath("//input[@name='_token']/@value")[0]

payload = {
    "email": "test_test@test.com",
    "password": "123qweasd",
    "_token": auth_token
 }

login = rs.post(login_url,data=payload, headers=dict(referer=login_url))

url = "https://cloud.webscraper.io/sitemaps"
result = rs.get(url, headers=dict(referer=url))

tree = html.fromstring(result.text)
sidebar_cat = tree.xpath('//*[@id="main-menu-inner"]/ul')

print(sidebar_cat)