自动化Python网络爬虫-如何始终防止原始输入?
我一直在尝试创建一个pythonweb爬虫,它可以查找网页、读取链接列表、返回预先指定位置的链接,并执行一定次数的操作(由count变量定义)。我的问题是,我无法找到一种方法来自动化这个过程,我必须不断地输入代码找到的链接 这是我的密码: 第一个URL是 计数_1等于7 这个位置等于8自动化Python网络爬虫-如何始终防止原始输入?,python,while-loop,web-scraping,web-crawler,Python,While Loop,Web Scraping,Web Crawler,我一直在尝试创建一个pythonweb爬虫,它可以查找网页、读取链接列表、返回预先指定位置的链接,并执行一定次数的操作(由count变量定义)。我的问题是,我无法找到一种方法来自动化这个过程,我必须不断地输入代码找到的链接 这是我的密码: 第一个URL是 计数_1等于7 这个位置等于8 ##Here is my code: import urllib from bs4 import BeautifulSoup count_1 = raw_input('Enter count: ') pos
##Here is my code:
import urllib
from bs4 import BeautifulSoup
count_1 = raw_input('Enter count: ')
position = raw_input('Enter position: ')
count = int(count_1)
while count > 0:
list_of_tags = list()
url = raw_input("Enter URL: ")
fhand = urllib.urlopen(url).read()
soup = BeautifulSoup(fhand,"lxml")
tags = soup("a")
for tag in tags:
list_of_tags.append(tag.get("href",None))
print list_of_tags[int(position)]
count -=1
非常感谢所有帮助我已经准备了一些带有注释的代码。如果您有任何疑问或进一步的问题,请告诉我 给你:
import requests
from lxml import html
def searchRecordInSpecificPosition(url, position):
## Making request to the specified URL
response = requests.get(url)
## Parsing the DOM to a tree
tree = html.fromstring(response.content)
## Creating a dict of links.
links_dict = dict()
## Format of the dictionary:
##
## {
## 1: {
## 'href': "http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Medina.html",
## 'text': "Medina"
## },
##
## 2: {
## 'href': "http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Chiara.html",
## 'text': "Chiara"
## },
##
## ... and so on...
## }
counter = 1
## For each <a> tag found, extract its text and link (href) and insert it into links_dict
for link in tree.xpath('//ul/li/a'):
href = link.xpath('.//@href')[0]
text = link.xpath('.//text()')[0]
links_dict[counter] = dict(href=href, text=text)
counter += 1
return links_dict[position]['text'], links_dict[position]['href']
times_to_search = int(raw_input("Enter the amount of times to search: "))
position = int(raw_input('Enter position: '))
count = 0
print ""
while count < times_to_search:
if count == 0:
name, url = searchRecordInSpecificPosition("http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Brenae.html", position)
else:
name, url = searchRecordInSpecificPosition(url, position)
print "[*] Name: {}".format(name)
print "[*] URL: {}".format(url)
print ""
count += 1
➜ python scraper.py
Enter the amount of times to search: 4
Enter position: 1
[*] Name: Medina
[*] URL: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Medina.html
[*] Name: Darrius
[*] URL: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Darrius.html
[*] Name: Caydence
[*] URL: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Caydence.html
[*] Name: Peaches
[*] URL: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Peaches.html
➜