Python 2.7 Python,BeautifulSoup代码似乎可以工作,但CSV中没有数据?
我在一个目录中有大约500个html文件,我想从中提取数据并将结果保存在CSV中 我正在使用的代码没有收到任何错误消息,似乎正在扫描所有文件,但生成的CSV是空的,除了最上面的一行 我是python新手,显然我做错了什么。我希望有人能帮忙Python 2.7 Python,BeautifulSoup代码似乎可以工作,但CSV中没有数据?,python-2.7,web-scraping,beautifulsoup,Python 2.7,Web Scraping,Beautifulsoup,我在一个目录中有大约500个html文件,我想从中提取数据并将结果保存在CSV中 我正在使用的代码没有收到任何错误消息,似乎正在扫描所有文件,但生成的CSV是空的,除了最上面的一行 我是python新手,显然我做错了什么。我希望有人能帮忙 from bs4 import BeautifulSoup import csv import urllib2 import os def processData( pageFile ): f = open(pageFile, "r") pa
from bs4 import BeautifulSoup
import csv
import urllib2
import os
def processData( pageFile ):
f = open(pageFile, "r")
page = f.read()
f.close()
soup = BeautifulSoup(page)
metaData = soup.find_all('div class="item_details"')
priceData = soup.find_all('div class="price_big"')
# define where we will store info
vendors = []
shipsfroms = []
shipstos = []
prices = []
for html in metaData:
text = BeautifulSoup(str(html).strip()).get_text().encode("utf-8").replace("\n", "")
vendors.append(text.split("vendor:")[1].split("ships from:")[0].strip())
shipsfroms.append(text.split("ships from:")[1].split("ships to:")[0].strip())
shipstos.append(text.split("ships to:")[1].strip())
for price in priceData:
prices.append(BeautifulSoup(str(price)).get_text().encode("utf-8").strip())
csvfile = open('drugs.csv', 'ab')
writer = csv.writer(csvfile)
for shipsto, shipsfrom, vendor, price in zip(shipstos, shipsfroms, vendors, prices):
writer.writerow([shipsto, shipsfrom, vendor, price])
csvfile.close()
dir = "drugs"
csvFile = "drugs.csv"
csvfile = open(csvFile, 'wb')
writer = csv.writer(csvfile)
writer.writerow(["Vendors", "ShipsTo", "ShipsFrom", "Prices"])
csvfile.close()
fileList = os.listdir(dir)
totalLen = len(fileList)
count = 1
for htmlFile in fileList:
path = os.path.join(dir, htmlFile)
processData(path)
print "Processed '" + path + "'(" + str(count) + "/" + str(totalLen) + ")..."
count = count + 1
我怀疑我在告诉BS查找html代码的错误部分?但我看不出它应该是什么。以下是html代码的摘录,其中包含我需要的信息:
</div>
<div class="item" style="overflow: hidden;">
<div class="item_image" style="width: 180px; height: 125px;" id="image_255"><a href="https://silkroad6ownowfk.onion.to/items/200mg-high-quality-dmt" style="display: block; width: 180px; height: 125px;"></a></div>
<div class="item_body">
<div class="item_title"><a href="https://silkroad6ownowfk.onion.to/items/200mg-high-quality-dmt">200mg High Quality DMT</a></div>
<div class="item_details">
vendor: <a href="https://silkroad6ownowfk.onion.to/users/ringo-deathstarr">ringo deathstarr</a><br>
ships from: United States<br>
ships to: Worldwide
</div>
</div>
<div class="item_price">
<div class="price_big">฿0.031052</div>
<a href="https://silkroad6ownowfk.onion.to/items/200mg-high-quality-dmt#shipping">add to cart</a>
</div>
</div>
供应商:
船舶来自:美国
发送至:全球
฿0.031052
免责声明:此信息用于一个关于在线药物交易的研究项目。您的做法是错误的。以下是一个工作示例:
metaData = soup.find_all("div", {"class":"item_details"})
priceData = soup.find_all("div", {"class":"price_big"})
您可以从中找到有关其用法的更多信息。缩小问题范围(只需添加打印语句,例如,
print(priceData)
)。您使用的soup.find_all
不正确:使用soup.find_all('div','item_details')
而不是soup.find_all('div class=“item_details”')
您看过文档了吗?谢谢J.F.,我非常感谢。