Python 用于is直接批处理任务的luigi批处理模块
我有500个链接要下载，并想批量他们的例子10个项目 这个伪代码是什么样子的Python 用于is直接批处理任务的luigi批处理模块,python,parallel-processing,luigi,Python,Parallel Processing,Luigi,我有500个链接要下载，并想批量他们的例子10个项目 这个伪代码是什么样子的 class BatchJobTask(luigi.Task) items = luigi.Parameter() def run(self): listURLs = [] with ('urls_chunk', 'r') as urls for line in urls: listURLs.append('http:
class BatchJobTask(luigi.Task)
items = luigi.Parameter()
def run(self):
listURLs = []
with ('urls_chunk', 'r') as urls
for line in urls:
listURLs.append('http://ggg'+line+'.org')
10_urls = listURLs[0:items] #10 items here
for i in 10_urls:
req = request.get(url)
req.contents
def output(self):
return self.LocalTarger("downloaded_filelist.txt")
class BatchWorker(luigi.Task)
def run(self)
# Here I should run BatchJobTask from 0 to 10, next 11 - 21 new etc...
它会是怎样的？这里有一种方法，可以做一些您想要做的事情，但是字符串列表存储在文件中作为单独的行
import luigi
import requests
BATCH_SIZE = 10
class BatchProcessor(luigi.Task):
items = luigi.ListParameter()
max = luigi.IntParameter()
def requires(self):
return None
def output(self):
return luigi.LocalTarget('processed'+str(max)+'.txt')
def run(self):
for item in self.items:
req = requests.get('http://www.'+item+'.org')
# do something useful here
req.contents
open("processed"+str(max)+".txt",'w').close()
class BatchCreator(luigi.Task):
file_with_urls = luigi.Parameter()
def requires(self):
required_tasks = []
f = open(self.file_with_urls)
batch_index = 0
total_index = 0
lines = []
while True:
line = f.readline()
if not line: break
total_index += 1
if batch_index < BATCH_SIZE:
lines.append(line)
batch_index += 1
else:
required_tasks.append(BatchProcessor(batch=lines))
lines = [line]
batch_index = 1
return required_tasks
def output(self):
return luigi.LocalTarget(str(self.file_with_urls) + 'processed')
def run(self):
open(str(self.file_with_urls) + 'processed', 'w').close()
class GetListtask(luigi.Task)
def run(self):
...
def output(self):
return luigi.LocalTarget(self.outputfile)
class GetJustOneFile(luigi.Task):
fid = luigi.IntParameter()
def requires(self):
pass
def run(self):
url = 'http://my-server.com/test' + str(self.fid) + '.txt'
download_file = requests.get(url, stream=True)
with self.output().open('w') as downloaded_file:
downloaded_file.write(str(download_file.content))
def output(self):
return luigi.LocalTarget("test{}.txt".format(self.fid))
class GetAllFiles(luigi.WrapperTask):
def requires(self):
listoffiles = [] # 0..999
for i in range(899):
listoffiles.append(i)
return [GetJustOneFile(fid=fileid) for fileid in listoffiles]
这段代码糟糕吗？你的URL列表在哪里？我已经更新了第一个帖子，意思是URL列表存储在哪里？在队列、数据库、文件中？你要做的是计算出有多少在那东西里，然后从那里建立你的块。我将在下面举个例子，但它不太可能与您的问题相关，因为您没有指定问题的相关部分。嗯，它不进行批处理，但是它应该可以工作。我如何从GetAllFiles中的GetListTask而不是预定义列表中输入文件？这就是我在我的
BatchCreatorrequiresmaxself.max