Python 3.x Luigi本地目标二进制文件
在我的项目中,我很难在Luigi管道中编写二进制Python 3.x Luigi本地目标二进制文件,python-3.x,luigi,Python 3.x,Luigi,在我的项目中,我很难在Luigi管道中编写二进制LocalTarget。我在这里隔离了问题: class LuigiTest(luigi.Task): def output(self): return luigi.LocalTarget('test.npz') def run(self): with self.output().open('wb') as fout: np.savez_compressed(fout, ar
LocalTarget
。我在这里隔离了问题:
class LuigiTest(luigi.Task):
def output(self):
return luigi.LocalTarget('test.npz')
def run(self):
with self.output().open('wb') as fout:
np.savez_compressed(fout, array=np.asarray([1, 2, 3]))
我试图以'w'
和'wb'
的身份打开,但我一直收到以下错误:
TypeError: write() argument must be str, not bytes
我使用的是python 3.5.1,我的luigi版本是2.1.1问题在于
LocalTarget
的格式。将其更改为:
return luigi.LocalTarget('test.npz', format=luigi.format.Nop)
解决了这个问题。然而,文档中对此没有任何内容。它解决了我在Hadoop中编写拼花文件的问题
format=luigi.format.Nop
成功了。谢谢
import luigi
import pandas as pd
import luigi.contrib.hdfs as hdfs
class Hdfs(luigi.Task):
"""
Writes files into output.
"""
def __init__(self, *args, **kwargs):
super(Hdfs, self).__init__( *args, **kwargs)
def output(self):
fname_template = f'/data/some_directory/test_luigi.parq'
return luigi.contrib.hdfs.HdfsTarget(fname_template, format=luigi.format.Nop)
def run(self):
with self.output().open('w') as f:
print(f.path)
d = pd.DataFrame({'sim_id':[1,2,3]})
d.to_parquet(f)
同样的问题在这里,非常有用!