Python 将url读入luigi参数

我正在尝试使用Luigi包读取本地驱动器中的csv文件，特别是作为文件名的Luigi.Parameter（），然后使用pd.read_csv将其读取到pandas数据帧中，并执行一些数据争用

这是我为此任务编写的代码：

import luigi
import pandas as pd
class read_blog(luigi.Task):
        fileName = luigi.Parameter()
        def run(self):
                full_file = pd.read_csv(fileName)
                read_blog = full_file[full_file['properties__url'].string.contain$
                        regex=False)]
                blog_readers = read_blog[['anonymous_id','channel',
                        'context__campaign__content','context__campaign__medium',
                        'context__campaign__name','context__campaign_source',
                        'context__campaign__term','timestamp','user_id',
                        'context__page__url','properties__url',
                        'properties__search','context__page__title',
                        'properties__path','context__user_agent',
                        'properties__referrer','rank']]
                blog_readers.to_csv('blog_readers.csv')
if __name__ == '__main__':
        luigi.run()

然后使用以下命令在终端上运行此命令：

python cleanup.py read_blog --local-scheduler --fileName '/Users/emmanuels/Desktop/attribute.csv'

根据我的理解，这应该在cleanup.py中运行read_blog类，并为fileName变量提供一个参数，该参数是指向我的csv文件的链接

然后，我的代码应将csv读取为熊猫数据帧，但这不会发生，这是我收到的完整错误消息：

===== Luigi Execution Summary =====

/Users/emmanuels/anaconda3/lib/python3.7/site-packages/luigi/configuration.py:54:UserWarning: LUIGI_CONFIG_PATH points to a file which does not exist. Invalid file: /Users/emmanuels/luigi_tutorial/luigi/luigi.conf
  warnings.warn("LUIGI_CONFIG_PATH points to a file which does not exist. Invalidfile: {path}".format(path=config_file))
DEBUG: Checking if read_blog(fileName=/Users/emmanuels/Desktop/attributiondata.csv) is complete
/Users/emmanuels/anaconda3/lib/python3.7/site-packages/luigi/worker.py:328: UserWarning: Task read_blog(fileName=/Users/emmanuels/Desktop/attributiondata.csv) without outputs has no custom complete() method
  is_complete = task.complete()
INFO: Informed scheduler that task   read_blog__Users_emmanuels_23aa7e1a57   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 94938] Worker Worker(salt=156803262, workers=1, host=Emmanuels-MacBook-Pro.local, username=emmanuels, pid=94938) running   read_blog(fileName=/Users/emmanuels/Desktop/attributiondata.csv)
ERROR: [pid 94938] Worker Worker(salt=156803262, workers=1, host=Emmanuels-MacBook-Pro.local, username=emmanuels, pid=94938) failed    read_blog(fileName=/Users/emmanuels/Desktop/attributiondata.csv)
Traceback (most recent call last):
  File "/Users/emmanuels/anaconda3/lib/python3.7/site-packages/luigi/worker.py", line 191, in run
    new_deps = self._run_get_new_deps()
  File "/Users/emmanuels/anaconda3/lib/python3.7/site-packages/luigi/worker.py", line 129, in _run_get_new_deps
    task_gen = self.task.run()
  File "cleanup.py", line 8, in run
    full_file = pd.read_csv(fileName)
NameError: name 'fileName' is not defined
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   read_blog__Users_emmanuels_23aa7e1a57   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 1 pending tasks possibly being run by other workers
DEBUG: There are 1 pending tasks unique to this worker
DEBUG: There are 1 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=156803262, workers=1, host=Emmanuels-MacBook-Pro.local, username=emmanuels, pid=94938) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 failed:
    - 1 read_blog(fileName=/Users/emmanuels/Desktop/attributiondata.csv)

This progress looks :( because there were failed tasks

---

fileName

是class

read\u blog

的一个属性，因此可以通过

self

访问

fileName

full\u file=pd.read\u csv（self.fileName）