Python &引用;传递的参数无效";dag使用气流将mysql数据加载到bigquery时出错
我运行一个DAG来提取MySQL数据并将其加载到airflow中的BigQuery。我当前收到以下错误: /usr/local/lib/python2.7/dist-packages/afflow/models.py:1927:pendingdepractionwarning:传递给mysqltologlecloudstorageoperator的参数无效。气流2.0将不再支持传递此类参数。无效参数为: *args:() **kwargs:{'google_cloud_storage_connn_id':'podioGCPConnection'}类别=PendingDeprecationWarning /usr/local/lib/python2.7/dist-packages/afflow/models.py:1927:pendingdepractionwarning:传递给GoogleCloudStorageToBigQueryOperator的参数无效。气流2.0将不再支持传递此类参数。无效参数为: *args:() **kwargs:{'project_id':'podio data'}类别=PendingDeprecationWarning dag的代码如下所示:Python &引用;传递的参数无效";dag使用气流将mysql数据加载到bigquery时出错,python,mysql,google-bigquery,airflow,Python,Mysql,Google Bigquery,Airflow,我运行一个DAG来提取MySQL数据并将其加载到airflow中的BigQuery。我当前收到以下错误: /usr/local/lib/python2.7/dist-packages/afflow/models.py:1927:pendingdepractionwarning:传递给mysqltologlecloudstorageoperator的参数无效。气流2.0将不再支持传递此类参数。无效参数为: *args:() **kwargs:{'google_cloud_storage_connn
my_connections = [
'podiotestmySQL'
]
my_tables = [
'logistics_orders',
'logistics_waybills',
'logistics_shipping_lines',
'logistics_info_requests'
]
default_args = {
'owner' : 'tia',
'start_date' : datetime(2018, 1, 2),
'depends_on_past' : False,
'retries' : 1,
'retry_delay':timedelta(minutes=5),
}
dag = DAG('etl', default_args=default_args,schedule_interval=timedelta(days=1))
slack_notify = SlackAPIPostOperator (
task_id = 'slack_notfiy',
token = 'xxxxxx',
channel='data-status',
username = 'airflow',
text = 'Successfully performed podio ETL operation',
dag=dag)
for connection in my_connections:
for table in my_tables:
extract = MySqlToGoogleCloudStorageOperator(
task_id="extract_mysql_%s_%s"%(connection,table),
mysql_conn_id = connection,
google_cloud_storage_connn_id = 'podioGCPConnection',
sql = "SELECT *, '%s' as source FROM podiodb.%s"%(connection,table),
bucket='podio-reader-storage',
filename= '%s/%s/%s{}.json'%(connection,table,table),
schema_filename='%s/schemas/%s.json'%(connection,table),
dag=dag)
load =GoogleCloudStorageToBigQueryOperator(
task_id = "load_bg_%s_%s"%(connection,table),
bigquery_conn_id = 'podioGCPConnection',
google_cloud_storage_conn_id = 'podioGCPConnection',
bucket = 'podio-reader-storage',
destination_project_dataset_table = "Podio_Data1.%s/%s"%(connection,table),
source_objects = ["%s/%s/%s*.json"%(connection,table,table)],
schema_object = "%s/schemas/%s.json"%(connection,table),
source_format = 'NEWLINE_DELIMITED_JSON',
create_disposition = 'CREATE_IF_NEEDED',
write_disposition = 'WRITE_TRUNCATE',
project_id = 'podio-data',
dag=dag)
load.set_upstream(extract)
slack_notify.set_upstream(load)
阅读这里的来源:
请从默认参数中删除这些参数:
google_cloud_storage_connn_id = 'podioGCPConnection'
project_id = 'podio-data',
您需要在气流仪表板中创建连接
我很高兴它对你有用!:)