Google bigquery 带有气流的BigQuery-缺少projectId

尝试以下示例：

运行其中一个命令时：

airflow test bigquery_github_trends_v1 bq_check_githubarchive_day 2017-06-02

获取错误：TypeError:缺少必需的参数projectId

错误堆栈：

---

如果您检查bigquery\u hook的代码，您会发现它正在检查项目\u id

默认连接是bigquery\u default，除非您覆盖它，请转到aiffair UI，转到admin->connection->bigquery\u default或您创建的任何内容->在那里添加项目id

---

我知道这是一个老问题，但我也很挣扎，因为对我来说UI不起作用。 刚刚了解了如何通过CLI实现这一点，并希望分享我的发现，因为它没有文档记录

实际上有三种方法：

如前所述，通过环境变量 export AIRFLOW\u CONN\u BIGQUERY\u DEFAULT=google云平台：/：@:？extra\u google\u cloud\u平台\u project=yourprojectid&extra\u google\u cloud\u平台\u key\u path=/path/to/keyfile.json 通过cli和URI 气流连接-d-连接id bigquery\u默认值 气流连接-a-conn\u id bigquery\u默认值-conn\u uri'google cloud platform://：：？extra\u google\u cloud\u platform\uu project=yourprojectd&extra\u google\u cloud\u platform\uu key\u path=/path/to/keyfile.json' 通过cli和params 气流连接-d-连接id bigquery\u默认值 气流连接-a-conn\u id bigquery\u default-conn\u类型谷歌云平台-conn\u extra'{extra\u谷歌云平台\u项目：yourprojectid，extra\u谷歌云平台\u密钥路径：/path/to/keyfile.json} 如果您省略了关键路径内容，则将使用gcloud命令行工具当前使用的凭据。通常是你的个人用户

---

完成此操作后，您可以使用该连接运行任何任务。。。或者气流测试…

这是非常有用的，你知道如何通过编程进行设置吗？这是唯一一个对我有效的解决方案2。在UI中更改它似乎没有效果。

[2017-09-11 16:32:26,630] {models.py:1126} INFO - Dependencies all met for <TaskInstance: bigquery_github_trends_v1.bq_check_githubarchive_day 2017-06-02 00:00:00 [None]>
[2017-09-11 16:32:26,631] {models.py:1126} INFO - Dependencies all met for <TaskInstance: bigquery_github_trends_v1.bq_check_githubarchive_day 2017-06-02 00:00:00 [None]>
[2017-09-11 16:32:26,632] {models.py:1318} INFO - 
-----------------------------------------------------------------------
---------
Starting attempt 1 of 6
-----------------------------------------------------------------------
---------

[2017-09-11 16:32:26,632] {models.py:1342} INFO - Executing <Task(BigQueryCheckOperator): bq_check_githubarchive_day> on 2017-06-02 00:00:00
[2017-09-11 16:32:26,643] {check_operator.py:75} INFO - Executing SQL check: 
#legacySql
SELECT table_id 
FROM [githubarchive:day.__TABLES__] 
WHERE table_id = "20170601"

[2017-09-11 16:32:26,646] {gcp_api_base_hook.py:73} INFO - Getting connection using `gcloud auth` user, since no key file is defined for hook.
[2017-09-11 16:32:26,671] {models.py:1417} ERROR - Missing required parameter "projectId"
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/site-packages/airflow/operators/check_operator.py", line 76, in execute
records = self.get_db_hook().get_first(self.sql)
  File "/usr/local/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 135, in get_first
cur.execute(sql)
  File "/usr/local/lib/python2.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line 752, in execute
self.job_id = self.run_query(bql)
  File "/usr/local/lib/python2.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line 244, in run_query
return self.run_with_configuration(configuration)
  File "/usr/local/lib/python2.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line 498, in run_with_configuration
.insert(projectId=self.project_id, body=job_data) \
  File "/usr/local/lib/python2.7/site-packages/googleapiclient/discovery.py", line 716, in method
raise TypeError('Missing required parameter "%s"' % name)
TypeError: Missing required parameter "projectId"