Google bigquery 为第三方模块创建的动态资源指定依赖项

我正在使用谷歌云平台发布的TerraformHandy来构建一个BigQuery数据集，该数据集包含多个表和多个视图。这非常有效，除了当视图依赖于表时，我必须重试我的

apply

，因为没有任何东西告诉Terraform先做一个再做另一个。我根据

assets

子目录的内容分配

表

和

视图

属性，这为我提供了所需的所有架构和模板查询

除了放弃尼斯模块重用模式之外，有没有一种受支持的方法可以做到这一点？我真的很满意我的模式

为了完整性起见，我调用模块的代码是

locals {
  #
  # Tables
  #

  # Description attribute for each table. If absent, no description is set (null).
  table_to_description = {
# ...
  }

  # Values that don't ever change set for this dataset.
  TABLE_CONSTANTS = {
    time_partitioning = null
    expiration_time   = null
    clustering        = []
    labels = {
      terraform_managed = "true"
    }
  }

  TABLE_SCHEMA_SUFFIX = ".json"


  table_schema_filenames = fileset(pathexpand("${path.module}/assets/schemas"), "*.json")
  // fileset() doesn't have an option to output full paths, so we need to re-expand them
  table_schema_paths = [for file_name in local.table_schema_filenames : pathexpand("${path.module}/assets/schemas/${file_name}")]

  # Build a vector of objects, one for each table
  table_inputs = [for full_path in local.table_schema_paths : {
    schema = full_path
    # TODO(jaycarlton) I do not yet see a way around doing the replacement twice, as it's not possible
    #   to refer to other values in the same object when defining it.
    table_id    = replace(basename(full_path), local.TABLE_SCHEMA_SUFFIX, "")
    description = lookup(local.table_to_description, replace(basename(full_path), local.TABLE_SCHEMA_SUFFIX, ""), null)
  }]

  # Merge calculated inputs with the ones we use every time.
  tables = [for table_input in local.table_inputs :
    merge(table_input, local.TABLE_CONSTANTS)
  ]

  #
  # Views
  #
  VIEW_CONSTANTS = {
    # Reporting Subsystem always uses Standard SQL Syntax
    use_legacy_sql = false,
    labels = {
      terraform_managed = "true"
    }
  }
  QUERY_TEMPLATE_SUFFIX = ".sql"
  # Local filenames for view templates. Returns something like ["latest_users.sql", "users_by_id.sql"]
  view_query_template_filenames = fileset("${path.module}/assets/views", "*.sql")
  # expanded to fully qualified path, e.g. ["/repos/workbench/terraform/modules/reporting/views/latest_users.sql", ...]
  //  view_query_template_paths = [for file_name in local.view_query_template_filenames : pathexpand("./reporting/views/${file_name}")]
  view_query_template_paths = [for file_name in local.view_query_template_filenames : pathexpand("${path.module}/assets/views/${file_name}")]

  # Create views for each .sql file in the views directory. There is no Terraform
  # dependency from the view to the table(s) it queries, and I  don't believe the SQL is even checked
  # for accuracy prior to creation on the BQ side.
  views = [for view_query_template_path in local.view_query_template_paths :
    merge({
      view_id = replace(basename(view_query_template_path), local.QUERY_TEMPLATE_SUFFIX, ""),
      query = templatefile(view_query_template_path, {
        project = var.project_id
        dataset = var.reporting_dataset_id
      })
  }, local.VIEW_CONSTANTS)]

}

# All BigQuery assets for Reporting subsystem
module "main" {
  source     = "terraform-google-modules/bigquery/google"
  version    = "~> 4.3"
  dataset_id = var.reporting_dataset_id
  project_id = var.project_id
  location   = "US"

  # Note: friendly_name is discovered in plan and apply steps, but can't be
  # entered here. Maybe they're just not exposed by the dataset module but the resources are looking
  # for them?
  dataset_name = "Workbench ${title(var.aou_env)} Environment Reporting Data" # exposed as friendly_name in plan
  description  = "Daily output of relational tables and time series views for analysis. Views are provided for general ad-hoc analysis."

  tables = local.tables

  # Note that, when creating this module fom the ground up, it's common to see an error like
  # `Error: googleapi: Error 404: Not found: Table my-project:my_dataset.my_table, notFound`. It seems
  # to be a momentary issue due to the dataset's existence not yet being observable to the table/view
  # create API. So far, it's always worked on a re-run.
  # TODO(jaycarlton) see if there's a way to put a retry on this. I'm not convinced that will work
  #   outside of a resource context (and inside a third-party module).
  views = local.views

}

错误如下所示：

Error: googleapi: Error 404: Not found: Table <MY_PROJECT>:<MY_DATASET>.user, notFound

  on .terraform/modules/workbench.reporting.main/main.tf line 76, in resource "google_bigquery_table" "view":
  76: resource "google_bigquery_table" "view" {

错误：googleapi:错误404:未找到：表：。用户，未找到
在.terraform/modules/workbench.reporting.main/main.tf第76行的资源“google\u bigquery\u表”视图中：
76：资源“google\u bigquery\u表”“视图”{

---

这些视图被Terraform接受，被BigQuery拒绝，因为它们所引用的表要么尚未创建，要么尚未可用。看起来

dependens\u on

在一个资源块中，但在这种情况下，这些都是从我的理解中抽象出来的。重试器也可以解决我的问题（但不太优雅），因为在所有情况下，重新运行

terraform apply

都是有效的。

如果我理解了这里必要的数据流，我认为一种方法是安排

本地.views

依赖于模块中与表相关的输出值之一。因为

本地.views

表达式包含一个

模板文件调用，您可以通过将表名传递到模板中来实现这一点：

  views = [for view_query_template_path in local.view_query_template_paths :
    merge({
      view_id = replace(basename(view_query_template_path), local.QUERY_TEMPLATE_SUFFIX, ""),
      query = templatefile(view_query_template_path, {
        project     = var.project_id
        dataset     = var.reporting_dataset_id
        table_names = module.main.table_names
      })
  }, local.VIEW_CONSTANTS)]

由于
table\u name
输出值取决于模块中的表资源，因此您可以使用它来声明对表的间接依赖关系。模块的
views
参数本身将取决于
local。views
，从而间接地（现在到删除的级别）取决于表

以上假设模块内的表的配置在任何方面都不依赖于视图。如果存在这种依赖关系，那么这将创建一个依赖循环，但从快速阅读的结果来看，这似乎不是一个问题

这个答案所依赖的一个关键因素是Terraform模块输入变量和输出变量在依赖关系图中都是一个单独的节点，而不是整个模块都是一个节点。因此，模块的一个输入变量可能取决于同一模块的输出值，只要模块内部的依赖关系不要让它造成依赖循环