如何在azure databricks数据库中一次获取所有表列

我一次需要所有表列，这些列在特定的 Azure数据块中的DB。我知道通过以下方法查找sql server： 使用以下查询。 我也需要在databricks中实现同样的功能

---

实际上有很多方法可以实现这一点，一种是使用一些基本的python数据框架、列表和循环。对不起，我的英语很差

让我们玩

首先，您必须检索所有表名，并使用这些表名检索包含所有数据类型的列名的表说明。 我们使用2sparksql查询

1:Table_name=spark.sqlSHOW TABLES FROM default

默认databricks默认数据库名称

结果

+----+-------+------+ |数据库| tableName | i临时| +----+-------+------+ |默认值|可转换|错误| |默认| delta|表| false| |默认|显示|错误| |默认|事件|错误| |默认值|导出| csv |错误| |默认|航班|错误| |默认|播放|存储|错误| +----+-------+------+

2:table_descripe=spark.sqldescripe default.play_存储

结果

+-------+-----+----+ |列名称|数据类型|注释| +-------+-----+----+ |应用程序|字符串|空| |类别|字符串|空| |评级|字符串|空| |评论|字符串|空| |大小|字符串|空| |安装|字符串| null| |类型|字符串| null| |价格|字符串|空| |内容分级|字符串|空| |类型|字符串|空| |上次更新的|字符串|空| |当前版本|字符串|空| |Android版本|字符串|空| +-------+-----+----+

现在是主要部分

Table_name = spark.sql("SHOW TABLES FROM default")
Table_name=Table_name.select('tableName').collect()
mvv_array = [(row.tableName) for row in Table_name]
cnt = 0
for idx,i in enumerate(mvv_array):
  if cnt == 0:
    tmp = spark.sql('DESCRIBE default.'+i).withColumn('table_name',lit(i))
    temp_df = tmp
    cnt = cnt + 1
  else:
    tmp = spark.sql('DESCRIBE default.'+i).withColumn('table_name',lit(i))
    temp_df=temp_df.unionAll(tmp)
temp_df.show()

TADAAAA

+---------+-----+----+-------+ |列名称|数据类型|注释|表名称| +---------+-----+----+-------+ |年份| int | null |可转换| |月份| int | null |可转换| |DayofMonth | int | null |可转换| |DayOfWeek | int | null |可转换| |DepTime | string | null |可转换| |CRSDepTime | int | null |可转换| |ArrTime | string | null |可转换| |CRARRTIME | int | null |可转换| |唯一载波|字符串|空|可转换| |FlightNum | int | null |可转换| |TailNum | string | null |可转换| |ActualLapsedTime |字符串| null |可转换| |CRSElapsedTime |字符串| null |可转换| |AirTime | string | null |可转换| |ArrDelay | string | null |可转换| |DepDelay |字符串| null |可转换| |原点|字符串| null |可转换| |Dest | string | null |可转换| |距离| int | null |可转换| |Taxin |字符串| null |可转换| +---------+-----+----+-------+ 仅显示前20行

从新的数据帧中获取所需的内容

希望它能满足你的需求

---

如果您发现它是您的解决方案，请不要忘记标记为“答案”并向上投票。

Sohel Reza的答案很好，但如果您想获得嵌套列StructType，它无法解决问题。它只显示最高级别的列名

如果您想要一个列出所有数据库、表和列的简单视图，我在这里介绍了一个这样的解决方案：

它创建一个具有数据库、表名和列名的视图

您还可以从下面复制/粘贴完整代码：

from pyspark.sql.types import StructType

# get field name from schema (recursive for getting nested values)
def get_schema_field_name(field, parent=None):
  if type(field.dataType) == StructType:
    if parent == None:
      prt = field.name
    else:
      prt = parent+"."+field.name # using dot notation
    res = []
    for i in field.dataType.fields:
      res.append(get_schema_field_name(i, prt))
    return res
  else:
    if parent==None:
      res = field.name
    else:
      res = parent+"."+field.name
    return res
  
# flatten list, from https://stackoverflow.com/a/12472564/4920394
def flatten(S):
  if S == []:
    return S
  if isinstance(S[0], list):
    return flatten(S[0]) + flatten(S[1:])
  return S[:1] + flatten(S[1:])

# list of databases
db_list = [x[0] for x in spark.sql("SHOW DATABASES").rdd.collect()]

for i in db_list:
  spark.sql("SHOW TABLES IN {}".format(i)).createOrReplaceTempView(str(i)+"TablesList")

# create a query for fetching all tables from all databases
union_string = "SELECT database, tableName FROM "
for idx, item in enumerate(db_list):
  if idx == 0:
    union_string += str(item)+"TablesList WHERE isTemporary = 'false'"
  else:
    union_string += " UNION ALL SELECT database, tableName FROM {}".format(str(item)+"TablesList WHERE isTemporary = 'false'")
spark.sql(union_string).createOrReplaceTempView("allTables")

# full list = schema, table, column
full_list = []
for i in spark.sql("SELECT * FROM allTables").collect():
  table_name = i[0]+"."+i[1]
  table_schema = spark.sql("SELECT * FROM {}".format(table_name))
  column_list = []
  for j in table_schema.schema:
    column_list.append(get_schema_field_name(j))
  column_list = flatten(column_list)
  for k in column_list:
    full_list.append([i[0],i[1],k])
spark.createDataFrame(full_list, schema = ['database', 'tableName', 'columnName']).createOrReplaceTempView("allColumns")```

---

嗨，这里我们只看到前20排。如何获取所有行。请告诉我。事实上，您并不是只得到20行数据框，而是显示前20行。在你的内心深处，你拥有所有的数据。我如何才能以excel的形式下载所有的行。因为我想看所有的行。非常感谢。我得到了所有的钱。我用了diplaytemp_df。它直接显示下载选项。

from pyspark.sql.types import StructType

# get field name from schema (recursive for getting nested values)
def get_schema_field_name(field, parent=None):
  if type(field.dataType) == StructType:
    if parent == None:
      prt = field.name
    else:
      prt = parent+"."+field.name # using dot notation
    res = []
    for i in field.dataType.fields:
      res.append(get_schema_field_name(i, prt))
    return res
  else:
    if parent==None:
      res = field.name
    else:
      res = parent+"."+field.name
    return res
  
# flatten list, from https://stackoverflow.com/a/12472564/4920394
def flatten(S):
  if S == []:
    return S
  if isinstance(S[0], list):
    return flatten(S[0]) + flatten(S[1:])
  return S[:1] + flatten(S[1:])

# list of databases
db_list = [x[0] for x in spark.sql("SHOW DATABASES").rdd.collect()]

for i in db_list:
  spark.sql("SHOW TABLES IN {}".format(i)).createOrReplaceTempView(str(i)+"TablesList")

# create a query for fetching all tables from all databases
union_string = "SELECT database, tableName FROM "
for idx, item in enumerate(db_list):
  if idx == 0:
    union_string += str(item)+"TablesList WHERE isTemporary = 'false'"
  else:
    union_string += " UNION ALL SELECT database, tableName FROM {}".format(str(item)+"TablesList WHERE isTemporary = 'false'")
spark.sql(union_string).createOrReplaceTempView("allTables")

# full list = schema, table, column
full_list = []
for i in spark.sql("SELECT * FROM allTables").collect():
  table_name = i[0]+"."+i[1]
  table_schema = spark.sql("SELECT * FROM {}".format(table_name))
  column_list = []
  for j in table_schema.schema:
    column_list.append(get_schema_field_name(j))
  column_list = flatten(column_list)
  for k in column_list:
    full_list.append([i[0],i[1],k])
spark.createDataFrame(full_list, schema = ['database', 'tableName', 'columnName']).createOrReplaceTempView("allColumns")```