Sql 雪花临时表排序
我试图在CTE(带)和临时表之间进行性能比较。我已经将查询转换为已利用的临时表,但是我看到了这样一种情况:当作为一个整体运行时,执行失败,因为一个早期的临时表执行被取消。 样本: 当tt3开始运行时,执行似乎被取消,tt3与tt1之间存在依赖关系。 我尝试过CTA,并使用标准的Sql 雪花临时表排序,sql,snowflake-cloud-data-platform,Sql,Snowflake Cloud Data Platform,我试图在CTE(带)和临时表之间进行性能比较。我已经将查询转换为已利用的临时表,但是我看到了这样一种情况:当作为一个整体运行时,执行失败,因为一个早期的临时表执行被取消。 样本: 当tt3开始运行时,执行似乎被取消,tt3与tt1之间存在依赖关系。 我尝试过CTA,并使用标准的CREATE table定义表,然后在中插入,但在取消执行的情况下,结果是相同的 有没有一种方法可以通过标准sql语法更好地对这些进行排序,而不必执行存储过程或任务 编辑[2] 下面是一个简单的场景,使用行为相同的COVI
CREATE table
定义表,然后在中插入,但在取消执行的情况下,结果是相同的
有没有一种方法可以通过标准sql语法更好地对这些进行排序,而不必执行存储过程或任务
编辑[2]
下面是一个简单的场景,使用行为相同的COVID数据集。两种情况下的终止时间似乎都在35秒左右。因此,出于复制目的,如果您可以生成一个运行2分钟的查询,然后创建一个简单的后续临时表,从您应该能够复制的第一个查询中进行选择
CREATE TEMPORARY TABLE PLATFORM_EVALUATIONS.PUBLIC.TT1 AS
select a.COUNTRY_REGION
, a.PROVINCE_STATE
, a.COUNTY
, current_date() c_date
, count(*) counts
from PUBLIC.JHU_COVID_19_TIMESERIES a
inner join public.jhu_dashboard_covid_19_global b
on a.country_region = b.country_region
where 1=1
and a.COUNTRY_REGION = 'United States'
group by a.COUNTRY_REGION
, a.PROVINCE_STATE
, a.COUNTY;
CREATE TEMPORARY TABLE PLATFORM_EVALUATIONS.PUBLIC.TT2 AS
SELECT COUNTRY_REGION
, COUNTS
FROM PLATFORM_EVALUATIONS.PUBLIC.TT1;
编辑:
您可以在屏幕截图中看到,有效条形码(tt1)的持续时间缩短(通常为2米14秒),其他步骤在其完成/终止之前开始
// this is tt1 in my example it produces 3,657,599 rows in 2m 14s but is cancelled short when run as a whole.
CREATE temporary table business_vault_dev.dnr._valid_barcodes as
SELECT DISTINCT
e.Barcode
FROM BUSINESS_VAULT_DEV.DNR.DIM_EVENT_VW e
INNER JOIN BUSINESS_VAULT_DEV.DNR.DIM_PACKAGE_VW p
on e.BARCODE = COALESCE(p.Barcode, p.Barcode)
INNER JOIN BUSINESS_VAULT_DEV.DNR.DIM_ORDER_VW o
on p.order_key = o.order_key
INNER JOIN BUSINESS_VAULT_DEV.SCORECARD.DIM_CUSTOMER_VW cl
on o.CUSTOMER_ID=cl.CUSTOMER_ID
WHERE e.Event_Type='Delivered'
and o.service_code not in ('XD', 'PU')
and lower(cl.NAME) IN ('a', 'b', 'c')
GROUP BY e.Barcode
HAVING date(min(e.EST_Event_DateTime)) between '2020-09-05' and '2020-10-02';
// this is tt2
CREATE temporary table business_vault_dev.dnr._driver_merge as
select
//redacted
from data_lake.driver.driver_table1
union
select
//redacted
from data_lake.driver.driver_table2
union
select
//redacted
from data_lake.driver.driver_table3
UNION
select
//redacted
from data_lake.driver.driver_table4;
// this is unreferenced in my example but completes successfully and references the previous temp table
CREATE temporary table business_vault_dev.dnr._driver as
select
// redacted
from business_vault_dev.dnr._driver_merge dm
group by dm.contractor_number;
// this is tt3 in my example
CREATE temporary table business_vault_dev.dnr._slef_dedup as
SELECT
e.Barcode
// redacting columns here
FROM BUSINESS_VAULT_DEV.DNR.DIM_EVENT_VW e
INNER JOIN business_vault_dev.dnr._valid_barcodes vb
ON e.Barcode = vb.Barcode
LEFT JOIN data_lake.driver.driver_table1 cn
ON //redacted
LEFT JOIN data_lake.driver.driver_table2 pcp
ON //redacted
;
Daniel,我可以用你的简单场景和COVID数据库在我的测试中重现这个问题。新UI似乎有问题。它开始运行第二个CREATETABLE命令,而没有完成第一个命令。请向Snowflake support提交一张票证
提交此票证时,生成并共享HAR文件将非常有用,并包含SQL以进行复制
您的最终查询似乎会失败,因为tt1
不存在。正确,但我在作业历史记录中看到,随着tt3的启动,t1被取消(因此无法完成表的创建)。在较小的临时表中,这似乎不是问题,因为执行和加载在下一次执行之前完成occurs@DanielZagales为什么会被取消呢?这是故意的吗?我不知道,我没有得到很多关于取消的信息<代码>SQL执行已取消
,这让我相信我需要以某种方式对执行排序以避免它。我相信我可以用任务或存储过程来完成,但我希望只使用sql。您好-如果您以相同的顺序单独运行这些语句,并等待每一个语句完成后再开始下一个语句,那么这三个语句运行正常吗?感谢Gokhan确认此行为。我已经提交了一张支持票。
// this is tt1 in my example it produces 3,657,599 rows in 2m 14s but is cancelled short when run as a whole.
CREATE temporary table business_vault_dev.dnr._valid_barcodes as
SELECT DISTINCT
e.Barcode
FROM BUSINESS_VAULT_DEV.DNR.DIM_EVENT_VW e
INNER JOIN BUSINESS_VAULT_DEV.DNR.DIM_PACKAGE_VW p
on e.BARCODE = COALESCE(p.Barcode, p.Barcode)
INNER JOIN BUSINESS_VAULT_DEV.DNR.DIM_ORDER_VW o
on p.order_key = o.order_key
INNER JOIN BUSINESS_VAULT_DEV.SCORECARD.DIM_CUSTOMER_VW cl
on o.CUSTOMER_ID=cl.CUSTOMER_ID
WHERE e.Event_Type='Delivered'
and o.service_code not in ('XD', 'PU')
and lower(cl.NAME) IN ('a', 'b', 'c')
GROUP BY e.Barcode
HAVING date(min(e.EST_Event_DateTime)) between '2020-09-05' and '2020-10-02';
// this is tt2
CREATE temporary table business_vault_dev.dnr._driver_merge as
select
//redacted
from data_lake.driver.driver_table1
union
select
//redacted
from data_lake.driver.driver_table2
union
select
//redacted
from data_lake.driver.driver_table3
UNION
select
//redacted
from data_lake.driver.driver_table4;
// this is unreferenced in my example but completes successfully and references the previous temp table
CREATE temporary table business_vault_dev.dnr._driver as
select
// redacted
from business_vault_dev.dnr._driver_merge dm
group by dm.contractor_number;
// this is tt3 in my example
CREATE temporary table business_vault_dev.dnr._slef_dedup as
SELECT
e.Barcode
// redacting columns here
FROM BUSINESS_VAULT_DEV.DNR.DIM_EVENT_VW e
INNER JOIN business_vault_dev.dnr._valid_barcodes vb
ON e.Barcode = vb.Barcode
LEFT JOIN data_lake.driver.driver_table1 cn
ON //redacted
LEFT JOIN data_lake.driver.driver_table2 pcp
ON //redacted
;