Azure data factory 无法将ADF polybase文件从Azure存储添加到Synapse池(SQL数据仓库)
如何让ADF copy polybase将一个文件从Azure存储复制到Synapse池(SQL数据仓库)中 我经常会遇到这样的错误:Azure data factory 无法将ADF polybase文件从Azure存储添加到Synapse池(SQL数据仓库),azure-data-factory,azure-synapse,polybase,Azure Data Factory,Azure Synapse,Polybase,如何让ADF copy polybase将一个文件从Azure存储复制到Synapse池(SQL数据仓库)中 我经常会遇到这样的错误: Operation on target Storage to Synapse failed: ErrorCode=FailedDbOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error happened when loading data
Operation on target Storage to Synapse failed: ErrorCode=FailedDbOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error happened when loading data into SQL Data Warehouse.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopSqlException: Error converting data type NVARCHAR to BIGINT.,Source=.Net SqlClient Data Provider,SqlErrorNumber=107090,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=107090,State=1,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopSqlException: Error converting data type NVARCHAR to BIGINT.,},],'
摘要:HadoopSqlException:将数据类型NVARCHAR转换为BIGINT.,},]时出错,
这是我的文件的示例,gzip压缩:
6872602179|359154818|84154822|1|37.00|68629.82|0.08|0.06|A|F|1992-09-01|1992-08-18|1992-09-05|TAKE BACK RETURN|REG AIR|efully. quickly bold deposits wake alo
6872602179|315023025|90023035|2|22.00|20509.94|0.02|0.05|A|F|1992-10-01|1992-07-06|1992-10-08|DELIVER IN PERSON|FOB| furious, pen
6872602179|591976355|41976366|3|41.00|57472.16|0.07|0.00|A|F|1992-07-23|1992-08-06|1992-08-18|TAKE BACK RETURN|AIR|have to are about the final instructions.
6872602179|1346584608|46584609|4|37.00|60135.36|0.00|0.05|R|F|1992-09-06|1992-07-23|1992-10-01|TAKE BACK RETURN|FOB|ackages according to the r
6872602179|47498972|97498973|5|31.00|61026.60|0.02|0.03|R|F|1992-06-05|1992-07-21|1992-06-07|NONE|REG AIR|ests cajole slyly quickly special attai
6872602179|315209731|90209741|6|31.00|56574.38|0.06|0.05|R|F|1992-06-15|1992-07-27|1992-07-15|COLLECT COD|SHIP|y bold instructions. even p
ADF中的源数据集设置:
ADF复制设置:
资料来源:
水槽:
映射:
目标表架构:
CREATE table dbo.[lineitem](
[L_OrderKey] [bigint] NULL,
[L_PartKey] [bigint] NULL,
[L_SuppKey] [bigint] NULL,
[L_LineNumber] [bigint] NULL,
[L_Quantity] [bigint] NULL,
[L_dboendedPrice] [decimal](13, 2) NULL,
[L_Discount] [decimal](13, 2) NULL,
[L_Tax] [decimal](13, 2) NULL,
[L_ReturnFlag] [varchar](64) NULL,
[L_LineStatus] [varchar](64) NULL,
[L_ShipDate] [datetime] NULL,
[L_CommitDate] [datetime] NULL,
[L_ReceiptDate] [datetime] NULL,
[L_ShipInstruct] [varchar](64) NULL,
[L_ShipMode] [varchar](64) NULL,
[L_Comment] [varchar](64) NULL
) ;
我建议在polybase表的开头只保留1个BigInt列来调试此问题,如果成功,这意味着第1列中没有出现此问题的数据,对于第二次调试运行,只保留2个BigInt列,然后进行检查 通过这种方式,您将能够知道哪个实际列存在问题,然后可以检查文件中的任何数据问题 如果有帮助,请告诉我
谢谢 请尝试在映射设置中设置源列类型字符串好吗?由于文件中有小数,所以可以尝试将第5列更改为小数点吗?如果不是这样,请尝试从映射中一次删除一个bigint列,直到它起作用,这样您就可以识别罪魁祸首。我的赌注在第五栏。@GregGalloway就是这样,我已经很久没有这样做了,非常令人沮丧,因为外部表立即起作用了……谢谢