Warning: file_get_contents(/data/phpspider/zhask/data//catemap/6/jenkins/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 自托管集成运行时加载ORC文件时Azure数据工厂管道失败:OutOfMemory异常,堆大小_Java_Azure_Azure Data Factory_Heap Memory_Azure Integration Runtime - Fatal编程技术网

Java 自托管集成运行时加载ORC文件时Azure数据工厂管道失败:OutOfMemory异常,堆大小

Java 自托管集成运行时加载ORC文件时Azure数据工厂管道失败:OutOfMemory异常,堆大小,java,azure,azure-data-factory,heap-memory,azure-integration-runtime,Java,Azure,Azure Data Factory,Heap Memory,Azure Integration Runtime,当我试图从Azure Data Factory加载ORC文件时,我目前面临一个问题。当文件太大时,ADF管道会抱怨我们的自托管集成运行时失败,并出现OutOfMemory异常,因为Java最大堆大小太小,无法完成加载 已经尝试过不同的解决方案,比如通过环境变量甚至注册表中的键来增加堆大小(有点像黑客)。具有自托管集成运行时的VM具有超过100GB的RAM 但仍然失败,因为当从ADF查询集成运行时时,这些值似乎一直被“默认”值覆盖。有什么想法吗 'Type=Microsoft.DataTransf

当我试图从Azure Data Factory加载ORC文件时,我目前面临一个问题。当文件太大时,ADF管道会抱怨我们的自托管集成运行时失败,并出现OutOfMemory异常,因为Java最大堆大小太小,无法完成加载

已经尝试过不同的解决方案,比如通过环境变量甚至注册表中的键来增加堆大小(有点像黑客)。具有自托管集成运行时的VM具有超过100GB的RAM

但仍然失败,因为当从ADF查询集成运行时时,这些值似乎一直被“默认”值覆盖。有什么想法吗

'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.nio.BufferOverflowException:Unable to retrieve Java exception..,Source=Microsoft.DataTransfer.Richfile.OrcTransferPlugin,StackTrace= at Microsoft.DataTransfer.ClientLibrary.OrcDeserializer.<GetRows>d__42.MoveNext()
at Microsoft.DataTransfer.Common.Shared.DeserializeControllerBase.GetEstimatedRowSize()
at Microsoft.DataTransfer.ClientLibrary.OrcDeserializeController..ctor(DataTable targetSchema, IEnumerable`1 streams, OrcFormatSetting settings, IErrorRowOutput errorRowOutput)
at Microsoft.DataTransfer.ClientLibrary.OrcSerializer.Deserialize(TransferStream stream)
at Microsoft.DataTransfer.Runtime.DeserializationStageProcessor.<Deserialize>d__14.MoveNext()
at Microsoft.DataTransfer.Runtime.TypeConversionStageProcessor.<CreateDataReader>d__5.MoveNext()
at Microsoft.DataTransfer.Runtime.SerializationStageProcessor.<Serialize>d__11.MoveNext()
at Microsoft.DataTransfer.Runtime.BinarySinkStageProcessor.<PopulateStreamName>d__10.MoveNext()
at Microsoft.DataTransfer.ClientLibrary.MultipartWriteSink.ConsumeStreams(IEnumerable`1 streams),''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,StackTrace= at Microsoft.DataTransfer.Richfile.Bridge.BaseObjectBridge.CallObject[TEnum](TEnum methodEnum, jValue[] args)
at Microsoft.DataTransfer.Richfile.Bridge.Orc.OrcBatchReaderBridge.MoveNext()
at Microsoft.DataTransfer.ClientLibrary.OrcDeserializer.<GetRows>d__42.MoveNext(),'
Job ID: daee1a1d-b880-ecb2-e56c-a59397547668
Log ID: Warning        
TraceComponentId: TransferClientLibrary
TraceMessageId: TasksCoordinatorFatalErrorCallback
@logId: Warning
jobId: daee1a1d-b880-ecb2-e56c-a59397547668
activityId: c643b611-8356-4f49-b6d6-e87ea50670e5
eventId: TasksCoordinatorFatalErrorCallback
message: 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.nio.BufferOverflowException:Unable to retrieve Java exception..,Source=Microsoft.DataTransfer.Richfile.OrcTransferPlugin,StackTrace= at Microsoft.DataTransfer.ClientLibrary.OrcDeserializer.<GetRows>d__42.MoveNext()
at Microsoft.DataTransfer.Common.Shared.DeserializeControllerBase.GetEstimatedRowSize()
at Microsoft.DataTransfer.ClientLibrary.OrcDeserializeController..ctor(DataTable targetSchema, IEnumerable`1 streams, OrcFormatSetting settings, IErrorRowOutput errorRowOutput)
at Microsoft.DataTransfer.ClientLibrary.OrcSerializer.Deserialize(TransferStream stream)
at Microsoft.DataTransfer.Runtime.DeserializationStageProcessor.<Deserialize>d__14.MoveNext()
at Microsoft.DataTransfer.Runtime.TypeConversionStageProcessor.<CreateDataReader>d__5.MoveNext()
at Microsoft.DataTransfer.Runtime.SerializationStageProcessor.<Serialize>d__11.MoveNext()
at Microsoft.DataTransfer.Runtime.BinarySinkStageProcessor.<PopulateStreamName>d__10.MoveNext()
at Microsoft.DataTransfer.ClientLibrary.MultipartWriteSink.ConsumeStreams(IEnumerable`1 streams),''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,StackTrace= at Microsoft.DataTransfer.Richfile.Bridge.BaseObjectBridge.CallObject[TEnum](TEnum methodEnum, jValue[] args)
at Microsoft.DataTransfer.Richfile.Bridge.Orc.OrcBatchReaderBridge.MoveNext()
at Microsoft.DataTransfer.ClientLibrary.OrcDeserializer.<GetRows>d__42.MoveNext(),'
'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=调用java时出错,Message:java.nio.BufferOverflowException:无法检索java异常..,Source=Microsoft.DataTransfer.Richfile.OrcTransferPlugin,StackTrace=at-Microsoft.DataTransfer.ClientLibrary.OrcDeserializer.d\u 42.MoveNext()
在Microsoft.DataTransfer.Common.Shared.DeserializeControllerBase.GetEstimatedRowSize()中
在Microsoft.DataTransfer.ClientLibrary.OrcDeserializeController..ctor(DataTable targetSchema,IEnumerable`1 streams,OrcFormatSetting设置,ierRowOutput errorRowOutput)
位于Microsoft.DataTransfer.ClientLibrary.OrcSerializer.Deserialize(TransferStream)
在Microsoft.DataTransfer.Runtime.DeserializationStageProcessor.d_u14.MoveNext()中
在Microsoft.DataTransfer.Runtime.TypeConversionStageProcessor.d_u5.MoveNext()中
在Microsoft.DataTransfer.Runtime.SerializationStageProcessor.d_u11.MoveNext()中
在Microsoft.DataTransfer.Runtime.BinarySinkStageProcessor.d_u10.MoveNext()中
在Microsoft.DataTransfer.ClientLibrary.MultipartWriteSink.ConsumerStreams(IEnumerable`1 streams)中,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.Bridge.BaseObjectBridge.CallObject[TEnum](TEnum methodEnum,jValue[]args)
位于Microsoft.DataTransfer.Richfile.Bridge.Orc.OrcBatchReaderBridge.MoveNext()处
在Microsoft.DataTransfer.ClientLibrary.OrcDeserializer.d_u42.MoveNext()中
作业ID:daee1a1d-b880-ecb2-e56c-a59397547668
日志ID:警告
TraceComponentId:TransferClientLibrary
TraceMessageId:TaskScoordinator FatalErrorCallback
@罗吉德:警告
作业ID:daee1a1d-b880-ecb2-e56c-a59397547668
活动ID:c643b611-8356-4f49-b6d6-e87ea50670e5
eventId:TasksCoordinatorFatalErrorCallback
message:'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,message=调用java时出错,message:java.nio.BufferOverflowException:无法检索java异常..,Source=Microsoft.DataTransfer.Richfile.OrcTransferPlugin,StackTrace=at Microsoft.DataTransfer.ClientLibrary.OrcDeserializer.d_42.MoveNext()
在Microsoft.DataTransfer.Common.Shared.DeserializeControllerBase.GetEstimatedRowSize()中
在Microsoft.DataTransfer.ClientLibrary.OrcDeserializeController..ctor(DataTable targetSchema,IEnumerable`1 streams,OrcFormatSetting设置,ierRowOutput errorRowOutput)
位于Microsoft.DataTransfer.ClientLibrary.OrcSerializer.Deserialize(TransferStream)
在Microsoft.DataTransfer.Runtime.DeserializationStageProcessor.d_u14.MoveNext()中
在Microsoft.DataTransfer.Runtime.TypeConversionStageProcessor.d_u5.MoveNext()中
在Microsoft.DataTransfer.Runtime.SerializationStageProcessor.d_u11.MoveNext()中
在Microsoft.DataTransfer.Runtime.BinarySinkStageProcessor.d_u10.MoveNext()中
在Microsoft.DataTransfer.ClientLibrary.MultipartWriteSink.ConsumerStreams(IEnumerable`1 streams)中,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.Bridge.BaseObjectBridge.CallObject[TEnum](TEnum methodEnum,jValue[]args)
位于Microsoft.DataTransfer.Richfile.Bridge.Orc.OrcBatchReaderBridge.MoveNext()处
在Microsoft.DataTransfer.ClientLibrary.OrcDeserializer.d_u42.MoveNext()中