Apache spark SparkSQL可以’；无法将数据放入内存

在spark中默认启用溢出到磁盘。如何使用Ambari（HortonWorks）配置属性

我试着运行一个pyspark脚本，它使用1 TB大小的数据集。脚本无法显示内存错误

如何在Horton works中启动溢出到磁盘功能

我得到以下错误：

18/05/07 15:03:15 INFO DAGScheduler: Executor lost: 63 (epoch 668)
18/05/07 15:03:15 INFO BlockManagerMasterEndpoint: Trying to remove executor 63 from BlockManagerMaster.
18/05/07 15:03:15 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(63, USAZEHDP0009L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 45164, None)
18/05/07 15:03:15 INFO BlockManagerMaster: Removed 63 successfully in removeExecutor
18/05/07 15:03:15 INFO DAGScheduler: Shuffle files lost for executor: 63 (epoch 668)
18/05/07 15:03:16 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 63.
18/05/07 15:03:16 ERROR YarnScheduler: Lost an executor 63 (already removed): Pending loss reason.
18/05/07 15:03:16 INFO BlockManagerMaster: Removal of executor 63 requested
18/05/07 15:03:16 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 63
18/05/07 15:03:16 INFO BlockManagerMasterEndpoint: Trying to remove executor 63 from BlockManagerMaster.
18/05/07 15:03:34 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 62.
18/05/07 15:03:34 INFO DAGScheduler: Executor lost: 62 (epoch 681)
18/05/07 15:03:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 62 from BlockManagerMaster.
18/05/07 15:03:34 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(62, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 60057, None)
18/05/07 15:03:34 INFO BlockManagerMaster: Removed 62 successfully in removeExecutor
18/05/07 15:03:34 INFO DAGScheduler: Shuffle files lost for executor: 62 (epoch 681)
18/05/07 15:03:34 ERROR YarnScheduler: Lost executor 62 on USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net: Container container_e529_1525685078238_0240_01_000369 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:03:34 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container container_e529_1525685078238_0240_01_000369 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:03:34 INFO TaskSetManager: Task 6375 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task.
18/05/07 15:03:34 INFO BlockManagerMaster: Removal of executor 62 requested
18/05/07 15:03:34 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 62
18/05/07 15:03:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 62 from BlockManagerMaster.
18/05/07 15:03:34 INFO TaskSetManager: Starting task 0.1 in stage 7.2 (TID 6376, USAZEHDP0012L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, executor 7, partition 14, NODE_LOCAL, 6105 bytes)
18/05/07 15:03:34 INFO BlockManagerInfo: Added broadcast_62_piece0 in memory on USAZEHDP0012L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:57624 (size: 20.4 KB, free: 17.4 GB)
18/05/07 15:03:37 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.12:36932) with ID 67
18/05/07 15:03:37 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:42889 with 17.4 GB RAM, BlockManagerId(67, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 42889, None)
18/05/07 15:03:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 67.
18/05/07 15:03:52 INFO DAGScheduler: Executor lost: 67 (epoch 694)
18/05/07 15:03:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 67 from BlockManagerMaster.
18/05/07 15:03:52 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(67, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 42889, None)
18/05/07 15:03:52 INFO BlockManagerMaster: Removed 67 successfully in removeExecutor
18/05/07 15:03:52 INFO DAGScheduler: Shuffle files lost for executor: 67 (epoch 694)
18/05/07 15:03:52 ERROR YarnScheduler: Lost executor 67 on USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net: Container container_e529_1525685078238_0240_01_000379 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:03:52 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container container_e529_1525685078238_0240_01_000379 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:03:52 INFO BlockManagerMaster: Removal of executor 67 requested
18/05/07 15:03:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 67
18/05/07 15:03:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 67 from BlockManagerMaster.
18/05/07 15:03:55 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.12:36981) with ID 68
18/05/07 15:03:55 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:46941 with 17.4 GB RAM, BlockManagerId(68, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 46941, None)
18/05/07 15:04:10 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container container_e529_1525685078238_0240_01_000380 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:04:10 ERROR YarnScheduler: Lost executor 68 on USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net: Container container_e529_1525685078238_0240_01_000380 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:04:10 INFO DAGScheduler: Executor lost: 68 (epoch 707)
18/05/07 15:04:10 INFO BlockManagerMasterEndpoint: Trying to remove executor 68 from BlockManagerMaster.
18/05/07 15:04:10 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(68, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 46941, None)
18/05/07 15:04:10 INFO BlockManagerMaster: Removed 68 successfully in removeExecutor
18/05/07 15:04:10 INFO DAGScheduler: Shuffle files lost for executor: 68 (epoch 707)
18/05/07 15:04:10 WARN TransportChannelHandler: Exception in connection from /10.215.45.12:36981
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:745)
18/05/07 15:04:10 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 68.
18/05/07 15:04:10 ERROR YarnScheduler: Lost an executor 68 (already removed): Pending loss reason.
18/05/07 15:04:10 INFO BlockManagerMaster: Removal of executor 68 requested
18/05/07 15:04:10 INFO BlockManagerMasterEndpoint: Trying to remove executor 68 from BlockManagerMaster.
18/05/07 15:04:10 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 68
18/05/07 15:04:13 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.12:37050) with ID 69
18/05/07 15:04:13 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:35823 with 17.4 GB RAM, BlockManagerId(69, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 35823, None)
18/05/07 15:04:28 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 69.
18/05/07 15:04:28 INFO DAGScheduler: Executor lost: 69 (epoch 720)
18/05/07 15:04:28 INFO BlockManagerMasterEndpoint: Trying to remove executor 69 from BlockManagerMaster.
18/05/07 15:04:28 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(69, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 35823, None)
18/05/07 15:04:28 INFO BlockManagerMaster: Removed 69 successfully in removeExecutor
18/05/07 15:04:28 INFO DAGScheduler: Shuffle files lost for executor: 69 (epoch 720)
18/05/07 15:04:28 ERROR YarnScheduler: Lost executor 69 on USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net: Container container_e529_1525685078238_0240_01_000381 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:04:28 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container container_e529_1525685078238_0240_01_000381 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:04:28 INFO BlockManagerMaster: Removal of executor 69 requested
18/05/07 15:04:28 INFO BlockManagerMasterEndpoint: Trying to remove executor 69 from BlockManagerMaster.
18/05/07 15:04:28 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 69
18/05/07 15:04:31 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.12:37130) with ID 70
18/05/07 15:04:31 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:59500 with 17.4 GB RAM, BlockManagerId(70, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 59500, None)
18/05/07 15:04:46 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container container_e529_1525685078238_0240_01_000382 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:04:46 ERROR YarnScheduler: Lost executor 70 on USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net: Container container_e529_1525685078238_0240_01_000382 on host: USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net was preempted.
18/05/07 15:04:46 INFO DAGScheduler: Executor lost: 70 (epoch 733)
18/05/07 15:04:46 INFO BlockManagerMasterEndpoint: Trying to remove executor 70 from BlockManagerMaster.
18/05/07 15:04:46 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(70, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 59500, None)
18/05/07 15:04:46 INFO BlockManagerMaster: Removed 70 successfully in removeExecutor
18/05/07 15:04:46 INFO DAGScheduler: Shuffle files lost for executor: 70 (epoch 733)
18/05/07 15:04:46 WARN TransportChannelHandler: Exception in connection from /10.215.45.12:37130
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:745)
18/05/07 15:04:46 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 70.
18/05/07 15:04:46 ERROR YarnScheduler: Lost an executor 70 (already removed): Pending loss reason.
18/05/07 15:04:46 INFO BlockManagerMaster: Removal of executor 70 requested
18/05/07 15:04:46 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 70
18/05/07 15:04:46 INFO BlockManagerMasterEndpoint: Trying to remove executor 70 from BlockManagerMaster.
18/05/07 15:04:49 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.12:37145) with ID 71
18/05/07 15:04:49 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:46298 with 17.4 GB RAM, BlockManagerId(71, USAZEHDP0008L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 46298, None)
18/05/07 15:05:31 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.13:46477) with ID 72
18/05/07 15:05:31 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0009L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:45374 with 17.4 GB RAM, BlockManagerId(72, USAZEHDP0009L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 45374, None)
18/05/07 15:05:36 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.9:37171) with ID 73
18/05/07 15:05:36 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0006L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:36424 with 17.4 GB RAM, BlockManagerId(73, USAZEHDP0006L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 36424, None)
18/05/07 15:05:37 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.8:45503) with ID 75
18/05/07 15:05:37 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0005L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:48522 with 17.4 GB RAM, BlockManagerId(75, USAZEHDP0005L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 48522, None)
18/05/07 15:05:57 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.215.45.16:39916) with ID 74
18/05/07 15:05:57 INFO BlockManagerMasterEndpoint: Registering block manager USAZEHDP0012L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net:51078 with 17.4 GB RAM, BlockManagerId(74, USAZEHDP0012L.e1v0dl01lwsevnbuy20d4bdyeg.bx.internal.cloudapp.net, 51078, None)

---

请说明此错误的含义？

能否显示脚本和错误？请添加最低工作代码以指导您。请帮助解决上述异常