Java 使用CQLOutputformat从hadoop写入时,Cassandra群集提供连接被拒绝错误
我有一个单节点cassandra集群,它没有与我的hadoop集群共存。我使用这个hadoop集群使用CqlOutputFormat和以下作业配置将数据批量加载到Cassandra集群中Java 使用CQLOutputformat从hadoop写入时,Cassandra群集提供连接被拒绝错误,java,hadoop,cassandra,cql3,cassandra-cli,Java,Hadoop,Cassandra,Cql3,Cassandra Cli,我有一个单节点cassandra集群,它没有与我的hadoop集群共存。我使用这个hadoop集群使用CqlOutputFormat和以下作业配置将数据批量加载到Cassandra集群中 Job job = new Job(conf); Configuration jobConf = job.getConfiguration(); ConfigHelper.setOutputInitialAddress(jobConf, "10.27.124.73"); Conf
Job job = new Job(conf);
Configuration jobConf = job.getConfiguration();
ConfigHelper.setOutputInitialAddress(jobConf, "10.27.124.73");
ConfigHelper.setInputInitialAddress(jobConf, "10.27.124.73");
//ConfigHelper.setOutputRpcPort(jobConf, "9160"); I tried setting the ports. Doesn't help.
//ConfigHelper.setInputRpcPort(jobConf, "9160");
ConfigHelper.setOutputPartitioner(jobConf,"Murmur3Partitioner");
ConfigHelper.setInputPartitioner(jobConf, "Murmur3Partitioner");
ConfigHelper.setInputColumnFamily(jobConf, "pinspace", "pinseries");
ConfigHelper.setOutputColumnFamily(jobConf, "pinspace", "pinseries");
System.out.println(ConfigHelper.getOutputColumnFamily(jobConf));
String query = "update pinspace.pinseries set timeseries = ?";
CqlConfigHelper.setOutputCql(jobConf, query);
CqlConfigHelper.setInputCQLPageRowSize(jobConf, "3");
job.setInputFormatClass(BoomInputFormat.class);
job.setMapperClass(GrepMapper.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setReducerClass(GrepReducer.class);
job.setNumReduceTasks(1024);
job.setOutputFormatClass(CqlOutputFormat.class);
我收到以下stacktrace的连接被拒绝错误:
Error: java.io.IOException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:280)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at org.apache.cassandra.thrift.TFramedTransportFactory.openTransport(TFramedTransportFactory.java:41)
at org.apache.cassandra.hadoop.AbstractColumnFamilyOutputFormat.createAuthenticatedClient(AbstractColumnFamilyOutputFormat.java:123)
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:271)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 4 more
我使用的是cassandra驱动核心2.1.5和cassandra全部2.1.3
服务器可以使用来自datanodes和telnet 10.27.124.73 9160以及telnet 10.27.124.73 9042的给定IP进行ping
编辑:
我试着在代码中使用,
setOutpurtPCPort(jobConf,“9042”)
然后错误更改为:
Error: java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Read a negative frame size (-2097152000)!
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.<init>(CqlRecordWriter.java:130)
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.<init>(CqlRecordWriter.java:88)
at org.apache.cassandra.hadoop.cql3.CqlOutputFormat.getRecordWriter(CqlOutputFormat.java:74)
at org.apache.cassandra.hadoop.cql3.CqlOutputFormat.getRecordWriter(CqlOutputFormat.java:55)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.thrift.transport.TTransportException: Read a negative frame size (-2097152000)!
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:133)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.cassandra.thrift.Cassandra$Client.recv_set_keyspace(Cassandra.java:608)
at org.apache.cassandra.thrift.Cassandra$Client.set_keyspace(Cassandra.java:595)
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.<init>(CqlRecordWriter.java:108)
... 11 more
错误:java.lang.RuntimeException:org.apache.thrift.transport.ttTransportException:读取负帧大小(-2097152000)!
位于org.apache.cassandra.hadoop.cql3.CqlRecordWriter(CqlRecordWriter.java:130)
位于org.apache.cassandra.hadoop.cql3.CqlRecordWriter(CqlRecordWriter.java:88)
位于org.apache.cassandra.hadoop.cql3.CqlOutputFormat.getRecordWriter(CqlOutputFormat.java:74)
位于org.apache.cassandra.hadoop.cql3.CqlOutputFormat.getRecordWriter(CqlOutputFormat.java:55)
位于org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter。(ReduceTask.java:540)
位于org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
位于org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
位于org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
位于java.security.AccessController.doPrivileged(本机方法)
位于javax.security.auth.Subject.doAs(Subject.java:415)
位于org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
位于org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
原因:org.apache.thrift.transport.ttTransportException:读取负帧大小(-2097152000)!
位于org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:133)
位于org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
位于org.apache.thrift.transport.tttransport.readAll(tttransport.java:86)
位于org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
位于org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
位于org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
位于org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
在org.apache.cassandra.thrift.cassandra$Client.recv_set_键空间(cassandra.java:608)
位于org.apache.cassandra.thrift.cassandra$Client.set_键空间(cassandra.java:595)
位于org.apache.cassandra.hadoop.cql3.CqlRecordWriter.(CqlRecordWriter.java:108)
... 还有11个