Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Cassandra 火花与卡桑德拉配置_Cassandra_Apache Spark - Fatal编程技术网

Cassandra 火花与卡桑德拉配置

Cassandra 火花与卡桑德拉配置,cassandra,apache-spark,Cassandra,Apache Spark,在cassandra身上运行spark获得TTransportException。有人能告诉我,我必须通过spark访问cassandra db吗? 我将以下变量设置到.bashrc&conf/spark-env.sh文件中: export SCALA_HOME=/opt/scala-2.10.3 export SCALA_LIBRARY_PATH=/opt/scala-2.10.3/lib export MAVEN_OPTS="-Xmx1300M -XX:MaxPermSize=512

在cassandra身上运行spark获得TTransportException。有人能告诉我,我必须通过spark访问cassandra db吗?
我将以下变量设置到.bashrc&conf/spark-env.sh文件中:

export SCALA_HOME=/opt/scala-2.10.3  
export SCALA_LIBRARY_PATH=/opt/scala-2.10.3/lib  
export MAVEN_OPTS="-Xmx1300M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"  
export SPARK_HOME=/opt/spark-0.9.0  
export SPARK_CONF_DIR="$SPARK_HOME/conf"  
export SPARK_LOCAL_IP=192.168.140.35  
export SPARK_MASTER_IP=192.168.140.35  
export SPARK_MASTER_PORT=7077  
export SPARK_WORKER_MEMORY=1g  
export SPARK_MASTER_WEBUI_PORT=7123  
export SPARK_WORKER_PORT=8077  
export SPARK_WORKER_DIR=$SPARK_HOME/work  
export SPARK_WORKER_CORES=1  
export SPARK_WORKER_WEBUI_PORT=7456  
export SPARK_WORKER_INSTANCES=1  
export SPARK_CLASSPATH=/opt/cassandra/lib/apache-cassandra-1.2.12-    SNAPSHOT.jar:/opt/cassandra/lib/apache-cassandra-clientutil-1.2.12-SNAPSHOT.jar:/opt/cassandra/lib/apache-cassandra-thrift-1.2.12-SNAPSHOT.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.2.jar:/opt/cassandra/lib/commons-lang-2.6.jar:/opt/cassandra/lib/compress-lzf-0.8.4.jar:/opt/cassandra/lib/cql-internal-only-1.4.1.zip:/opt/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/cassandra/lib/jackson-core-asl-1.9.2.jar:/opt/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/opt/cassandra/lib/amm-0.2.5.jar:/opt/cassandra/lib/jbcrypt-0.3m.jar:/opt/cassandra/lib/jline-1.0.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/libthrift-0.7.0.jar:/opt/cassandra/lib/log4j-1.2.16.jar:/opt/cassandra/lib/lz4-1.1.0.jar:/opt/cassandra/lib/metrics-core-2.2.0.jar:/opt/cassandra/lib/netty-3.6.6.Final.jar:/opt/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/cassandra/lib/slf4j-api-1.7.2.jar:/opt/cassandra/lib/lf4j-log4j12-1.7.2.jar:/opt/cassandra/lib/snakeyaml-1.6.jar:/opt/cassandra/lib/snappy-java-1.0.5.jar:/opt/cassandra/lib/snaptree-0.1.jar  
export PATH=$PATH:$M2:$JAVA_HOME/bin:$HADOOP_HOME/bin:$CASSANDRA_HOME/bin:$SQOOP_HOME/bin:$OOZIE_HOME/bin:$PIG_HOME/bin:$HIVE_HOME/bin:$ANT_HOME/bin:$SCALA_HOME/bin
我已经成功地尝试了这个指令

快乐测试


编辑:对不起,实际上,我已经尝试将给定的博客文章与来自的非常简单的文档结合起来。

看看这个spark cassandra集成库。DataStax还发布了一个spark驱动程序,可以在这里找到它。

这个博客是针对带有cassandra的hive而不是针对带有cassandra的spark&我找不到第二部分这是我的博客。如果你知道这个博客的第二部分,让我知道链接。刚刚编辑了我的文章。很抱歉事实上,我也在使用DATASTAX发行版。我问过卡森德拉的火花,而不是蜂箱的鲨鱼。我知道,在中间你仍然需要Hadoop(HDFS)。另外,您是如何运行测试的?您在何处/何时收到异常?执行以下命令时,我收到异常&供您参考,master&worker正在正常运行。/bin/run example fullyqualifiedPackageName.ClassName参数