Datastax enterprise cassandra中的批量加载错误
早上好 我试图以批量加载为指导,实现海量数据转储cassandra示例 在本例中,使用脚本解析依赖项,但我发现cassandra库所涵盖的依赖项不在此处列出的目录中,因为我使用的版本是dse with cassandra 2.0。那么,如果试图覆盖这些依赖项,将得到以下脚本Datastax enterprise cassandra中的批量加载错误,datastax-enterprise,bulkloader,Datastax Enterprise,Bulkloader,早上好 我试图以批量加载为指导,实现海量数据转储cassandra示例 在本例中,使用脚本解析依赖项,但我发现cassandra库所涵盖的依赖项不在此处列出的目录中,因为我使用的版本是dse with cassandra 2.0。那么,如果试图覆盖这些依赖项,将得到以下脚本 #!/bin/sh # paths to the cassandra source tree, cassandra jar and java CASSANDRA_HOME="/usr/share/dse/cassandr
#!/bin/sh
# paths to the cassandra source tree, cassandra jar and java
CASSANDRA_HOME="/usr/share/dse/cassandra"
# CASSANDRA_JAR="./apache-cassandra-2.0.10.jar"
JAVA=`which java`
# Java classpath. Must include:
# - directory of DataImportExample
# - directory with cassandra/log4j config files
# - cassandra jar
# - cassandra depencies jar
CLASSPATH=".:/usr/share/dse/dse.jar:./slf4j-1.7.7/slf4-nop-1.7.7.jar:./slf4j-1.7.7/slf4j-simple-1.7.7.jar:/etc/dse/cassandra"
for jar in $CASSANDRA_HOME/lib/*.jar; do
CLASSPATH=$CLASSPATH:$jar
done
$JAVA -ea -cp $CLASSPATH -Xmx256M \
-Dlog4j.configuration=log4j-tools.properties \
CassandraDataBulk "$@"
CASSANDRA_JAR已被注释,我使用的CASSANDRA-all-2.0.8.39.JAR位于/usr/share/dse/CASSANDRA/lib文件夹中,并且已经包含在内
我在1.7.7版本中解决了slf4j依赖关系
由于cassandra版本的差异,我也不得不习惯SSTableImpleUseredWriter builder
IPartitioner partitioner = new RandomPartitioner();
SSTableSimpleUnsortedWriter sourcesWriter = new SSTableSimpleUnsortedWriter(
directory,
partitioner,
keyspace,
table,
AsciiType.instance,
null,
64
);
今天的问题似乎是仍然存在依赖关系。
下面是我得到的跟踪错误
有一个依赖关系,但似乎是org.apache.commons.configuration.ConfigurationRuntimeException,真正的问题可能是另一个,可能是配置不好cassandra.yaml
谢谢,
问候
[dmdb@vm-dmdb01 ~]$ ./init_env.sh export.csv
[main] ERROR org.apache.cassandra.cql3.QueryProcessor - Unable to initialize MemoryMeter (jamm not specified as javaagent). This means Cassandra will be unable to measure object sizes accurately and may consequently OOM.
[main] INFO org.apache.cassandra.config.YamlConfigurationLoader - Loading settings from file:/etc/dse/cassandra/cassandra.yaml
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Data files directories: [/data01, /data02]
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Commit log directory: /datatmp/commitlog
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - disk_failure_policy is stop
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - commit_failure_policy is stop
[main] INFO org.apache.cassandra.config.DatabaseDescriptor - Global memtable threshold is enabled at 61MB
[main] INFO com.datastax.bdp.snitch.Workload - Setting my workload to Cassandra
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationRuntimeException
at com.datastax.bdp.config.ConfigUtil.defaultValue(ConfigUtil.java:18)
at com.datastax.bdp.config.DseConfig.<clinit>(DseConfig.java:51)
at com.datastax.bdp.snitch.DseDelegateSnitch.<init>(DseDelegateSnitch.java:42)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:488)
at org.apache.cassandra.config.DatabaseDescriptor.createEndpointSnitch(DatabaseDescriptor.java:508)
at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:341)
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:111)
at org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.<init>(AbstractSSTableSimpleWriter.java:50)
at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:96)
at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:80)
at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter.<init>(SSTableSimpleUnsortedWriter.java:91)
at CassandraDataBulk.main(CassandraDataBulk.java:35)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.ConfigurationRuntimeException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 17 more
java调用中缺少javaagent参数。添加以下内容:
-javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar
您的最后一次呼叫应该如下所示:
$JAVA -ea -cp $CLASSPATH -Xmx256M \
-Dlog4j.configuration=log4j-tools.properties \
-javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar
CassandraDataBulk "$@"
注意:根据需要调整jamm.jar的路径
至于运行时配置错误,请下载ApacheCommons“lang”库并将其包含到类路径中
如果在实现修复后收到新的异常,请下载google-common.jar和guava-16.0.1.jar,并将它们也包含到类路径中。这些是我自己的散装装载机到目前为止所需的所有罐子。谢谢!此解决方案修复了我遇到的第一个错误。[main]错误org.apache.cassandra.cql3.QueryProcessor-无法初始化未指定为javaagent的MemoryMeter jamm。这意味着Cassandra将无法准确测量对象大小,因此可能会出现问题。但不幸的是,我仍然像昨天一样有同样的例外,而另一个例外是什么?我在你原来的帖子中只看到一个堆栈跟踪。编辑:无需担心。我意识到异常跟踪与jamm警告是一个独立的问题。当我找到解决方案时,我将编辑我的答案,并为我答案的例外添加解决方案。测试它是否有效我并没有亲自测试它,但这是我从自己编写批量加载程序的经验中所记得的。我唯一的问题是依赖项,我应用的解决方案是下载并将所有依赖项添加到我的类路径中。在这个网页上我可以找到所有必要的东西