Java 使用SpringXD将数据加载到Hadoop2.2

我正在尝试从多个源创建数据流，并将其加载到hdfs中

为了证明概念，我在同一台机器上使用单节点hadoop安装和Spring XD。Hadoop是与Ambari一起部署的，报告称它是功能正常的HDP2.0。springxd安装在/var/springxd中，环境变量$XD_HOME设置为/var/springxd/XD；Spring的服务器和客户端都使用-hadoopDistro hdp20标志运行。但是，到目前为止，我只能在日志文件中获取这些消息：

18:00:06,346  INFO DeploymentsPathChildrenCache-0 Configuration.deprecation:840 - fs.default.name is deprecated. Instead, use fs.defaultFS
18:00:06,692  WARN DeploymentsPathChildrenCache-0 util.NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18:00:07,944  INFO DeploymentsPathChildrenCache-0 logging.ClasspathLoggingApplicationListener:55 - Application failed to start with classpath: [file:/var/spring-xd/xd/modules/processor/scripts/, file:/var/spring-xd/xd/config/, file:/var/spring-xd/xd/, file:/var/spring-xd/xd/lib/activation-1.1.jar, file:/var/spring-xd/xd/lib/amqp-client-3.2.4.jar, file:/var/spring-xd/xd/lib/aopalliance-1.0.jar, file:/var/spring-xd/xd/lib/args4j-2.0.16.jar, file:/var/spring-xd/xd/lib/asm-3.1.jar, file:/var/spring-xd/xd/lib/aspectjrt-1.6.6.jar, file:/var/spring-xd/xd/lib/aspectjweaver-1.6.6.jar, file:/var/spring-xd/xd/lib/avro-1.7.5.jar, file:/var/spring-xd/xd/lib/cglib-2.2.1-v20090111.jar, file:/var/spring-xd/xd/lib/cloudfoundry-connector-0.9.2.jar, file:/var/spring-xd/xd/lib/com.ibm.jbatch-tck-spi-1.0.jar, file:/var/spring-xd/xd/lib/commons-beanutils-1.7.0.jar, file:/var/spring-xd/xd/lib/commons-beanutils-core-1.8.0.jar, file:/var/spring-xd/xd/lib/commons-cli-1.2.jar, file:/var/spring-xd/xd/lib/commons-codec-1.7.jar, file:/var/spring-xd/xd/lib/commons-collections-3.2.1.jar, file:/var/spring-xd/xd/lib/commons-compress-1.4.1.jar, file:/var/spring-xd/xd/lib/commons-configuration-1.6.jar, file:/var/spring-xd/xd/lib/commons-daemon-1.0.13.jar, file:/var/spring-xd/xd/lib/commons-dbcp-1.2.2.jar, file:/var/spring-xd/xd/lib/commons-digester-1.8.jar, file:/var/spring-xd/xd/lib/commons-el-1.0.jar, file:/var/spring-xd/xd/lib/commons-fileupload-1.2.1.jar, file:/var/spring-xd/xd/lib/commons-httpclient-3.1.jar, file:/var/spring-xd/xd/lib/commons-io-2.4.jar, file:/var/spring-xd/xd/lib/commons-jexl-2.1.1.jar, file:/var/spring-xd/xd/lib/commons-lang-2.6.jar, file:/var/spring-xd/xd/lib/commons-math-2.1.jar, file:/var/spring-xd/xd/lib/commons-net-3.3.jar, file:/var/spring-xd/xd/lib/commons-pool-1.3.jar, file:/var/spring-xd/xd/lib/core-0.9.2.jar, file:/var/spring-xd/xd/lib/curator-client-2.4.1.jar, file:/var/spring-xd/xd/lib/curator-framework-2.4.1.jar, file:/var/spring-xd/xd/lib/curator-recipes-2.4.1.jar, file:/var/spring-xd/xd/lib/ehcache-core-2.3.0.jar, file:/var/spring-xd/xd/lib/freemarker-2.3.15.jar, file:/var/spring-xd/xd/lib/groovy-all-2.2.2.jar, file:/var/spring-xd/xd/lib/guava-14.0.1.jar, file:/var/spring-xd/xd/lib/guice-3.0.jar, file:/var/spring-xd/xd/lib/guice-servlet-3.0.jar, file:/var/spring-xd/xd/lib/hibernate-validator-4.3.1.Final.jar, file:/var/spring-xd/xd/lib/hsqldb-2.3.0.jar, file:/var/spring-xd/xd/lib/jackson-annotations-2.3.0.jar, file:/var/spring-xd/xd/lib/jackson-core-2.3.2.jar, file:/var/spring-xd/xd/lib/jackson-core-asl-1.9.13.jar, file:/var/spring-xd/xd/lib/jackson-databind-2.3.2.jar, file:/var/spring-xd/xd/lib/jackson-mapper-asl-1.9.13.jar, file:/var/spring-xd/xd/lib/javax.batch-api-1.0.jar, file:/var/spring-xd/xd/lib/javax.inject-1.jar, file:/var/spring-xd/xd/lib/javax.mail-1.5.0.jar, file:/var/spring-xd/xd/lib/jboss-logging-3.1.0.CR2.jar, file:/var/spring-xd/xd/lib/jcl-over-slf4j-1.7.6.jar, file:/var/spring-xd/xd/lib/jersey-guice-1.9.jar, file:/var/spring-xd/xd/lib/jersey-test-framework-grizzly2-1.9.jar, file:/var/spring-xd/xd/lib/jettison-1.2.jar, file:/var/spring-xd/xd/lib/joda-time-2.1.jar, file:/var/spring-xd/xd/lib/jolokia-core-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-jmx-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-jvm-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-spring-1.1.5.jar, file:/var/spring-xd/xd/lib/json-path-0.8.1.jar, file:/var/spring-xd/xd/lib/json-simple-1.1.jar, file:/var/spring-xd/xd/lib/json-smart-1.1.1.jar, file:/var/spring-xd/xd/lib/kite-data-core-0.12.1.jar, file:/var/spring-xd/xd/lib/kryo-2.22.jar, file:/var/spring-xd/xd/lib/lettuce-2.3.3.jar, file:/var/spring-xd/xd/lib/log4j-1.2.17.jar, file:/var/spring-xd/xd/lib/mysql-connector-java-5.1.23.jar, file:/var/spring-xd/xd/lib/netty-3.7.0.Final.jar, file:/var/spring-xd/xd/lib/objenesis-1.3.jar, file:/var/spring-xd/xd/lib/opencsv-2.3.jar, file:/var/spring-xd/xd/lib/paranamer-2.3.jar, file:/var/spring-xd/xd/lib/parquet-avro-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-column-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-common-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-encoding-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-format-1.0.0.jar, file:/var/spring-xd/xd/lib/parquet-generator-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-hadoop-1.2.5.jar, file:/var/spring-xd/xd/lib/postgresql-9.2-1002-jdbc4.jar, file:/var/spring-xd/xd/lib/slf4j-api-1.7.6.jar, file:/var/spring-xd/xd/lib/slf4j-log4j12-1.7.6.jar, file:/var/spring-xd/xd/lib/snakeyaml-1.12.jar, file:/var/spring-xd/xd/lib/snappy-java-1.0.5.jar, file:/var/spring-xd/xd/lib/spring-amqp-1.3.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-aop-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-batch-admin-manager-1.3.0.M1.jar, file:/var/spring-xd/xd/lib/spring-batch-admin-resources-1.3.0.M1.jar, file:/var/spring-xd/xd/lib/spring-batch-core-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-batch-infrastructure-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-batch-integration-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-beans-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-actuator-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-autoconfigure-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-context-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-context-support-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-core-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-data-commons-1.6.2.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-data-redis-1.2.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-expression-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-hateoas-0.9.0.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-integration-amqp-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-core-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-event-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-file-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-ftp-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-groovy-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-http-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-jmx-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-redis-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-scripting-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-jdbc-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-messaging-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-plugin-core-0.8.0.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-rabbit-1.3.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-retry-1.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-service-connector-0.9.2.jar, file:/var/spring-xd/xd/lib/spring-tx-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-web-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-webmvc-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-xd-analytics-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-batch-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-dirt-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-hadoop-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-module-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-module-spi-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-rest-domain-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-tuple-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/tomcat-el-api-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-embed-core-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-embed-logging-juli-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-jdbc-7.0.42.jar, file:/var/spring-xd/xd/lib/tomcat-jsp-api-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-juli-7.0.42.jar, file:/var/spring-xd/xd/lib/tomcat-servlet-api-7.0.35.jar, file:/var/spring-xd/xd/lib/uuid-3.2.jar, file:/var/spring-xd/xd/lib/validation-api-1.0.0.GA.jar, file:/var/spring-xd/xd/lib/xmlenc-0.52.jar, file:/var/spring-xd/xd/lib/xmlpull-1.1.3.1.jar, file:/var/spring-xd/xd/lib/xpp3_min-1.1.4c.jar, file:/var/spring-xd/xd/lib/xstream-1.4.4.jar, file:/var/spring-xd/xd/lib/xz-1.0.jar, file:/var/spring-xd/xd/lib/zookeeper-3.4.6.jar, file:/var/spring-xd/xd/lib/hdp20/avro-1.7.5.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-annotations-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-auth-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-distcp-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-hdfs-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-core-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-jobclient-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-shuffle-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-streaming-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-api-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-client-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-server-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-server-nodemanager-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/jetty-util-6.1.26.jar, file:/var/spring-xd/xd/lib/hdp20/protobuf-java-2.5.0.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-batch-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-core-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-store-2.0.0.RC2-hdp20.jar]
18:00:07,944  INFO DeploymentsPathChildrenCache-0 logging.AutoConfigurationReportLoggingInitializer:101 - 

Error starting ApplicationContext. To display the auto-configuration report enabled debug logging (start with --debug)


18:00:07,946 ERROR DeploymentsPathChildrenCache-0 cache.PathChildrenCache:550 - 
java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storeWriter': Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanInitializationException: failed to initialize; nested exception is java.net.ConnectException: Call From CentOS/10.0.0.4 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at org.springframework.xd.dirt.server.ContainerRegistrar.deployStreamModule(ContainerRegistrar.java:448)
    at org.springframework.xd.dirt.server.ContainerRegistrar.onChildAdded(ContainerRegistrar.java:347)

我仍然能够成功地执行以下命令

hadoop fs ls /

从xd shell中成功获取结果

---

我知道SpringXDAPI，但现在想留在XDShell中。任何帮助都将不胜感激

它无法连接到localhost:8020，即名称节点服务。很可能您的hadoop安装没有正确启动。

检查防火墙规则并运行一些标准hadoop命令

XD正在运行Zookeeper，可能会被阻止

使用SpringXD单节点进行测试-它更容易一些，而且没有zookeeper 确保您在HDFS中设置了访问权限，我以前见过这个问题。 如果Hadoop在您桌面上的虚拟机上运行，而SpringXD在您的主机上，那么有时会出现IP/DNS问题来阻止它。

---

对不起，我忘了提。hadoop fs命令在xd shell中仍然可以正常工作。我把它添加到问题中。@JagdCrab:嗯，这看起来很奇怪。如果您使用telnet localhost 8020，它会连接吗？telnet不喜欢它并返回未知主机，但如果我通过浏览器在10.0.0.4:8020上发出http请求，这是此计算机的内部ip。它也是网络中唯一一台我得到正确回复的机器，它是Hadoop IPC端口，我应该使用web接口端口进行http请求。奇怪的是，它无法解析localhost，你可能会问。我想解决这个问题。同时，如果您可以将配置更改为指向10.0.0.4而不是本地主机，那么它将有问题。至少能够连接。在该计算机上打开一个浏览器，可以显示DFS运行状况屏幕。最重要的是，它应该告诉您NameNode的配置地址。如果不是“localhost:8020”，则需要修改$XD_HOME/XD/config/servers.yml并将spring:hadoop:fsUri设置为hdfs://替换为DFS运行状况屏幕上的内容。servers.yml中已经有一个注释掉的条目，只需修改它即可。