Sqoop导出错误-原因:org.apache.hadoop.mapreduce.lib.input.InvalidInputException:输入路径不存在
我正在开发一个java程序 java程序将数据从配置单元导出到mysql 首先,我编写代码Sqoop导出错误-原因:org.apache.hadoop.mapreduce.lib.input.InvalidInputException:输入路径不存在,hadoop,export,hive,sqoop,Hadoop,Export,Hive,Sqoop,我正在开发一个java程序 java程序将数据从配置单元导出到mysql 首先,我编写代码 ProcessBuilder pb = new ProcessBuilder("sqoop-export", "export", "--connect", "jdbc:mysql://localhost/mydb", "--hadoop-home", "/home/yoonhok/development/hadoop-1.1.1",
ProcessBuilder pb = new ProcessBuilder("sqoop-export", "export",
"--connect", "jdbc:mysql://localhost/mydb",
"--hadoop-home", "/home/yoonhok/development/hadoop-1.1.1",
"--table", "mytable",
"--export-dir", "/user/hive/warehouse/tbl_2",
"--username", "yoonhok",
"--password", "1234");
try {
Process p = pb.start();
if (p.waitFor() != 0) {
System.out.println("Error: sqoop-export failed.");
return false;
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
它工作得很好
但是我学会了在java中使用sqoop的新方法
Sqoop还不支持客户端api
所以我添加了sqoop lib,并编写了sqoop.run()
其次,我用新的方法重新编写代码
String[] str = {"export",
"--connect", "jdbc:mysql://localhost/mydb",
"--hadoop-home", "/home/yoonhok/development/hadoop-1.1.1",
"--table", "mytable",
"--export-dir", "/user/hive/warehouse/tbl_2",
"--username", "yoonhok",
"--password", "1234"
};
if (Sqoop.runTool(str) == 1) {
System.out.println("Error: sqoop-export failed.");
return false;
}
但它没有运行
我错了
13/02/14 16:17:09 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/02/14 17:43:12 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
13/02/14 16:17:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/02/14 16:17:09 INFO tool.CodeGenTool: Beginning code generation
13/02/14 16:17:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1
13/02/14 16:17:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1
13/02/14 16:17:09 INFO orm.CompilationManager: HADOOP_HOME is /home/yoonhok/development/hadoop-1.1.1
Note: /tmp/sqoop-yoonhok/compile/45dd1a113123726796a4ed4ce10c9110/tbl_2.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/02/14 16:17:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yoonhok/compile/45dd1a113123726796a4ed4ce10c9110/tbl_2.jar
13/02/14 16:17:10 INFO mapreduce.ExportJobBase: Beginning export of tbl_2
13/02/14 16:17:10 WARN mapreduce.ExportJobBase: Input path file:/user/hive/warehouse/tbl_2 does not exist
13/02/14 16:17:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/02/14 16:17:11 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-yoonhok/mapred/staging/yoonhok314601126/.staging/job_local_0001
13/02/14 16:17:11 ERROR security.UserGroupInformation: PriviledgedActionException as:yoonhok cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hive/warehouse/tbl_2
13/02/14 16:17:11 ERROR tool.ExportTool: Encountered IOException running export job: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hive/warehouse/tbl_2
我看到环境中没有设置$SQOOP\u CONF\u DIR。
所以我补充说
SQOOP_CONF_DIR=/home/yoonhok/development/SQOOP-1.4.2.bin_hadoop-1.0.0/CONF
在
/etc/环境
然后再试一次,但是错误
13/02/14 16:17:09 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/02/14 16:17:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/02/14 16:17:09 INFO tool.CodeGenTool: Beginning code generation
13/02/14 16:17:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1
13/02/14 16:17:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1
13/02/14 16:17:09 INFO orm.CompilationManager: HADOOP_HOME is /home/yoonhok/development/hadoop-1.1.1
Note: /tmp/sqoop-yoonhok/compile/45dd1a113123726796a4ed4ce10c9110/tbl_2.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/02/14 16:17:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yoonhok/compile/45dd1a113123726796a4ed4ce10c9110/tbl_2.jar
13/02/14 16:17:10 INFO mapreduce.ExportJobBase: Beginning export of tbl_2
13/02/14 16:17:10 WARN mapreduce.ExportJobBase: Input path file:/user/hive/warehouse/tbl_2 does not exist
13/02/14 16:17:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/02/14 16:17:11 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-yoonhok/mapred/staging/yoonhok314601126/.staging/job_local_0001
13/02/14 16:17:11 ERROR security.UserGroupInformation: PriviledgedActionException as:yoonhok cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hive/warehouse/tbl_2
13/02/14 16:17:11 ERROR tool.ExportTool: Encountered IOException running export job: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hive/warehouse/tbl_2
我认为出口导向是个问题
我使用“/user/hive/warehouse/tbl_2”
当我运行“hadoop fs-ls/user/hive/warehouse/”时,表“tbl_2”就存在了
我认为
“输入路径不存在:文件:/user/hive/warehouse/tbl_2”不正常
“输入路径不存在:hdfs:/user/hive/warehouse/tbl_2”正常
但我不知道我该如何修复它
就在我得到提示之前 我编辑了《导出目录》
--export-dir hdfs://localhost:9000/user/hive/warehouse/tbl_2
但是。。。这是错误的。。。T.T
13/02/15 15:17:20 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/02/15 15:17:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/02/15 15:17:20 INFO tool.CodeGenTool: Beginning code generation
13/02/15 15:17:20 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1
13/02/15 15:17:20 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1
13/02/15 15:17:20 INFO orm.CompilationManager: HADOOP_HOME is /home/yoonhok/development/hadoop-1.1.1/libexec/..
Note: /tmp/sqoop-yoonhok/compile/697590ee9b90c022fb8518b8a6f1d86b/tbl_2.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/02/15 15:17:22 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yoonhok/compile/697590ee9b90c022fb8518b8a6f1d86b/tbl_2.jar
13/02/15 15:17:22 INFO mapreduce.ExportJobBase: Beginning export of tbl_2
13/02/15 15:17:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/02/15 15:17:23 INFO input.FileInputFormat: Total input paths to process : 1
13/02/15 15:17:23 INFO input.FileInputFormat: Total input paths to process : 1
13/02/15 15:17:23 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-yoonhok/mapred/staging/yoonhok922915382/.staging/job_local_0001
13/02/15 15:17:23 ERROR security.UserGroupInformation: PriviledgedActionException as:yoonhok cause:java.io.FileNotFoundException: File /user/hive/warehouse/tbl_2/000000_0 does not exist.
13/02/15 15:17:23 ERROR tool.ExportTool: Encountered IOException running export job: java.io.FileNotFoundException: File /user/hive/warehouse/tbl_2/000000_0 does not exist.
当我检查hdfs时
hadoop fs -ls /user/hive/warehouse/tbl_2
或
该文件已存在
-rw-r--r--1 yoonhok supergroup 14029022 2013-02-15 12:16/用户/蜂巢/仓库/tbl_2/000000_0
我尝试在终端中使用shell命令
sqoop-export --connect jdbc:mysql://localhost/detector --table tbl_2 --export-dir hdfs://localhost:9000/user/hive/warehouse/tbl_2 --username yoonhok --password 1234
这是工作
有什么问题吗
我不知道
您能帮助我吗?您需要加载并提供您的Hadoop配置文件。默认情况下,它们是从类路径读取的,但是您可以通过(无需保证)覆盖它。您指的是HADOOP_HOME/conf/hdfs-site.xml、mapred-site.xml等等。。。正确的?
sqoop-export --connect jdbc:mysql://localhost/detector --table tbl_2 --export-dir hdfs://localhost:9000/user/hive/warehouse/tbl_2 --username yoonhok --password 1234