Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 如何使用JDBC(clickhouse)源在Spark(Spark提交)中写入和读取数据?_Java_Apache Spark_Apache Spark Sql - Fatal编程技术网

Java 如何使用JDBC(clickhouse)源在Spark(Spark提交)中写入和读取数据?

Java 如何使用JDBC(clickhouse)源在Spark(Spark提交)中写入和读取数据?,java,apache-spark,apache-spark-sql,Java,Apache Spark,Apache Spark Sql,我有一个类似的问题 我的问题是,我必须使用spark submit来运行我的java jar文件,这是一个jar文件(java项目),它使用clickhouse JDBC并接受2个应用程序参数(args[0]->write/read args[1]->file path/table_name) 在我的java中,我有 public class Main { public static void main(String[] args) throws SQLException { System

我有一个类似的问题

我的问题是,我必须使用spark submit来运行我的java jar文件,这是一个jar文件(java项目),它使用clickhouse JDBC并接受2个应用程序参数(args[0]->write/read args[1]->file path/table_name)

在我的java中,我有

public class Main {

public static void main(String[] args) throws SQLException {

 System.out.println("TEST ! : " + args[0]);
 System.out.println("TESTEST ! : " + args[1]);


    String url = "jdbc:clickhouse://ipaddr:8123/test";
    ClickHouseProperties properties = new ClickHouseProperties();


    properties.setDatabase("test");
    properties.setUser("user");
    properties.setPassword("1234");


    ClickHouseDataSource dataSource = new ClickHouseDataSource(url, properties);
    String sql = "";



 if(args[0] == "read"){

    //read: args[0] = read , args[1] = DB table name to read
    sql = "SELECT * FROM test." + args[1];

    sqlContext.read.jdbc(url=url, table="test", properties=properties);

 } else if (args[0] == "write"){

    //write: args[0] = write args[1] = file path to be saved
    sql = "";

     DataFrame.write.jdbc(url=url, table="test_write", properties=properties);

 } else {

     System.out.println("args 0 either [read] or [write]");

 }
    System.out.println("sql : " + sql);


    try (ClickHouseConnection conn = dataSource.getConnection();
         ClickHouseStatement stmt = conn.createStatement();
         ResultSet rs = stmt.executeQuery(sql)) {
             //what to write here,,

    }

}
}

我试图在链接的帮助下编写我的java代码,但我无法真正理解代码中发生了什么

  • 关于[DataFrame.write~~]