如何使用spark和java在mysql中插入模型
我是apache spark的新手,我想在java中使用spark将模型类插入mysql数据库 我试过使用下面的代码。但它不能正常工作 型号:如何使用spark和java在mysql中插入模型,java,mysql,apache-spark,apache-spark-sql,Java,Mysql,Apache Spark,Apache Spark Sql,我是apache spark的新手,我想在java中使用spark将模型类插入mysql数据库 我试过使用下面的代码。但它不能正常工作 型号: public class User implements Serializable{ private static final long serialVersionUID = 1L; private int id; private String name; private String email; private String city; priva
public class User implements Serializable{
private static final long serialVersionUID = 1L;
private int id;
private String name;
private String email;
private String city;
private String country;
private String ip;
private static StructType structType = DataTypes.createStructType(new StructField[] {
DataTypes.createStructField("id", DataTypes.IntegerType, false),
DataTypes.createStructField("name", DataTypes.StringType, true),
DataTypes.createStructField("email", DataTypes.StringType, true),
DataTypes.createStructField("city", DataTypes.StringType, true),
DataTypes.createStructField("country", DataTypes.StringType, true),
DataTypes.createStructField("ip", DataTypes.StringType, true)
});
//getters & setters
public class MyApp {
private static final String MYSQL_USERNAME = "root";
private static final String MYSQL_PWD = "";
private static final String MYSQL_CONNECTION_URL = "jdbc:mysql://localhost:3306/sample?user=" + MYSQL_USERNAME + "&password=" + MYSQL_PWD;
private static final JavaSparkContext sc =
new JavaSparkContext(new SparkConf().setAppName("SparkSaveToDb").setMaster("local[*]"));
private static final SQLContext sqlContext = new SQLContext(sc);
public static void main(String[] args) {
//Sample data-frame loaded from a JSON file
try{
List<User> list = new ArrayList<User>();
User us = new User();
us.setId(1);
us.setName("Ravi");
us.setEmail("abc.xyz0@gmail.com");
us.setCity("hyderabad");
us.setCountry("India");
us.setIp("127.0.0.1");
list.add(us);
//Creating RDD
JavaRDD<User> personsRDD = sc.parallelize(list);
DataFrame userDf = sqlContext.createDataFrame(personsRDD, User.class);
//JdbcUtils.saveTable(userDf, MYSQL_CONNECTION_URL, "users", prop);
userDf.write().mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL, "users", new java.util.Properties());
}catch(Exception e){
System.out.println(e);
System.exit(0);
}
}
MyApp:
public class User implements Serializable{
private static final long serialVersionUID = 1L;
private int id;
private String name;
private String email;
private String city;
private String country;
private String ip;
private static StructType structType = DataTypes.createStructType(new StructField[] {
DataTypes.createStructField("id", DataTypes.IntegerType, false),
DataTypes.createStructField("name", DataTypes.StringType, true),
DataTypes.createStructField("email", DataTypes.StringType, true),
DataTypes.createStructField("city", DataTypes.StringType, true),
DataTypes.createStructField("country", DataTypes.StringType, true),
DataTypes.createStructField("ip", DataTypes.StringType, true)
});
//getters & setters
public class MyApp {
private static final String MYSQL_USERNAME = "root";
private static final String MYSQL_PWD = "";
private static final String MYSQL_CONNECTION_URL = "jdbc:mysql://localhost:3306/sample?user=" + MYSQL_USERNAME + "&password=" + MYSQL_PWD;
private static final JavaSparkContext sc =
new JavaSparkContext(new SparkConf().setAppName("SparkSaveToDb").setMaster("local[*]"));
private static final SQLContext sqlContext = new SQLContext(sc);
public static void main(String[] args) {
//Sample data-frame loaded from a JSON file
try{
List<User> list = new ArrayList<User>();
User us = new User();
us.setId(1);
us.setName("Ravi");
us.setEmail("abc.xyz0@gmail.com");
us.setCity("hyderabad");
us.setCountry("India");
us.setIp("127.0.0.1");
list.add(us);
//Creating RDD
JavaRDD<User> personsRDD = sc.parallelize(list);
DataFrame userDf = sqlContext.createDataFrame(personsRDD, User.class);
//JdbcUtils.saveTable(userDf, MYSQL_CONNECTION_URL, "users", prop);
userDf.write().mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL, "users", new java.util.Properties());
}catch(Exception e){
System.out.println(e);
System.exit(0);
}
}
任何人都可以让我知道如何解决这个问题,并插入到mysql数据库。提前感谢首先 如果您的表实现了
自动增量
,如果您的id
如下所示:
id int AUTO_INCREMENT
Insert into mytable (id, ...) values ('hyderabad', ...);
那么您就不需要设置idus.setId(1)代码>,这可能会产生问题,考虑到你已经使用了这个“代码> ID=1 < /代码>,所以当你把<代码>我们。SETID(1);<代码>这是一个冲突,所以不要使用us.setId(1)代码>
否则,您应该首先检查此id是否不存在
秒
第1行“id”列的整数值“hyderabad”不正确
此错误意味着您将'hydrabad'
字符串设置为integer类型的id,因此您正在进行如下操作:
id int AUTO_INCREMENT
Insert into mytable (id, ...) values ('hyderabad', ...);
如果这是您的桌子:
create table mytable(
id int,
....
);
然后,您的查询将如下所示:
id int AUTO_INCREMENT
Insert into mytable (id, ...) values ('hyderabad', ...);
这是不正确的,所以请检查您的插入查询,也许您只是更改了属性的顺序。谢谢您的回复。是的,'id'是'AUTO_INCREMENT',但即使我注释'us.sedId(1)'我也会得到错误'java.sql.SQLException:不正确的整数值:第1行'id'列的'hydrabad',并且该表是新创建的表。