将大型Derby Embed数据库迁移到HSQLDB会抛出java.lang.OutOfMemoryError:java堆空间

我正试图在Spring引导服务中将一个大数据库从derby迁移到HSQLDB，比如在10列的几个表中迁移1.5M个regs。我正在用VisualVM检查；字节和字符占用大量内存。但时间上最大的增量出现在德比级别

有时会在此处抛出错误，但有时会在其他控制器中抛出错误。我不想通过触摸所有文件来添加要重新启动的catchOutofMemory

以下是我的代码版本，块注释显示流程的恢复：

run(){//thread inside static function.
    while(keepMigrating){
        keepMigrating=Migrate();
    }
}
private static boolean Migrate(JdbcTemplate derby,JdbcTemplate hsql){
    int regs = 100000;
    PreparedStatement statement = null;
    ResultSet rs = null;
    PreparedStatement statementHSQL = null;
    try {
        for (String table : tables) {//tables contains all tables to migrate
        //check how many registers left asd asign to cant, if cant is 0 the empty is true.
        PreparedStatement statementUpd[];
            while (!empty) {
                if (trys <= 0) throw new Exception("redo");
                //check how many registers left asd asign to cant, if cant is 0 the empty is true and out of bucle and ready to check next table
                /*
                *Next process resume as:
                *fetch data from derby that hasnt been migrated limited by cant
                *create a batch to insert in hsql
                *create a update for derby
                *create a delete in case someting goes wrong
                *excecute insert and update, if someting in batch fail delete the entry in migrate table
                *reduce regs to get out of migrate method at some ponint.
                */
                statement = derby.getDataSource().getConnection().prepareStatement(
                MessageFormat.format(select_all_migrate_false_and_fetch_cant,table));
                statementUpd = new PreparedStatement[cant];
                ArrayList<String> deleteIds = new ArrayList<>();
                StringBuilder columnNames = new StringBuilder();
                StringBuilder updateSQL = new StringBuilder();
                StringBuilder bindVariables = new StringBuilder();
                try {
                    ResultSetMetaData meta = rs.getMetaData();
                    for (int i = 1; i <= meta.getColumnCount(); i++) {
                        if (!meta.getColumnName(i).equals("MIGRATED")) {
                            if (i > 1) {
                                columnNames.append(", ");
                                bindVariables.append(", ");
                            }
                            columnNames.append(meta.getColumnName(i));
                            bindVariables.append('?');
                        }
                    }
                    String sql = "INSERT INTO " + table.substring(4) + " ("
                            + columnNames
                            + ") VALUES ("
                            + bindVariables
                            + ")";
                    statementHSQL = hsql.getDataSource().getConnection().prepareStatement(sql);
                    HashMap<String, Object> data = new HashMap<>();
                    int row = 0;
                    int lastId = 0;
                    String columnName;
                    while (rs.next()) {
                        for (int i = 1; i <= meta.getColumnCount(); i++) {
                            columnName = meta.getColumnName(i);
                            Object o = rs.getObject(i);
                            statementHSQL.setObject(i, o);
                            if (columnName.equals(mainColumn))
                                deleteIds.add(String.valueOf(o));
                            if (!(meta.getColumnType(i) == 2004)) data.put(columnName, o);
                            if (columnName.equals(mainColumn)) id = rs.getObject(i);
                        }
                        int c = 1;
                        String update = MessageFormat.format("INSERT INTO {0}M ({1}M, MIGRATED) VALUES(?, TRUE)",
                                table.substring(4), mainColumn).replace("\"M", "M\"");//migrated state is saved in other table
                        lastId = Integer.valueOf(String.valueOf(id));
                        statementUpd[row] = derby.getDataSource().getConnection().prepareStatement(update);
                        statementUpd[row].setObject(1, rs.getObject(mainColumn));
                        updateSQL = new StringBuilder();
                        statementHSQL.addBatch();
                        row += 1;
                    }
                    /*
                     * Build delete query in case of inserted values in HSQLDB but not updated in DERBY
                     */
                    StringBuilder builder = new StringBuilder();
                    builder.append("(");
                    int count = 1;
                    for (String s : deleteIds) {
                        if (count > 1) builder.append(", ");
                        builder.append("?");
                        count++;
                    }
                    builder.append(")");
                    String str = builder.toString();
                    String queryDelete = "DELETE FROM " + table.substring(4) + " WHERE " + mainColumn + " IN " + str;
                    PreparedStatement statementHSQLDel = hsql.getDataSource().getConnection().prepareStatement
                            (queryDelete);
                    int c = 1;
                    for (String s : deleteIds) {
                        statementHSQLDel.setObject(c, s);
                        c++;
                    }
                    boolean deletes = statementHSQLDel.execute();
                    statementHSQLDel.close();
                    try {
                        DatabaseUtils.close(statementHSQLDel);
                    } catch (Exception e) {
                        catchOutOfMemory(e);
                    }
                    int[] result = statementHSQL.executeBatch();
                    StringBuilder resultS = new StringBuilder();
                    int stCounter = 0;
                    int stCounterInsert = 0;
                    int stCounterUpdate = 0;
                    String notarydebug;
                    for (int i : result) {
                        int upd = 0;
                        try {
                            if (i == 1) upd = statementUpd[stCounter].executeUpdate();
                        } catch (Exception e) {
                            catchOutOfMemory(e);
                        }
                        stCounterInsert += i;
                        stCounterUpdate += upd;
                        resultS.append(",").append(String.valueOf(i)).append("-").append(String.valueOf(upd));
                        stCounter += 1;
                    }
                    statementHSQL.clearBatch();
                    try {
                        DatabaseUtils.close(statementHSQL);
                    } catch (Exception e) {
                        catchOutOfMemory(e);
                    }
                } catch (SQLException se) {
                    catchOutOfMemory(se);//otherstuff
                } catch (Exception e) {
                    catchOutOfMemory(e);
                }
                try {
                    DatabaseUtils.close(rs);
                    DatabaseUtils.close(statement);
                } catch (Exception e) {
                    catchOutOfMemory(e);
                }
                regs=regs-cant;
            }
        }
    }catch (Exception e) {
        if (e.getMessage().equals("redo")) return true;//end the loop of regs maximun and get out of method.
    }
return false;//end migration succesfully
}
private static int catchOutOfMemory(Throwable e) {
    if (e == null) return 0;
    if (e instanceof OutOfMemoryError) {
        Application.restartBat();
        return 1;
    } else {
        return catchOutOfMemory(e.getCause());
    }
}

但我得到了相同的堆内存消耗：

---

提供的代码中不清楚HSQLDB中的表类型。必须对每个表使用此语句一次，以确保表数据存储在

finename.data

文件中：

SET TABLE tableName TYPE CACHED

报告的批插入顺序不正确。使用以下顺序：

Connection hsqlCon;
PrepareStatement hsqlStm;
hsqlCon = JdbcHSQLDB.getDataSource().getConnection();
hsqlStm = hsqlCon.prepareStatement(sql);

{ // repeat this block until all is finished
    { // repeat for 1000 rows
        hsqlStm.addBatch();
    } 
    hsqlStm.executeBatch(); // after every 1000 rows
} 

hsqlStm.close();
hsqlCon.close();

---

您使用的HSQLDB的URL是什么？您使用的是内存中的URL而不是基于文件的URL吗？bot DERBY和HSQLDB是文件。我看到的另一个问题是，您在使用连接和语句时没有正确关闭它们（尽管您的代码由于其结构很难遵循），这会导致内存中的对象过多。谢谢@Mark，如果一条语句保持打开状态，我会再次逐个检查，我在某些点使用null并找到了一条。您需要在多个事务中执行这种传输，并提交每个事务。HSQLDB将未提交的行保留在内存中，如果数据库很大，内存可能会耗尽。

Connection hsqlCon;
PrepareStatement hsqlStm;
hsqlCon = JdbcHSQLDB.getDataSource().getConnection();
hsqlStm = hsqlCon.prepareStatement(sql);

{ // repeat this block until all is finished
    { // repeat for 1000 rows
        hsqlStm.addBatch();
    } 
    hsqlStm.executeBatch(); // after every 1000 rows
} 

hsqlStm.close();
hsqlCon.close();