Java 在配置c3p0连接池的Mysql主从复制中偶尔面临通信链路从失败

我使用Mysql复制驱动程序和c3p0连接池配置了一个主从复制。有时在从机中会遇到以下连接故障问题。在当前设置中，有一个主设备和一个从设备

   org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.TransactionException: JDBC begin transaction failed:
            at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
            at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
            at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
            at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
            at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
            at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
            at com.sun.proxy.$Proxy264.get(Unknown Source)
    /* 
    getSomeDataFromSlave()
    */      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
    Caused by: javax.persistence.PersistenceException: org.hibernate.TransactionException: JDBC begin transaction failed:
            at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
            at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)
            at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1771)
            at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:64)
            at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:170)
            at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
            ... 16 more
    Caused by: org.hibernate.TransactionException: JDBC begin transaction failed:
            at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:76)
            at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162)
            at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1435)
            at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:61)
            ... 18 more
    Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet successfully received from the server was 5,804 milliseconds ago.  The last packet sent successfully to the server was 3,206 milliseconds ago.
            at sun.reflect.GeneratedConstructorAccessor897.newInstance(Unknown Source)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
            at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
            at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:981)
            at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3465)
            at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3365)
            at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3805)
            at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2478)
            at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2625)
            at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2547)
            at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4874)
            at com.mysql.jdbc.MultiHostMySQLConnection.setAutoCommit(MultiHostMySQLConnection.java:2064)
            at sun.reflect.GeneratedMethodAccessor367.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at com.mysql.jdbc.LoadBalancedConnectionProxy.invokeMore(LoadBalancedConnectionProxy.java:484)
            at com.mysql.jdbc.MultiHostConnectionProxy.invoke(MultiHostConnectionProxy.java:452)
            at com.sun.proxy.$Proxy232.setAutoCommit(Unknown Source)
            at com.mysql.jdbc.MultiHostMySQLConnection.setAutoCommit(MultiHostMySQLConnection.java:2064)
            at sun.reflect.GeneratedMethodAccessor367.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at com.mysql.jdbc.ReplicationConnectionProxy.invokeMore(ReplicationConnectionProxy.java:293)
            at com.mysql.jdbc.MultiHostConnectionProxy.invoke(MultiHostConnectionProxy.java:452)
            at com.sun.proxy.$Proxy233.setAutoCommit(Unknown Source)
            at com.mchange.v2.c3p0.impl.NewProxyConnection.setAutoCommit(NewProxyConnection.java:881)
            at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:72)
            ... 21 more
    Caused by: java.net.SocketException: Connection reset
            at java.net.SocketInputStream.read(SocketInputStream.java:196)
            at java.net.SocketInputStream.read(SocketInputStream.java:122)
            at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:100)
            at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:143)
            at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:173)
            at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2954)
            at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3375)
            ... 43 more

以下是当前的配置：

**c3p0 properties:**

db.maxPoolSize=20
db.minPoolSize=10
db.maxConnectionIdleTimeInSec=300
db.idleConnectionTestPeriodInSec=300
db.testConnectionOnCheckin=true
db.testConnectionOnCheckout=true
db.connectionTestQuery=select 1

** DB config **

jdbc.driverClassName=com.mysql.jdbc.ReplicationDriver
jdbc.url=jdbc:mysql:replication://url1,url2/schema

**I have done some c3p0 finer logging following are some traces**
.....
FINER] MBean: com.mchange.v2.c3p0:type=PooledDataSource[z8kflt9j9jerlpms8xe0|8ac49e] registered.

2016-09-13 12:39:51 [localhost-startStop-1] INFO o.s.o.j.LocalContainerEntityManagerFactoryBean -

Building JPA container EntityManagerFactory for persistence unit 'default'

[FINEST] incremented pending_acquires: 1

[FINEST] incremented pending_acquires: 2

[FINEST] incremented pending_acquires: 3

[FINER] com.mchange.v2.resourcepool.BasicResourcePool@37ca3e27 config: [start -> 3; min -> 3; max -> 10; inc -> 3; num_acq_attempts -> 30; acq_attempt_delay -> 1000; check_idle_resources_delay -> 60000; mox_resource_age -> 0; max_idle_time -> 100000; excess_max_idle_time -> 0; destroy_unreturned_resc_time -> 10000; expiration_enforcement_delay -> 2500; break_on_acquisition_failure -> false; debug_store_checkout_exceptions -> true]

[INFO] Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> z8kflt9j9jerlpms8xe0|8ac49e, debugUnreturnedConnectionStackTraces -> true, description -> null, driverClass -> com.mysql.jdbc.ReplicationDriver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> z8kflt9j9jerlpms8xe0|8ac49e, idleConnectionTestPeriod -> 60, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql:replication://url1,url2/schema, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 100, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 10, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> select 1, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 10, usesTraditionalReflectiveProxies -> false ]

[FINER] acquire test -- pool size: 0; target_pool_size: 3; desired target? 1

[FINE] awaitAvailable(): [unknown]

[
....

我的假设是，池中的从属连接从Mysql端关闭，但池中的从属连接仍然没有更新，也没有标记为非活动。假设它是一个活动连接，则应用程序尝试从从属服务器获取数据，但失败。 你知道这里会有什么问题吗？连接池在使用连接之前没有测试从属连接并定期刷新连接，这是否是一个问题

尝试了用于connectionTest的自定义连接类，但没有成功

公共类QueryReplicationConnectionTester扩展了DefaultConnectionTester{

private static final long serialVersionUID = -3450145378350470297L;

/**
 * during testing we need to make sure, that not only master
 * but also the slave connection is used. Therefore we need to set
 * the connection to "readonly" to make sure, that the slave 
 * connection is used.
 * 
 * CAUTION: this will only work for ONE SLAVE ENVIRONMENT, since
 * this does not make sure all slaves are checked.
 */
@Override
public int activeCheckConnection(Connection connection, String arg1, Throwable[] arg2) {

    // Initially set to ok
    int status = CONNECTION_IS_OKAY;

    try {

        // remember state and 
        boolean autoCommit = connection.getAutoCommit();
        boolean readOnly = connection.isReadOnly();

        // switch to slave and check slave
        connection.setReadOnly(true);
        connection.setAutoCommit(false);
        status = super.activeCheckConnection(connection, arg1, arg2);

        // if slave is fine, lets check the master
        if ( status == CONNECTION_IS_OKAY ){
            connection.setReadOnly(false);
            connection.setAutoCommit(autoCommit);
            status = super.activeCheckConnection(connection, arg1, arg2);
        }

        connection.setAutoCommit(autoCommit);
        connection.setReadOnly(readOnly);

    } catch (SQLException e) {
        status = CONNECTION_IS_INVALID;
    }

    // return final state
    return status;
}

}

---

还检查了Mysql日志。我可以看到PreferredTestQuerySelect1被激发到master，但由于某种原因它没有被激发到slave。

如果问题是您认为的问题，那么一个简单的解决方法可能是在不保存时设置ReadFromMaster。然后，您可以使用c3p0的内置DefaultConnectionTester，只要主机可用，连接就可以工作。如果主设备停机，那么连接测试将失败，客户端甚至无法从从属设备获取连接，直到主设备恢复正常。但是，除非所有连接的使用都是只读的，否则这可能就是您想要的行为。如果主控器被按下，C3P0释放到应用程序的连接，则无法知道这些连接是否将用于只读目的，因此应该考虑这些连接断开。在这种情况下，您可以通过复制获得一些负载分配，但是如果主服务器关闭，您无法回切到从属服务器。不过，当从机停机时，您应该回切到主机

如果应用程序对连接的所有使用都是只读的，则可以编写一个调用onAcquire中setReadOnlytrue的。。。方法c3p0将跟踪setReadOnly…（设置只读）的覆盖，并确保客户端看到您设置的值，即使在签入/签出周期之后也是如此。然后，假定默认情况下连接到从机。如果将readFromMasterWhenNoSlaves设置为readFromMasterWhenNoSlaves，则当从机不可用时，应用程序应正确地回切到主设备。请注意，如果客户端使用的连接完全是只读的，则不应将setReadOnlyfalse设置为只读

但是，更可能的情况是，您的客户机不是普遍只读的，因此您应该使用普通的连接测试，而不首先设置只读，而是在设置readFromMasterWhenNoSlaves时使用ReadFromMaster。当主设备停机时，连接将无效，这是应该的，但在从设备出现问题时，连接应该不会失效

我不确定为什么您没有看到针对从机的连接测试，但您可以尝试在自定义ConnectionTester中使用setAutoCommittrue而不是setAutoCommitfalse进行针对从机的测试。但我认为最终，您不会想使用这个连接测试仪，默认的连接测试仪就足够了

您可能还希望将连接属性autoReconnect设置为true

---

注意：我没有使用MySQL的ReplicationDriver，这都是快速阅读的猜测。

我突然想到，但有没有办法确保定期刷新从属连接？您在自定义查询连接测试仪中看到任何问题？