Amazon web services AWS DynamoDB与Titan的索引不一致

我们在其中一项服务中使用Titan分布式图形数据库。我们有一个使用CassandraDB作为存储后端的工作配置

现在，我们正在使用AWS DynamoDB存储后端运行该服务。如果我们在本地模式下运行它，一切都可以正常工作。然而，在移动到云中的AWS DynamoDB之后，我们观察到一些不确定的问题

我们有在空数据库上运行的单元测试，生成测试数据，执行一些CRUD，然后清理一切。卡桑德拉通过了测试。它们也传递给当地的发电机。不幸的是，它们往往在本地DynamoDB和云中的AWS DynamoDB上失败。通常需要运行几次才能发生故障。发生此类故障后，我们可以在数据库中找到一些重影顶点。看起来索引与图形不同步。在结果中，我们可以在使用索引和不使用索引查询图形时看到不同的结果

日志中还有一个异常。最有趣的是：com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException:在parallelMutate期间被中断：

13:19:12[信息][c.h.c.c.t.TitanTransactionManager][]交易-数据集（） 13:19:12[信息][c.h.c.c.t.TitanTransactionManager][]交易-多贝金（） 13:19:13[信息][c.h.c.c.t.TitanTransactionManager][]交易-文档管理（） 13:19:13[错误][c.h.c.c.h.HystrixDiagnosticLogger][]HYSTRIX:命令类别-v1.CreateCegory失败（com.hybris.caas.category.command）。CreateCategoryCommand@7a77b317); 执行事件：[超时，回退\u缺失] java.util.concurrent.TimeoutException:null 在com.netflix.hystrix.AbstractCommand$9.call（AbstractCommand.java:600）~[hystrix-core-1.5.1.jar:1.5.1] 在com.netflix.hystrix.AbstractCommand$9.call（AbstractCommand.java:580）~[hystrix-core-1.5.1.jar:1.5.1] 在rx.internal.operators.operatornerrorResumeNextViaFunction$4.onError（operatornerrorResumeNextViaFunction.java:139）~[rxjava-1.1.1.jar:1.1.1] 在rx.internal.operators.OperatorDoOnEach$1.onError（OperatorDoOnEach.java:71）[rxjava-1.1.1.jar:1.1.1] 在rx.internal.operators.OperatorDoOnEach$1.onError（OperatorDoOnEach.java:71）[rxjava-1.1.1.jar:1.1.1] 在com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run（AbstractCommand.java:955）[hystrix-core-1.5.1.jar:1.5.1] 在com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call（HystrixContextRunnable.java:41）[hystrix-core-1.5.1.jar:1.5.1] 在com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call（HystrixContextRunnable.java:37）[hystrix-core-1.5.1.jar:1.5.1] 在com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call（DelegatingMDCCallable.java:58）[service-sdk-logging-4.7.0.jar:4.7.0] 在com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run（HystrixContextRunnable.java:57）[hystrix-core-1.5.1.jar:1.5.1] 在com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick（AbstractCommand.java:972）[hystrix-core-1.5.1.jar:1.5.1] 在com.netflix.hystrix.util.HystrixTimer$1.run（HystrixTimer.java:99）[hystrix-core-1.5.1.jar:1.5.1] 在java.util.concurrent.Executors$RunnableAdapter.call（Executors.java:511）[na:1.8.0\u 102] 在java.util.concurrent.FutureTask.runAndReset（FutureTask.java:308）[na:1.8.0\u 102] 在java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301（ScheduledThreadPoolExecutor.java:180）[na:1.8.0102] 在java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run（ScheduledThreadPoolExecutor.java:294）[na:1.8.0102] 位于java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）[na:1.8.0102] 在java.util.concurrent.ThreadPoolExecutor$Worker.run（ThreadPoolExecutor.java:617）[na:1.8.0_102] 在java.lang.Thread.run（Thread.java:745）[na:1.8.0_102] 13:19:13[错误][c.t.t.g.database.StandardTitanGraph][]无法提交事务[9]，因为提交中存在存储异常 com.thinkaurelius.titan.core.TitanException:由于后端异常，无法执行操作 在com.thinkaurelius.titan.diskstorage.util.backinOperation.execute（backinOperation.java:44）~[titan-core-1.0.0.jar:na] 在com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist（CacheTransaction.java:87）~[titan-core-1.0.0.jar:na] 在com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal（CacheTransaction.java:141）~[titan-core-1.0.0.jar:na] 在com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit（CacheTransaction.java:198）~[titan-core-1.0.0.jar:na] 在com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage（BackendTransaction.java:118）~[titan-core-1.0.0.jar:na] 在com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit（StandardTitanGraph.java:718）~[titan-core-1.0.0.jar:na] 在com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit（StandardTitanTx.java:1352）[titan-core-1.0.0.jar:na] 在com.hybris.caas.category.titan.TitanTransactionManager.doCommit（TitanTransactionManager.java:126）[classes/：na] 位于org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit（AbstractPlatformTransactionManager.java:761）[spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] 在org.springframework.transaction.support.AbstractPlatformTransactionManager.commit（AbstractPlatformTransactionManager.java:730）[spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] 在org.springframework.transaction.interceptor.TransactionSpectSupport.commitTransactionAfterReturning（TransactionSpectSupport.java:485）[spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] 位于org.springframework.transaction.interceptor.TransactionSpectSupport.invokeWithinTransaction（Tra 13:19:12 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:12 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:13 [ERROR] [c.h.c.c.h.HystrixDiagnosticLogger] [] HYSTRIX: command category-v1.createCategory failed (com.hybris.caas.category.command.CreateCategoryCommand@7a77b317); execution events: [TIMEOUT, FALLBACK_MISSING] java.util.concurrent.TimeoutException: null at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:600) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:580) ~[hystrix-core-1.5.1.jar:1.5.1] at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:139) ~[rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) [rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:955) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) [hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:972) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99) [hystrix-core-1.5.1.jar:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_102] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_102] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] 13:19:13 [ERROR] [c.t.t.g.database.StandardTitanGraph] [] Could not commit transaction [9] due to storage exception in commit com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:na] at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) [classes/:na] at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1] at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na] ... 40 common frames omitted Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na] ... 41 common frames omitted 13:19:13 [INFO ] [c.h.c.c.c.w.m.TimeoutExceptionMapper] [] Mapping TimeoutException java.util.concurrent.TimeoutException: null at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:600) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:580) ~[hystrix-core-1.5.1.jar:1.5.1] at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:139) ~[rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:955) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) ~[hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) ~[service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:972) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99) ~[hystrix-core-1.5.1.jar:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_102] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_102] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102] 13:19:13 [ERROR] [c.t.t.g.database.StandardTitanGraph] [] Could not commit transaction [9] due to exception com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:na] at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) [classes/:na] at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1] at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na] ... 40 common frames omitted Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na] ... 41 common frames omitted 13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doRollback() 13:19:13 [ERROR] [c.h.c.c.t.TitanTransactionManager] [] Commit exception overridden by rollback exception com.thinkaurelius.titan.core.TitanException: Could not commit transaction due to exception during persistence at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1363) ~[titan-core-1.0.0.jar:na] at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) ~[classes/:na] at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1] at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] Caused by: com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) ~[titan-core-1.0.0.jar:na] ... 34 common frames omitted Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na] ... 40 common frames omitted Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na] ... 41 common frames omitted 13:19:13 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/CategoryCreation.json 13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:14 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/AssignmentCreation.json 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:14 [INFO ] [c.h.c.c.r.impl.utils.ReferenceVertex] [] Creating vertex for com.hybris.caas.category.model.Reference@2accc739[id=NEW,type=product,url=product-service.dev.cf.hybris.com/NEW] 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:14 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/AssignmentCreation.json 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doRollback() 13:19:14 [INFO ] [c.h.c.c.c.DeleteCategoriesCommand] [] [AUDIT [ remove_all_categories ]] [user=User[name=,roles=[hybris.category_delete_all]], tenant=womdev1s] Delete all categories. 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:14 [INFO ] [c.h.c.c.r.i.GraphCategoryRepository] [] Deleting all categories in tenant 'womdev1s' 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:16 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/CategoryCreation.json 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:16 [INFO ] [c.h.c.c.r.impl.utils.CategoryVertex] [] Creating root category vertex for tenant 'womdev1s' 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() ......