Scala 卡斯巴'；大量返回对象的问题

Casbah（或mongodb的java驱动程序）似乎在处理大量返回的对象时遇到问题。例如，以下代码段将产生IllegalArgumentException，并且不会返回单个结果（下面是完整的堆栈跟踪）。然而，如果我把“限制（…）”降低到1994年，一切似乎都很好

for (link <- links; query = link $exists true) {
    val group = new HashMap[String, Set[(String, String)]] with MultiMap[String, (String, String)]
    log.find(query, fieldsToGet.result).limit(1996) foreach {

      x => {
        group.addBinding(x.get(link).toString, (x.get("_id").toString(), x.get("eventType").toString))
      }

    }
    allGroups += link -> group
  }

Apr 26, 2011 8:23:40 PM com.mongodb.DBTCPConnector$MyPort error
SEVERE: MyPort.error called
java.lang.IllegalArgumentException: response too long: 1278031173
    at com.mongodb.Response.<init>(Response.java:40)
    at com.mongodb.DBPort.go(DBPort.java:101)
    at com.mongodb.DBPort.go(DBPort.java:66)
    at com.mongodb.DBPort.call(DBPort.java:56)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266)
    at com.mongodb.DBCursor._check(DBCursor.java:309)
    at com.mongodb.DBCursor._hasNext(DBCursor.java:431)
    at com.mongodb.DBCursor.hasNext(DBCursor.java:456)
    at com.mongodb.casbah.MongoCursorBase$class.hasNext(MongoCursor.scala:72)
    at com.mongodb.casbah.MongoCursor.hasNext(MongoCursor.scala:517)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at com.mongodb.casbah.MongoCursor.foreach(MongoCursor.scala:517)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:23)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:20)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at Sequencer$.<init>(Sequencer.scala:20)
    at Sequencer$.<clinit>(Sequencer.scala)
    at Sequencer.main(Sequencer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Exception in thread "main" java.lang.ExceptionInInitializerError
    at Sequencer.main(Sequencer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Caused by: java.lang.IllegalArgumentException: response too long: 1278031173
    at com.mongodb.Response.<init>(Response.java:40)
    at com.mongodb.DBPort.go(DBPort.java:101)
    at com.mongodb.DBPort.go(DBPort.java:66)
    at com.mongodb.DBPort.call(DBPort.java:56)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266)
    at com.mongodb.DBCursor._check(DBCursor.java:309)
    at com.mongodb.DBCursor._hasNext(DBCursor.java:431)
    at com.mongodb.DBCursor.hasNext(DBCursor.java:456)
    at com.mongodb.casbah.MongoCursorBase$class.hasNext(MongoCursor.scala:72)
    at com.mongodb.casbah.MongoCursor.hasNext(MongoCursor.scala:517)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at com.mongodb.casbah.MongoCursor.foreach(MongoCursor.scala:517)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:23)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:20)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at Sequencer$.<init>(Sequencer.scala:20)
    at Sequencer$.<clinit>(Sequencer.scala)
    ... 6 more

它可能是由返回的特定对象引起的吗？或者这是关于卡斯巴的

谢谢，

---

Derek

看起来Java驱动程序正在检查当前响应块是否大于32 MB，然后抛出异常

如果在光标上设置batchSize（文档的数量减少），这将减少数据库中的锁定时间，并返回少于32MB的数据

我将使用batchSize，看看什么最适合您的应用程序

Java驱动程序中的最大值可能应该增加

ByteArrayInputStream bin = new ByteArrayInputStream( b );
_len = Bits.readInt( bin );
if ( _len > ( 32 * 1024 * 1024 ) )
 throw new IllegalArgumentException( "response too long: " + _len );

奇怪的是，你的回复说它返回了约1.19 GB的数据

---

如果您的响应没有那么多数据，则可能表明集合已损坏。

这将在java驱动程序级别。我会调查一下。mongodb和Casbah的哪个版本？我正在使用mongodb 1.8.1。对于casbah，库版本为2.8.1-2.0.2