Java 缓冲随机存取文件

RandomAccessFile对文件的随机访问速度非常慢。您经常阅读有关在其上实现缓冲层的内容，但在线找不到这样做的代码

所以我的问题是：你们知道这个类的任何开源实现，会共享一个指针还是共享你们自己的实现

如果这个问题能够成为关于这个问题的有用链接和代码的集合，那就太好了。我相信，很多人都分享了这个问题，SUN也从来没有正确地解决过这个问题

---

请不要参考MemoryMapping，因为文件可能比Integer.MAX\u值大很多。

好吧，我看不出有理由不使用java.nio.MappedByteBuffer，即使文件比Integer.MAX\u值大

显然，不允许为整个文件定义一个MappedByteBuffer。但是您可以让多个MappedByteBuffer访问文件的不同区域

filechannel.map中的位置和大小定义为long类型，这意味着您可以提供大于Integer.MAX\u值的值，唯一需要注意的是缓冲区的大小将不会大于Integer.MAX\u值

因此，您可以这样定义多个贴图：

buffer[0] = fileChannel.map(FileChannel.MapMode.READ_WRITE,0,2147483647L);
buffer[1] = fileChannel.map(FileChannel.MapMode.READ_WRITE,2147483647L, Integer.MAX_VALUE);
buffer[2] = fileChannel.map(FileChannel.MapMode.READ_WRITE, 4294967294L, Integer.MAX_VALUE);
...

总之，大小不能大于Integer.MAX_值，但起始位置可以在文件中的任何位置

在书中，作者Ron Hitchens指出：

通过 内存映射机制可以是远程的 比阅读或写作更有效率 通过常规方法获取数据，即使 使用频道。没有明确的制度 需要打电话，可以 费时的更重要的是 操作系统的虚拟存储系统 系统自动缓存内存 页。这些页面将被缓存 正在使用系统内存，并且不会 消耗JVM内存中的空间 堆

一旦内存页变得有效 （从磁盘引入），可以 以全硬件速度再次访问 不需要再做一次 系统调用以获取数据。大,， 包含索引的结构化文件 或引用的其他章节 或者经常更新可以从中受益 极大地依赖于内存映射。什么时候 与文件锁定相结合以保护 临界截面与控制 事务原子性，您开始 了解如何使用内存映射缓冲区 善用

我真的怀疑你会发现第三方API做得比这更好。也许您会发现在这个架构之上编写了一个API来简化工作

---

你不认为这种方法应该适合你吗？

如果你在64位机器上运行，那么内存映射文件是最好的方法。只需将整个文件映射到一个大小相等的缓冲区数组中，然后根据需要为每个记录选择一个缓冲区（即edalorzo的答案，但是您需要重叠的缓冲区，这样就不会有跨越边界的记录）

如果您在32位JVM上运行，那么您将不得不使用

RandomAccessFile

。但是，您可以使用它读取包含整个记录的

byte[]

，然后使用

ByteBuffer

从该数组中检索单个值。在最坏的情况下，您应该需要进行两次文件访问：一次用于检索记录的位置/大小，另一次用于检索记录本身

但是，请注意，如果创建大量的

byte[]

s，您可能会开始对垃圾收集器施加压力，如果在整个文件中跳出，您将保持IO绑定

RandomAccessFile对文件的随机访问速度非常慢。您经常阅读有关在其上实现缓冲层的内容，但在线找不到这样做的代码

嗯，可以在网上找到。
首先，jpeg2000中的JAI源代码有一个实现，以及一个更无障碍的impl，位于：

javadocs：

---

您可以使用以下代码从RandomAccessFile创建BufferedInputStream：

 RandomAccessFile raf = ...
 FileInputStream fis = new FileInputStream(raf.getFD());
 BufferedInputStream bis = new BufferedInputStream(fis);

一些需要注意的事情

关闭FileInputStream将关闭RandomAccessFile，反之亦然

RandomAccessFile和FileInputStream指向同一位置，因此从FileInputStream读取将推进RandomAccessFile的文件指针，反之亦然

也许你想用这种方式

RandomAccessFile raf = ...
FileInputStream fis = new FileInputStream(raf.getFD());
BufferedInputStream bis = new BufferedInputStream(fis);

//do some reads with buffer
bis.read(...);
bis.read(...);

//seek to a a different section of the file, so discard the previous buffer
raf.seek(...);
bis = new BufferedInputStream(fis);
bis.read(...);
bis.read(...);

导入java.io.File；
导入java.io.FileNotFoundException；
导入java.io.IOException；
导入java.io.RandomAccessFile；
/**
*向随机访问文件添加缓存。
* 
*而不是直接写到磁盘或系统中
*随机访问文件/文件通道的作用是，添加一个小缓冲区并从
*如果可能的话。创建一个缓冲区，这意味着在附近进行读取或写入
*彼此将有一个加速。不在缓存块内的读/写操作
*不会被加速。
* 
*
*/
公共类BufferedRandomAccessFile实现自动关闭{
私有静态final int DEFAULT_BUFSIZE=4096；
/**
*包装后的随机访问文件，我们将在其周围保留一个缓存。
*/
私人最终文件raf；
/**
*缓冲区的大小
*/
私人最终int bufsize；
/**
*缓冲区。
*/
私有最终字节buf[]；
/**
*文件中的当前位置。
*/
私人长pos=0；
/**
*读取缓冲区后，这会告诉我们缓冲区在文件中的位置
*开始于。
*/
私有长bufBlockStart=long.MAX_值；
//必须在写入文件时更新
私有长实际文件长度=-1；
布尔changeMadeToBuffer=false；
//必须在写入缓冲区时更新。
私有长虚拟文件长度=-1；
公共BufferedRandomAccessFile（文件名，字符串模式）引发FileNotFoundException{
这（名称、模式、默认大小）；
}
/**
* 
*@param文件
*@param mode如何进行o
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;

/**
 * Adds caching to a random access file.
 * 
 * Rather than directly writing down to disk or to the system which seems to be
 * what random access file/file channel do, add a small buffer and write/read from
 * it when possible. A single buffer is created, which means reads or writes near 
 * each other will have a speed up. Read/writes that are not within the cache block 
 * will not be speed up. 
 * 
 *
 */
public class BufferedRandomAccessFile implements AutoCloseable {

    private static final int DEFAULT_BUFSIZE = 4096;

    /**
     * The wrapped random access file, we will hold a cache around it.
     */
    private final RandomAccessFile raf;

    /**
     * The size of the buffer
     */
    private final int bufsize;

    /**
     * The buffer.
     */
    private final byte buf[];


    /**
     * Current position in the file.
     */
    private long pos = 0;

    /**
     * When the buffer has been read, this tells us where in the file the buffer
     * starts at.
     */
    private long bufBlockStart = Long.MAX_VALUE;


    // Must be updated on write to the file
    private long actualFileLength = -1;

    boolean changeMadeToBuffer = false;

    // Must be update as we write to the buffer.
    private long virtualFileLength = -1;

    public BufferedRandomAccessFile(File name, String mode) throws FileNotFoundException {
        this(name, mode, DEFAULT_BUFSIZE);
    }

    /**
     * 
     * @param file
     * @param mode how to open the random access file.
     * @param b size of the buffer
     * @throws FileNotFoundException
     */
    public BufferedRandomAccessFile(File file, String mode, int b) throws FileNotFoundException {
        this(new RandomAccessFile(file, mode), b);
    }

    public BufferedRandomAccessFile(RandomAccessFile raf) throws FileNotFoundException {
        this(raf, DEFAULT_BUFSIZE);
    }

    public BufferedRandomAccessFile(RandomAccessFile raf, int b) {
        this.raf = raf;
        try {
            this.actualFileLength = raf.length();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        this.virtualFileLength = actualFileLength;
        this.bufsize = b;
        this.buf = new byte[bufsize];
    }

    /**
     * Sets the position of the byte at which the next read/write should occur.
     * 
     * @param pos
     * @throws IOException
     */
    public void seek(long pos) throws IOException{
        this.pos = pos;
    }

    /**
     * Sets the length of the file.
     */
    public void setLength(long fileLength) throws IOException {
        this.raf.setLength(fileLength);
        if(fileLength < virtualFileLength) {
            virtualFileLength = fileLength;
        }
    }

    /**
     * Writes the entire buffer to disk, if needed.
     */
    private void writeBufferToDisk() throws IOException {
        if(!changeMadeToBuffer) return;
        int amountOfBufferToWrite = (int) Math.min((long) bufsize, virtualFileLength - bufBlockStart);
        if(amountOfBufferToWrite > 0) {
            raf.seek(bufBlockStart);
            raf.write(buf, 0, amountOfBufferToWrite);
            this.actualFileLength = virtualFileLength;
        }
        changeMadeToBuffer = false;
    }

    /**
     * Flush the buffer to disk and force a sync.
     */
    public void flush() throws IOException {
        writeBufferToDisk();
        this.raf.getChannel().force(false);
    }

    /**
     * Based on pos, ensures that the buffer is one that contains pos
     * 
     * After this call it will be safe to write to the buffer to update the byte at pos,
     * if this returns true reading of the byte at pos will be valid as a previous write
     * or set length has caused the file to be large enough to have a byte at pos.
     * 
     * @return true if the buffer contains any data that may be read. Data may be read so long as
     * a write or the file has been set to a length that us greater than the current position.
     */
    private boolean readyBuffer() throws IOException {
        boolean isPosOutSideOfBuffer = pos < bufBlockStart || bufBlockStart + bufsize <= pos;

        if (isPosOutSideOfBuffer) {

            writeBufferToDisk();

            // The buffer is always positioned to start at a multiple of a bufsize offset.
            // e.g. for a buf size of 4 the starting positions of buffers can be at 0, 4, 8, 12..
            // Work out where the buffer block should start for the given position. 
            long bufferBlockStart = (pos / bufsize) * bufsize;

            assert bufferBlockStart >= 0;

            // If the file is large enough, read it into the buffer.
            // if the file is not large enough we have nothing to read into the buffer,
            // In both cases the buffer will be ready to have writes made to it.
            if(bufferBlockStart < actualFileLength) {
                raf.seek(bufferBlockStart);
                raf.read(buf);
            }

            bufBlockStart = bufferBlockStart;
        }

        return pos < virtualFileLength;
    }

    /**
     * Reads a byte from the file, returning an integer of 0-255, or -1 if it has reached the end of the file.
     * 
     * @return
     * @throws IOException 
     */
    public int read() throws IOException {
        if(readyBuffer() == false) {
            return -1;
        }
        try {
            return (buf[(int)(pos - bufBlockStart)]) & 0x000000ff ; 
        } finally {
            pos++;
        }
    }

    /**
     * Write a single byte to the file.
     * 
     * @param b
     * @throws IOException
     */
    public void write(byte b) throws IOException {
        readyBuffer(); // ignore result we don't care.
        buf[(int)(pos - bufBlockStart)] = b;
        changeMadeToBuffer = true;
        pos++;
        if(pos > virtualFileLength) {
            virtualFileLength = pos;
        }
    }

    /**
     * Write all given bytes to the random access file at the current possition.
     * 
     */
    public void write(byte[] bytes) throws IOException {
        int writen = 0;
        int bytesToWrite = bytes.length;
        {
            readyBuffer();
            int startPositionInBuffer = (int)(pos - bufBlockStart);
            int lengthToWriteToBuffer = Math.min(bytesToWrite - writen, bufsize - startPositionInBuffer);
            assert  startPositionInBuffer + lengthToWriteToBuffer <= bufsize;

            System.arraycopy(bytes, writen,
                            buf, startPositionInBuffer,
                            lengthToWriteToBuffer);
            pos += lengthToWriteToBuffer;
            if(pos > virtualFileLength) {
                virtualFileLength = pos;
            }
            writen += lengthToWriteToBuffer;
            this.changeMadeToBuffer = true;
        }

        // Just write the rest to the random access file
        if(writen < bytesToWrite) {
            writeBufferToDisk();
            int toWrite = bytesToWrite - writen;
            raf.write(bytes, writen, toWrite);
            pos += toWrite;
            if(pos > virtualFileLength) {
                virtualFileLength = pos;
                actualFileLength = virtualFileLength;
            }
        }
    }

    /**
     * Read up to to the size of bytes,
     * 
     * @return the number of bytes read.
     */
    public int read(byte[] bytes) throws IOException {
        int read = 0;
        int bytesToRead = bytes.length;
        while(read < bytesToRead) {

            //First see if we need to fill the cache
            if(readyBuffer() == false) {
                //No more to read;
                return read;
            }

            //Now read as much as we can (or need from cache and place it
            //in the given byte[]
            int startPositionInBuffer = (int)(pos - bufBlockStart);
            int lengthToReadFromBuffer = Math.min(bytesToRead - read, bufsize - startPositionInBuffer);

            System.arraycopy(buf, startPositionInBuffer, bytes, read, lengthToReadFromBuffer);

            pos += lengthToReadFromBuffer;
            read += lengthToReadFromBuffer;
        }

        return read;
    }

    public void close() throws IOException {
        try {
            this.writeBufferToDisk();
        } finally {
            raf.close();
        }
    }

    /**
     * Gets the length of the file.
     * 
     * @return
     * @throws IOException
     */
    public long length() throws IOException{
        return virtualFileLength;
    }

}