File io 比fprintf（）更快地写入CSV数据？

我在GPU上进行数值模拟，大部分时间都花在定期将CSV数据写入.dat文件上。有没有比使用fprintf（）更快的方法将数据写入.dat文件？我认为fwrite（）不起作用，因为我需要CSV数据

这是我将数据写入文件以供参考的代码

for(int k = 0;k<gridsize;k++){
    for(int j = 0;j<gridsize;j++){
        fprintf(tempE, "%f,", h_Estate[j*gridsize + k] );
    }
}
fprintf(tempE,"\n");

---

for（int k=0；k将如此大量的数据以文本形式写入磁盘不是明智的选择。文本形式是人类的，但250000个数字对人类眼睛来说毫无意义

我想您需要CSV格式，以便在EXCEL或Matlab中进行进一步统计。您最好在C代码中进行统计，并将结果写入磁盘，磁盘的数据大小应该较小。如果使用Matlab，二进制数据也是可以接受的。在这种情况下，应使用单个fwrite（）而不是数千个fprintf（）

其他解决方案包括使用单独的程序将二进制数据重新格式化为CSV文本格式。
根据底层操作系统如何处理fprintf（），fwrite（）可能会提高一些效率

正如您所指出的，您不能直接执行fwrite（），但可以使用sprintf（）格式化csv文本，然后将其推入一个大缓冲区。当缓冲区满时，您可以fwrite（）整个缓冲区

通常操作系统中文件I/O的实现已经在这样做了，因此fwrite（）可能不会比fprintf（）更有效

正如Eric在回答中指出的那样，保存这些数据最有效的方法是直接使用二进制格式。如果您可以对其进行预处理以减少使用量，那就更好了--

例如，您的数据是否需要完整的浮点精度？能否将其转换为16位定点整数，并在为报告的计算保持足够精度的同时，每32位无符号整数保存两个数据点？如果将其视为一组有符号整数，则16位有符号整数相当于5位精度

如果你对数据进行了进一步的处理，你当然不想使用Excel或MATLAB，因为处理时间将失去控制。如果你在C或C++中开发处理算法，那么二进制数据格式不会是问题。
如果您正在绘制这些数据，图形显示将基本上减少数据的采样，因此您也可以向下处理到类似于10k点的数据，并输出对绘制有意义的统计数据
好吧，不管怎样，这是我的想法。它的目的是在一个更一般的意义上，因为你可能已经解决了你的问题，所以这可能会被其他有类似问题的人阅读

编辑：下面是我运行的一个有趣的测试，完整的可编译源代码

    // what's faster, fwrite or fprintf?

#include <stdio.h>
#include <stdlib.h>

#include <windows.h>

#define HUGE_NUMBER  1000

LARGE_INTEGER   ticksPerSecond;
LARGE_INTEGER   time1;
LARGE_INTEGER   time2;

float floatDiffTime;
const int runs = 1000000;

int main(int argc, char* argv[])
{
    // Get the speed of the CPU
    QueryPerformanceFrequency( &ticksPerSecond );
    printf( "Your computer does %lld ticks per second\n", ticksPerSecond.QuadPart );
    // %lld means type "long long" int, which is the
    // 64 bit int which is what we want here.

    // define some random valued variables to use
    // in the print statements
    int a    = 5;
    double b = 9.2919e92;
    char c   = 'x';
    char * d = "blah blah blah";

    // test start:  open a file to write 
    FILE *outfile = fopen( "testfile.txt", "w" );

    char buf[HUGE_NUMBER];
    int i;
    int index = 0;

    //Test line-by-line fprintf
    // START timing
    QueryPerformanceCounter( &time1 );
    memset(buf,'\0', HUGE_NUMBER);
    for(i=0; i<runs; i++)
    {
        fprintf(outfile, "blah %i %f %c %s\n", a, b, c, d );
    }
    fflush ( outfile );
    fclose( outfile );

    // STOP timing
    QueryPerformanceCounter( &time2 );

    // get the difference between time1 and time2,
    // and that is how long the for loop took to run.
    floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
    printf( "line-by-line fprintf took %f seconds\n", floatDiffTime );

    //Test fprintf
    // START timing
    QueryPerformanceCounter( &time1 );
    memset(buf,'\0', HUGE_NUMBER);
    for(i=0; i<runs; i++)
    {
        sprintf(&buf[index], "blah %i %f %c %s\n", a, b, c, d );
        index += strlen(&buf[index]);
        if(index >= HUGE_NUMBER) {
            fprintf(outfile, "%s", buf );
            index = 0;
            memset(buf,'\0', HUGE_NUMBER);
        }
    }
    fflush ( outfile );
    fclose( outfile );

    // STOP timing
    QueryPerformanceCounter( &time2 );

    // get the difference between time1 and time2,
    // and that is how long the for loop took to run.
    floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
    printf( "fprintf took %f seconds\n", floatDiffTime );

    //Test fwrite
    outfile = fopen( "testfile.txt", "w" );
    index = 0;
    /////////////////////
    // START timing
    QueryPerformanceCounter( &time1 );  
    memset(buf,'\0', HUGE_NUMBER);
    for(i=0; i<runs; i++)
    {
        sprintf(&buf[index], "blah %i %f %c %s\n", a, b, c, d );
        index += strlen(&buf[index]);
        if(index >= HUGE_NUMBER) {
            fwrite( buf, 1, strlen(buf), outfile );
            index = 0;
            //printf("buf size: %d\n", strlen(buf));
            memset(buf,'\0', HUGE_NUMBER);
        }
    }

    fflush(outfile);
    fclose( outfile );
    ////////////////////
    // STOP timing
    QueryPerformanceCounter( &time2 );

    // get the difference between time1 and time2,
    // and that is how long the for loop took to run.
    floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
    printf( "fwrite took %f seconds\n", floatDiffTime );

    //Test WriteFile
    outfile = fopen( "testfile.txt", "w" );
    index = 0;
    DWORD bWritten = 0;
    /////////////////////
    // START timing
    QueryPerformanceCounter( &time1 );  
    memset(buf,'\0', HUGE_NUMBER);
    for(i=0; i<runs; i++)
    {
        sprintf(&buf[index], "blah %i %f %c %s\n", a, b, c, d );
        index += strlen(&buf[index]);
        if(index >= HUGE_NUMBER) {
            WriteFile( outfile, buf, strlen(buf), &bWritten, NULL );
            index = 0;
            //printf("buf size: %d\n", strlen(buf));
            memset(buf,'\0', HUGE_NUMBER);
        }
    }

    fflush(outfile);
    fclose( outfile );
    ////////////////////
    // STOP timing
    QueryPerformanceCounter( &time2 );

    // get the difference between time1 and time2,
    // and that is how long the for loop took to run.
    floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
    printf( "WriteFile took %f seconds\n", floatDiffTime );


    //Test WriteFile
    outfile = fopen( "testfile.txt", "w" );
    index = 0;
    bWritten = 0;
    /////////////////////
    // START timing
    QueryPerformanceCounter( &time1 );  
    memset(buf,'\0', HUGE_NUMBER);
    for(i=0; i<runs; i++)
    {
        sprintf(&buf[index], "blah %i %f %c %s\n", a, b, c, d );
        WriteFile( outfile, buf, strlen(buf), &bWritten, NULL );
        memset(buf,'\0', strlen(buf));
    }

    fflush(outfile);
    fclose( outfile );
    ////////////////////
    // STOP timing
    QueryPerformanceCounter( &time2 );

    // get the difference between time1 and time2,
    // and that is how long the for loop took to run.
    floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
    printf( "WriteFile line-by-line took %f seconds\n", floatDiffTime );


   return 0;    
}

看起来，将大量数据作为字符串进行缓冲，然后发送到fprintf（）（portable）或Windows WriteFile（）（如果使用Windows）调用是处理此问题的最有效方法

编译器命令：

gcc write_speed_test.c -o wspt

编译器版本：

$ gcc -v
Using built-in specs.
Target: i686-w64-mingw32
Configured with: ../gcc44-svn/configure --target=i686-w64-mingw32 --host=i686-w64-mingw32 --disable-multilib --disable-nls --disable-win32-registry --prefix=/mingw32 --with-gmp=/mingw32 --with-mpfr=/mingw32 --enable-languages=c,c++
Thread model: win32
gcc version 4.4.3 (GCC)

您必须写入多少数据？嵌套for循环中60000到250000个浮点值之间的任意位置，具体取决于网格的大小。
$ gcc -v
Using built-in specs.
Target: i686-w64-mingw32
Configured with: ../gcc44-svn/configure --target=i686-w64-mingw32 --host=i686-w64-mingw32 --disable-multilib --disable-nls --disable-win32-registry --prefix=/mingw32 --with-gmp=/mingw32 --with-mpfr=/mingw32 --enable-languages=c,c++
Thread model: win32
gcc version 4.4.3 (GCC)