C# 如何从内存映射文件中获取int
我有一个以空格分隔的数字的文件。它的大小约为1Gb，我想从中获取数字。我已经决定使用内存映射文件来快速读取，但我不知道如何做到这一点。我试着做下一步：C# 如何从内存映射文件中获取int,c#,memory-mapped-files,memory-mapping,C#,Memory Mapped Files,Memory Mapping,我有一个以空格分隔的数字的文件。它的大小约为1Gb，我想从中获取数字。我已经决定使用内存映射文件来快速读取，但我不知道如何做到这一点。我试着做下一步： var mmf = MemoryMappedFile.CreateFromFile("test", FileMode.Open, "myFile"); var mmfa = mmf.CreateViewAccessor(0, 0, MemoryMappedFileAccess.Read); var nums = new int[6]; var a
var mmf = MemoryMappedFile.CreateFromFile("test", FileMode.Open, "myFile");
var mmfa = mmf.CreateViewAccessor(0, 0, MemoryMappedFileAccess.Read);
var nums = new int[6];
var a = mmfa.ReadArray<int>(0, nums, 0, 6);
我在网上搜索过，但没有找到任何关于我的问题。仅关于字节数组或进程间通信。您能告诉我如何在num[0]中获取1吗？48=0x30='0'，49=0x31='1' 所以你得到了真正的字符，它们只是ASCII编码的 字符串01包含2个字节，可以放入一个int中，因此您可以在一个int中同时获取它们。如果您想单独获取它们，则需要请求字节数组 编辑：如果需要将01解析为常量1，即从ASCII表示形式转换为二进制形式，则需要换一种方式。我建议 不要使用内存映射文件， 使用StreamReader逐行读取文件参见示例 使用string.split将每行拆分为块 使用string.parse将每个块解析为数字
48=0x30='0'，49=0x31='1' 所以你得到了真正的字符，它们只是ASCII编码的 字符串01包含2个字节，可以放入一个int中，因此您可以在一个int中同时获取它们。如果您想单独获取它们，则需要请求字节数组 编辑：如果需要将01解析为常量1，即从ASCII表示形式转换为二进制形式，则需要换一种方式。我建议 不要使用内存映射文件， 使用StreamReader逐行读取文件参见示例 使用string.split将每行拆分为块 使用string.parse将每个块解析为数字
您需要解析文件内容，将字符转换为数字-类似以下内容：
List<int> nums = new List<int>();
long curPos = 0;
int curV = 0;
bool hasCurV = false;
while (curPos < mmfa.Capacity) {
byte c;
mmfa.Read(curPos++, out c);
if (c == 0) {
break;
}
if (c == 32) {
if (hasCurV) {
nums.Add(curV);
curV = 0;
}
hasCurV = false;
} else {
curV = checked(curV*10 + (int)(c-48));
hasCurV = true;
}
}
if (hasCurV) {
nums.Add(curV);
}
假设mmfa.Capacity是要读取的字符总数，并且文件只包含由空格分隔的数字，即没有结束行或其他空格，则需要解析文件内容，将字符转换为数字-类似于以下内容：
List<int> nums = new List<int>();
long curPos = 0;
int curV = 0;
bool hasCurV = false;
while (curPos < mmfa.Capacity) {
byte c;
mmfa.Read(curPos++, out c);
if (c == 0) {
break;
}
if (c == 32) {
if (hasCurV) {
nums.Add(curV);
curV = 0;
}
hasCurV = false;
} else {
curV = checked(curV*10 + (int)(c-48));
hasCurV = true;
}
}
if (hasCurV) {
nums.Add(curV);
}
假设mmfa.Capacity是要读取的字符总数，并且该文件只包含由空格分隔的数字，即没有结束行或其他空格，则以下示例将以最快的方式从内存映射文件中读取ASCII整数，而不创建任何字符串。MiMo提供的解决方案要慢得多。它的运行速度为5 MB/s，这对您没有多大帮助。MiMo解决方案的最大问题是，它确实会为每个字符调用一个读取方法，这会使性能降低15倍。 我想知道，如果您最初的问题是性能问题，为什么您会接受他的解决方案。使用哑字符串读取器并将字符串解析为整数，可以获得20 MB/s。通过方法调用获取每个字节确实会破坏可能的读取性能
Did read 258.888.890 bytes with 77 MB/s
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.MemoryMappedFiles;
using System.Runtime.InteropServices;
unsafe class Program
{
static void Main(string[] args)
{
new Program().Start();
}
private void Start()
{
var sw = Stopwatch.StartNew();
string fileName = @"C:\Source\BigFile.txt";//@"C:\Source\Numbers.txt";
var file = MemoryMappedFile.CreateFromFile(fileName);
var fileSize = new FileInfo(fileName).Length;
int viewSize = 200 * 100 * 1000;
long offset = 0;
for (; offset < fileSize-viewSize; offset +=viewSize ) // create 200 MB views
{
using (var accessor = file.CreateViewAccessor(offset, viewSize))
{
int unReadBytes = ReadData(accessor, offset);
offset -= unReadBytes;
}
}
using (var rest = file.CreateViewAccessor(offset, fileSize - offset))
{
ReadData(rest, offset);
}
sw.Stop();
Console.WriteLine("Did read {0:N0} bytes with {1:F0} MB/s", fileSize, (fileSize / (1024 * 1024)) / sw.Elapsed.TotalSeconds);
}
List<int> Data = new List<int>();
private int ReadData(MemoryMappedViewAccessor accessor, long offset)
{
using(var safeViewHandle = accessor.SafeMemoryMappedViewHandle)
{
byte* pStart = null;
safeViewHandle.AcquirePointer(ref pStart);
ulong correction = 0;
// needed to correct offset because the view handle does not start at the offset specified in the CreateAccessor call
// This makes AquirePointer nearly useless.
// http://connect.microsoft.com/VisualStudio/feedback/details/537635/no-way-to-determine-internal-offset-used-by-memorymappedviewaccessor-makes-safememorymappedviewhandle-property-unusable
pStart = Helper.Pointer(pStart, offset, out correction);
var len = safeViewHandle.ByteLength - correction;
bool digitFound = false;
int curInt = 0;
byte current =0;
for (ulong i = 0; i < len; i++)
{
current = *(pStart + i);
if (current == (byte)' ' && digitFound)
{
Data.Add(curInt);
// Console.WriteLine("Add {0}", curInt);
digitFound = false;
curInt = 0;
}
else
{
curInt = curInt * 10 + (current - '0');
digitFound = true;
}
}
// scan backwards to find partial read number
int unread = 0;
if (curInt != 0 && digitFound)
{
byte* pEnd = pStart + len;
while (true)
{
pEnd--;
if (*pEnd == (byte)' ' || pEnd == pStart)
{
break;
}
unread++;
}
}
safeViewHandle.ReleasePointer();
return unread;
}
}
public unsafe static class Helper
{
static SYSTEM_INFO info;
static Helper()
{
GetSystemInfo(ref info);
}
public static byte* Pointer(byte *pByte, long offset, out ulong diff)
{
var num = offset % info.dwAllocationGranularity;
diff = (ulong)num; // return difference
byte* tmp_ptr = pByte;
tmp_ptr += num;
return tmp_ptr;
}
[DllImport("kernel32.dll", SetLastError = true)]
internal static extern void GetSystemInfo(ref SYSTEM_INFO lpSystemInfo);
internal struct SYSTEM_INFO
{
internal int dwOemId;
internal int dwPageSize;
internal IntPtr lpMinimumApplicationAddress;
internal IntPtr lpMaximumApplicationAddress;
internal IntPtr dwActiveProcessorMask;
internal int dwNumberOfProcessors;
internal int dwProcessorType;
internal int dwAllocationGranularity;
internal short wProcessorLevel;
internal short wProcessorRevision;
}
}
void GenerateNumbers()
{
using (var file = File.CreateText(@"C:\Source\BigFile.txt"))
{
for (int i = 0; i < 30 * 1000 * 1000; i++)
{
file.Write(i.ToString() + " ");
}
}
}
}
Did read 258.888.890 bytes with 77 MB/s
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.MemoryMappedFiles;
using System.Runtime.InteropServices;
unsafe class Program
{
static void Main(string[] args)
{
new Program().Start();
}
private void Start()
{
var sw = Stopwatch.StartNew();
string fileName = @"C:\Source\BigFile.txt";//@"C:\Source\Numbers.txt";
var file = MemoryMappedFile.CreateFromFile(fileName);
var fileSize = new FileInfo(fileName).Length;
int viewSize = 200 * 100 * 1000;
long offset = 0;
for (; offset < fileSize-viewSize; offset +=viewSize ) // create 200 MB views
{
using (var accessor = file.CreateViewAccessor(offset, viewSize))
{
int unReadBytes = ReadData(accessor, offset);
offset -= unReadBytes;
}
}
using (var rest = file.CreateViewAccessor(offset, fileSize - offset))
{
ReadData(rest, offset);
}
sw.Stop();
Console.WriteLine("Did read {0:N0} bytes with {1:F0} MB/s", fileSize, (fileSize / (1024 * 1024)) / sw.Elapsed.TotalSeconds);
}
List<int> Data = new List<int>();
private int ReadData(MemoryMappedViewAccessor accessor, long offset)
{
using(var safeViewHandle = accessor.SafeMemoryMappedViewHandle)
{
byte* pStart = null;
safeViewHandle.AcquirePointer(ref pStart);
ulong correction = 0;
// needed to correct offset because the view handle does not start at the offset specified in the CreateAccessor call
// This makes AquirePointer nearly useless.
// http://connect.microsoft.com/VisualStudio/feedback/details/537635/no-way-to-determine-internal-offset-used-by-memorymappedviewaccessor-makes-safememorymappedviewhandle-property-unusable
pStart = Helper.Pointer(pStart, offset, out correction);
var len = safeViewHandle.ByteLength - correction;
bool digitFound = false;
int curInt = 0;
byte current =0;
for (ulong i = 0; i < len; i++)
{
current = *(pStart + i);
if (current == (byte)' ' && digitFound)
{
Data.Add(curInt);
// Console.WriteLine("Add {0}", curInt);
digitFound = false;
curInt = 0;
}
else
{
curInt = curInt * 10 + (current - '0');
digitFound = true;
}
}
// scan backwards to find partial read number
int unread = 0;
if (curInt != 0 && digitFound)
{
byte* pEnd = pStart + len;
while (true)
{
pEnd--;
if (*pEnd == (byte)' ' || pEnd == pStart)
{
break;
}
unread++;
}
}
safeViewHandle.ReleasePointer();
return unread;
}
}
public unsafe static class Helper
{
static SYSTEM_INFO info;
static Helper()
{
GetSystemInfo(ref info);
}
public static byte* Pointer(byte *pByte, long offset, out ulong diff)
{
var num = offset % info.dwAllocationGranularity;
diff = (ulong)num; // return difference
byte* tmp_ptr = pByte;
tmp_ptr += num;
return tmp_ptr;
}
[DllImport("kernel32.dll", SetLastError = true)]
internal static extern void GetSystemInfo(ref SYSTEM_INFO lpSystemInfo);
internal struct SYSTEM_INFO
{
internal int dwOemId;
internal int dwPageSize;
internal IntPtr lpMinimumApplicationAddress;
internal IntPtr lpMaximumApplicationAddress;
internal IntPtr dwActiveProcessorMask;
internal int dwNumberOfProcessors;
internal int dwProcessorType;
internal int dwAllocationGranularity;
internal short wProcessorLevel;
internal short wProcessorRevision;
}
}
void GenerateNumbers()
{
using (var file = File.CreateText(@"C:\Source\BigFile.txt"))
{
for (int i = 0; i < 30 * 1000 * 1000; i++)
{
file.Write(i.ToString() + " ");
}
}
}
}
下面的示例将以最快的方式从内存映射文件中读取ASCII整数，而无需创建任何字符串。MiMo提供的解决方案要慢得多。它的运行速度为5 MB/s，这对您没有多大帮助。MiMo解决方案的最大问题是，它确实会为每个字符调用一个读取方法，这会使性能降低15倍。 我想知道，如果您最初的问题是性能问题，为什么您会接受他的解决方案。使用哑字符串读取器并将字符串解析为整数，可以获得20 MB/s。通过方法调用获取每个字节确实会破坏可能的读取性能
Did read 258.888.890 bytes with 77 MB/s
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.MemoryMappedFiles;
using System.Runtime.InteropServices;
unsafe class Program
{
static void Main(string[] args)
{
new Program().Start();
}
private void Start()
{
var sw = Stopwatch.StartNew();
string fileName = @"C:\Source\BigFile.txt";//@"C:\Source\Numbers.txt";
var file = MemoryMappedFile.CreateFromFile(fileName);
var fileSize = new FileInfo(fileName).Length;
int viewSize = 200 * 100 * 1000;
long offset = 0;
for (; offset < fileSize-viewSize; offset +=viewSize ) // create 200 MB views
{
using (var accessor = file.CreateViewAccessor(offset, viewSize))
{
int unReadBytes = ReadData(accessor, offset);
offset -= unReadBytes;
}
}
using (var rest = file.CreateViewAccessor(offset, fileSize - offset))
{
ReadData(rest, offset);
}
sw.Stop();
Console.WriteLine("Did read {0:N0} bytes with {1:F0} MB/s", fileSize, (fileSize / (1024 * 1024)) / sw.Elapsed.TotalSeconds);
}
List<int> Data = new List<int>();
private int ReadData(MemoryMappedViewAccessor accessor, long offset)
{
using(var safeViewHandle = accessor.SafeMemoryMappedViewHandle)
{
byte* pStart = null;
safeViewHandle.AcquirePointer(ref pStart);
ulong correction = 0;
// needed to correct offset because the view handle does not start at the offset specified in the CreateAccessor call
// This makes AquirePointer nearly useless.
// http://connect.microsoft.com/VisualStudio/feedback/details/537635/no-way-to-determine-internal-offset-used-by-memorymappedviewaccessor-makes-safememorymappedviewhandle-property-unusable
pStart = Helper.Pointer(pStart, offset, out correction);
var len = safeViewHandle.ByteLength - correction;
bool digitFound = false;
int curInt = 0;
byte current =0;
for (ulong i = 0; i < len; i++)
{
current = *(pStart + i);
if (current == (byte)' ' && digitFound)
{
Data.Add(curInt);
// Console.WriteLine("Add {0}", curInt);
digitFound = false;
curInt = 0;
}
else
{
curInt = curInt * 10 + (current - '0');
digitFound = true;
}
}
// scan backwards to find partial read number
int unread = 0;
if (curInt != 0 && digitFound)
{
byte* pEnd = pStart + len;
while (true)
{
pEnd--;
if (*pEnd == (byte)' ' || pEnd == pStart)
{
break;
}
unread++;
}
}
safeViewHandle.ReleasePointer();
return unread;
}
}
public unsafe static class Helper
{
static SYSTEM_INFO info;
static Helper()
{
GetSystemInfo(ref info);
}
public static byte* Pointer(byte *pByte, long offset, out ulong diff)
{
var num = offset % info.dwAllocationGranularity;
diff = (ulong)num; // return difference
byte* tmp_ptr = pByte;
tmp_ptr += num;
return tmp_ptr;
}
[DllImport("kernel32.dll", SetLastError = true)]
internal static extern void GetSystemInfo(ref SYSTEM_INFO lpSystemInfo);
internal struct SYSTEM_INFO
{
internal int dwOemId;
internal int dwPageSize;
internal IntPtr lpMinimumApplicationAddress;
internal IntPtr lpMaximumApplicationAddress;
internal IntPtr dwActiveProcessorMask;
internal int dwNumberOfProcessors;
internal int dwProcessorType;
internal int dwAllocationGranularity;
internal short wProcessorLevel;
internal short wProcessorRevision;
}
}
void GenerateNumbers()
{
using (var file = File.CreateText(@"C:\Source\BigFile.txt"))
{
for (int i = 0; i < 30 * 1000 * 1000; i++)
{
file.Write(i.ToString() + " ");
}
}
}
}
Did read 258.888.890 bytes with 77 MB/s
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.MemoryMappedFiles;
using System.Runtime.InteropServices;
unsafe class Program
{
static void Main(string[] args)
{
new Program().Start();
}
private void Start()
{
var sw = Stopwatch.StartNew();
string fileName = @"C:\Source\BigFile.txt";//@"C:\Source\Numbers.txt";
var file = MemoryMappedFile.CreateFromFile(fileName);
var fileSize = new FileInfo(fileName).Length;
int viewSize = 200 * 100 * 1000;
long offset = 0;
for (; offset < fileSize-viewSize; offset +=viewSize ) // create 200 MB views
{
using (var accessor = file.CreateViewAccessor(offset, viewSize))
{
int unReadBytes = ReadData(accessor, offset);
offset -= unReadBytes;
}
}
using (var rest = file.CreateViewAccessor(offset, fileSize - offset))
{
ReadData(rest, offset);
}
sw.Stop();
Console.WriteLine("Did read {0:N0} bytes with {1:F0} MB/s", fileSize, (fileSize / (1024 * 1024)) / sw.Elapsed.TotalSeconds);
}
List<int> Data = new List<int>();
private int ReadData(MemoryMappedViewAccessor accessor, long offset)
{
using(var safeViewHandle = accessor.SafeMemoryMappedViewHandle)
{
byte* pStart = null;
safeViewHandle.AcquirePointer(ref pStart);
ulong correction = 0;
// needed to correct offset because the view handle does not start at the offset specified in the CreateAccessor call
// This makes AquirePointer nearly useless.
// http://connect.microsoft.com/VisualStudio/feedback/details/537635/no-way-to-determine-internal-offset-used-by-memorymappedviewaccessor-makes-safememorymappedviewhandle-property-unusable
pStart = Helper.Pointer(pStart, offset, out correction);
var len = safeViewHandle.ByteLength - correction;
bool digitFound = false;
int curInt = 0;
byte current =0;
for (ulong i = 0; i < len; i++)
{
current = *(pStart + i);
if (current == (byte)' ' && digitFound)
{
Data.Add(curInt);
// Console.WriteLine("Add {0}", curInt);
digitFound = false;
curInt = 0;
}
else
{
curInt = curInt * 10 + (current - '0');
digitFound = true;
}
}
// scan backwards to find partial read number
int unread = 0;
if (curInt != 0 && digitFound)
{
byte* pEnd = pStart + len;
while (true)
{
pEnd--;
if (*pEnd == (byte)' ' || pEnd == pStart)
{
break;
}
unread++;
}
}
safeViewHandle.ReleasePointer();
return unread;
}
}
public unsafe static class Helper
{
static SYSTEM_INFO info;
static Helper()
{
GetSystemInfo(ref info);
}
public static byte* Pointer(byte *pByte, long offset, out ulong diff)
{
var num = offset % info.dwAllocationGranularity;
diff = (ulong)num; // return difference
byte* tmp_ptr = pByte;
tmp_ptr += num;
return tmp_ptr;
}
[DllImport("kernel32.dll", SetLastError = true)]
internal static extern void GetSystemInfo(ref SYSTEM_INFO lpSystemInfo);
internal struct SYSTEM_INFO
{
internal int dwOemId;
internal int dwPageSize;
internal IntPtr lpMinimumApplicationAddress;
internal IntPtr lpMaximumApplicationAddress;
internal IntPtr dwActiveProcessorMask;
internal int dwNumberOfProcessors;
internal int dwProcessorType;
internal int dwAllocationGranularity;
internal short wProcessorLevel;
internal short wProcessorRevision;
}
}
void GenerateNumbers()
{
using (var file = File.CreateText(@"C:\Source\BigFile.txt"))
{
for (int i = 0; i < 30 * 1000 * 1000; i++)
{
file.Write(i.ToString() + " ");
}
}
}
}
实际上，我想知道如何从文件中读取所有数字。@Vasilii:请参阅编辑后的帖子：您需要读取字符，而不是整数。好的，所以无法从文件中获取实际整数？在我得到字节数组后，我将不得不解析它？@Vasilii:对不起，更好的是bytes:因为char在c中有2个字节长，而解析是最简单的选择。您没有整数文件，您有一个字符文件，恰好在“0”到“9”之间，由空格分隔。实际上，我想知道如何读取文件中的所有数字。@Vasilii:请参阅编辑后的文章：您需要读取字符，而不是整数。好的，所以无法从文件中实际获取整数？在我得到字节数组后，我将不得不解析它？@Vasilii:对不起，更好的是bytes:因为char在c中有2个字节长，而解析是最简单的选择。您没有整数文件，您有一个字符文件，其范围恰好在“0”到“9”之间，由空格分隔。如果您的数据是ASCII，则需要先对其进行解析，然后才能将其转换为整数。另一个选项是编写一个转换器，逐行读取您的文件，并将整数作为二进制值写入文件。然后你可以用上面的方法来读取整数。@Alois:nice catch，OP实际上希望将ASCII转换为二进制表示形式。如果速度是您主要关心的问题，那么可以查看以下内容：如果您的数据是ASCII，则需要先对其进行解析，然后才能将其转换为整数。另一个选项是编写一个转换器，该转换器逐行读取文件，并将整数作为二进制值写入文件。然后你可以用上面的方法来读取整数。@Alois:nice catch，OP实际上希望将ASCII转换为二进制表示形式。如果速度是您主要关心的问题，那么看看：您也需要处理换行符：并注意overflows@Vlad：Thank-修复了除新行之外的其他错误-“假设……没有端点或其他空格”溢出异常，在curV=checkedcurV*10+intc-48中@瓦西里鲁佐夫：如果文件中的数字大于2^31-1，则溢出是由checked导致的预期行为-如果不是，则我的代码中存在错误…否。我的数字不超过2000。如果c==0，我就加上；现在你也需要处理换行符了：注意overflows@Vlad：Thank-修复了除新行之外的其他错误-“假设……没有端点或其他空格”溢出异常，在curV=checkedcurV*10+intc-48中@瓦西里鲁佐夫：如果文件中的数字大于2^31-1，则溢出是由checked导致的预期行为-如果不是，则我的代码中存在错误…否。我的数字不超过2000。如果c==0，我就加上；现在它可以工作了。实际上，我认为内存映射文件与常规文件相比不会有任何速度提升。如果速度是主要考虑因素，那么二进制格式将是最佳选择。@Vlad:我已经更新了我的答案，以获得最大的速度。它确实比Mimos解决方案高出11倍。代码很棒。工作真的很快！实际上它符合我的要求。它不会变得更快。如果您大致知道要读取多少数据以最小化列表重新分配，则可以使用固定大小预先分配列表。但是你已经达到了极限，你可以用.NET做什么。。。但是，等等，如果您将文件拆分为两个视图，并使用两个线程读取数据，然后在第二个视图中读取数据，则可以加快速度。这可能会给你带来另一个因素2。实际上，我并不认为内存映射文件与常规文件相比会有任何速度提升。如果速度是主要考虑因素，那么二进制格式将是最佳选择。@Vlad:我已经更新了我的答案，以获得最大的速度。它确实比Mimos解决方案高出11倍。代码很棒。工作真的很快！实际上它符合我的要求。它不会变得更快。如果您大致知道要读取多少数据以最小化列表重新分配，则可以使用固定大小预先分配列表。但是你已经达到了极限，你可以用.NET做什么。。。但是，等等，如果您将文件拆分为两个视图，并使用两个线程读取数据，然后在第二个视图中读取数据，则可以加快速度。这可能会给你另一个因素2。