C#-OutOfMemoryException在JSON文件中保存列表
我正在尝试保存压力图的流数据。 基本上，我有一个压力矩阵，定义为：C#-OutOfMemoryException在JSON文件中保存列表,c#,.net,json,multithreading,jsonconvert,C#,.net,Json,Multithreading,Jsonconvert,我正在尝试保存压力图的流数据。 基本上，我有一个压力矩阵，定义为： double[,] pressureMatrix = new double[e.Data.GetLength(0), e.Data.GetLength(1)]; 基本上，我每10毫秒就会得到一个pressureMatrix，我想把所有信息保存在一个JSON文件中，以便以后能够重现 我要做的是，首先，写下我称之为标题的内容，以及所有用于录制的设置，如下所示： recordedData.softwareVersion = Asse
double[,] pressureMatrix = new double[e.Data.GetLength(0), e.Data.GetLength(1)];
pressureMatrixrecordedData.softwareVersion = Assembly.GetExecutingAssembly().GetName().Version.Major.ToString() + "." + Assembly.GetExecutingAssembly().GetName().Version.Minor.ToString();
recordedData.calibrationConfiguration = calibrationConfiguration;
recordedData.representationConfiguration = representationSettings;
recordedData.pressureData = new List<PressureMap>();
var json = JsonConvert.SerializeObject(csvRecordedData, Formatting.None);
File.WriteAllText(this.filePath, json);
列表太大
如何处理此问题以保存数据？我想保存24-48小时的信息。您的基本问题是将所有压力图样本保存在内存中，而不是单独写入每个样本，然后允许对其进行垃圾收集。更糟糕的是，您在两个不同的地方执行此操作：
在将字符串写入文件之前，将整个样本列表序列化为JSON字符串JSON
相反，如中所述，在这种情况下，您应该直接对文件进行序列化和反序列化。有关如何执行此操作的说明，请参见和
recordedData.pressureData=newlist（）累积所有压力图样本，然后在每次取样时写入所有样本
更好的解决方案是编写每个示例一次，然后忘记它，但是每个示例都需要嵌套在JSON中的一些容器对象中，这使得如何做到这一点并不明显
public class PressureMap
{
public double[,] PressureMatrix { get; set; }
}
public class CalibrationConfiguration
{
// Data model not included in question
}
public class RepresentationConfiguration
{
// Data model not included in question
}
public class RecordedDataHeader
{
public string SoftwareVersion { get; set; }
public CalibrationConfiguration CalibrationConfiguration { get; set; }
public RepresentationConfiguration RepresentationConfiguration { get; set; }
}
public class RecordedData
{
// Ensure the header is serialized first.
[JsonProperty(Order = 1)]
public RecordedDataHeader RecordedDataHeader { get; set; }
// Ensure the pressure data is serialized last.
[JsonProperty(Order = 2)]
public IEnumerable<PressureMap> PressureData { get; set; }
}
注:
- 此解决方案使用了这样一个事实：当序列化
IEnumerable
时，Json.NET将而不是将可枚举项具体化为列表。相反，它将充分利用惰性计算，并简单地枚举它，编写然后忘记遇到的每一项
- 第一个线程对压力数据进行采样，并将其添加到阻塞集合中
- 第二个线程将阻塞集合包装在
IEnumerable
中，然后将其序列化为RecordedData.PressureData
在序列化过程中，序列化程序将通过IEnumerable
enumerable枚举，将每个文件流式传输到JSON文件，然后继续执行下一个文件——有效地阻塞，直到有一个文件可用为止
- 您需要做一些实验，以确保序列化线程能够“跟上”采样线程，可能是通过在构造期间设置一个。如果没有，您可能需要采取不同的策略
PressureMap GetPressureMap（int count）
应该是您的某种方法（问题中未显示）返回当前压力图样本- 在这种技术中，JSON文件在采样会话期间保持打开状态。如果采样异常终止，文件可能会被截断。我试图通过定期刷新编写器来改善这个问题
- 虽然数据序列化不再需要无限量的内存，但稍后对
记录的数据进行反序列化将压力数据
数组反序列化为具体的列表
。这可能会在下游处理期间导致内存问题
演示小提琴#1
选项2将从JSON文件切换到JSON文件。这样的文件由由换行符分隔的JSON对象序列组成。在您的情况下，您可以使第一个对象包含RecordedDataHeader
信息，随后的对象类型为PressureMap
：
var sampleCount = 100; // Or whatever
var sampleInterval = 10;
var recordedDataHeader = new RecordedDataHeader
{
SoftwareVersion = softwareVersion,
CalibrationConfiguration = calibrationConfiguration,
RepresentationConfiguration = representationConfiguration,
};
var settings = new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver(),
};
// Write the header
Console.WriteLine("Beginning serialization of sample data to {0}.", this.filePath);
using (var stream = new FileStream(this.filePath, FileMode.Create))
{
JsonExtensions.ToNewlineDelimitedJson(stream, new[] { recordedDataHeader });
}
// Write each sample incrementally
for (int i = 0; i < sampleCount; i++)
{
Thread.Sleep(sampleInterval);
Console.WriteLine("Performing sample {0} of {1}", i, sampleCount);
var map = GetPressureMap(i);
using (var stream = new FileStream(this.filePath, FileMode.Append))
{
JsonExtensions.ToNewlineDelimitedJson(stream, new[] { map });
}
}
Console.WriteLine("Finished serialization of sample data to {0}.", this.filePath);
var sampleCount=100；//或者别的什么
var采样间隔=10；
var recordedDataHeader=新的recordedDataHeader
{
SoftwareVersion=SoftwareVersion，
校准配置=校准配置，
RepresentationConfiguration=RepresentationConfiguration，
};
var设置=新的JsonSerializerSettings
{
ContractResolver=新的CamelCasePropertyNamesContractResolver（），
};
//写标题
WriteLine（“开始将样本数据序列化到{0}.”，this.filePath）；
使用（var stream=new FileStream（this.filePath，FileMode.Create））
{
JsonExtensions.ToNewlineDelimitedJson（流，new[]{recordedDataHeader}）；
}
//以增量方式写入每个示例
对于（int i=0；i
使用扩展方法：
public static partial class JsonExtensions
{
// Adapted from the answer to
// https://stackoverflow.com/questions/44787652/serialize-as-ndjson-using-json-net
// by dbc https://stackoverflow.com/users/3744182/dbc
public static void ToNewlineDelimitedJson<T>(Stream stream, IEnumerable<T> items)
{
// Let caller dispose the underlying stream
using (var textWriter = new StreamWriter(stream, new UTF8Encoding(false, true), 1024, true))
{
ToNewlineDelimitedJson(textWriter, items);
}
}
public static void ToNewlineDelimitedJson<T>(TextWriter textWriter, IEnumerable<T> items)
{
var serializer = JsonSerializer.CreateDefault();
foreach (var item in items)
{
// Formatting.None is the default; I set it here for clarity.
using (var writer = new JsonTextWriter(textWriter) { Formatting = Formatting.None, CloseOutput = false })
{
serializer.Serialize(writer, item);
}
// http://specs.okfnlabs.org/ndjson/
// Each JSON text MUST conform to the [RFC7159] standard and MUST be written to the stream followed by the newline character \n (0x0A).
// The newline charater MAY be preceeded by a carriage return \r (0x0D). The JSON texts MUST NOT contain newlines or carriage returns.
textWriter.Write("\n");
}
}
// Adapted from the answer to
// https://stackoverflow.com/questions/29729063/line-delimited-json-serializing-and-de-serializing
// by Yuval Itzchakov https://stackoverflow.com/users/1870803/yuval-itzchakov
public static IEnumerable<TBase> FromNewlineDelimitedJson<TBase, THeader, TRow>(TextReader reader)
where THeader : TBase
where TRow : TBase
{
bool first = true;
using (var jsonReader = new JsonTextReader(reader) { CloseInput = false, SupportMultipleContent = true })
{
var serializer = JsonSerializer.CreateDefault();
while (jsonReader.Read())
{
if (jsonReader.TokenType == JsonToken.Comment)
continue;
if (first)
{
yield return serializer.Deserialize<THeader>(jsonReader);
first = false;
}
else
{
yield return serializer.Deserialize<TRow>(jsonReader);
}
}
}
}
}
公共静态部分类JsonExtensions
{
//根据答案改编为
// https://stackoverflow.com/questions/44787652/serialize-as-ndjson-using-json-net
//由dbc提供https://stackoverflow.com/users/3744182/dbc
公共静态void ToNewlineDelimitedJson（流、IEnumerable项）
{
//让调用者处理底层流
使用（var textWriter=newstreamWriter（流，新UTF8Encoding（false，true），1024，true））
{
ToNewlineDelimitedJson（文本编写器，项目）；
}
}
公共静态void ToNewlineDelimitedJson（TextWriter TextWriter，IEnumerable items）
{
var serializer=JsonSerializer.CreateDefault（）；
foreach（项目中的var项目）
{
//格式设置。无是默认设置；为了清晰起见，我在这里设置它。
使用（var writer=newjsontextwriter（textWriter）{Formatting=Formatting.No
var sampleCount = 100; // Or whatever
var sampleInterval = 10;
var recordedDataHeader = new RecordedDataHeader
{
SoftwareVersion = softwareVersion,
CalibrationConfiguration = calibrationConfiguration,
RepresentationConfiguration = representationConfiguration,
};
var settings = new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver(),
};
// Write the header
Console.WriteLine("Beginning serialization of sample data to {0}.", this.filePath);
using (var stream = new FileStream(this.filePath, FileMode.Create))
{
JsonExtensions.ToNewlineDelimitedJson(stream, new[] { recordedDataHeader });
}
// Write each sample incrementally
for (int i = 0; i < sampleCount; i++)
{
Thread.Sleep(sampleInterval);
Console.WriteLine("Performing sample {0} of {1}", i, sampleCount);
var map = GetPressureMap(i);
using (var stream = new FileStream(this.filePath, FileMode.Append))
{
JsonExtensions.ToNewlineDelimitedJson(stream, new[] { map });
}
}
Console.WriteLine("Finished serialization of sample data to {0}.", this.filePath);
public static partial class JsonExtensions
{
// Adapted from the answer to
// https://stackoverflow.com/questions/44787652/serialize-as-ndjson-using-json-net
// by dbc https://stackoverflow.com/users/3744182/dbc
public static void ToNewlineDelimitedJson<T>(Stream stream, IEnumerable<T> items)
{
// Let caller dispose the underlying stream
using (var textWriter = new StreamWriter(stream, new UTF8Encoding(false, true), 1024, true))
{
ToNewlineDelimitedJson(textWriter, items);
}
}
public static void ToNewlineDelimitedJson<T>(TextWriter textWriter, IEnumerable<T> items)
{
var serializer = JsonSerializer.CreateDefault();
foreach (var item in items)
{
// Formatting.None is the default; I set it here for clarity.
using (var writer = new JsonTextWriter(textWriter) { Formatting = Formatting.None, CloseOutput = false })
{
serializer.Serialize(writer, item);
}
// http://specs.okfnlabs.org/ndjson/
// Each JSON text MUST conform to the [RFC7159] standard and MUST be written to the stream followed by the newline character \n (0x0A).
// The newline charater MAY be preceeded by a carriage return \r (0x0D). The JSON texts MUST NOT contain newlines or carriage returns.
textWriter.Write("\n");
}
}
// Adapted from the answer to
// https://stackoverflow.com/questions/29729063/line-delimited-json-serializing-and-de-serializing
// by Yuval Itzchakov https://stackoverflow.com/users/1870803/yuval-itzchakov
public static IEnumerable<TBase> FromNewlineDelimitedJson<TBase, THeader, TRow>(TextReader reader)
where THeader : TBase
where TRow : TBase
{
bool first = true;
using (var jsonReader = new JsonTextReader(reader) { CloseInput = false, SupportMultipleContent = true })
{
var serializer = JsonSerializer.CreateDefault();
while (jsonReader.Read())
{
if (jsonReader.TokenType == JsonToken.Comment)
continue;
if (first)
{
yield return serializer.Deserialize<THeader>(jsonReader);
first = false;
}
else
{
yield return serializer.Deserialize<TRow>(jsonReader);
}
}
}
}
}
using (var stream = File.OpenRead(filePath))
using (var textReader = new StreamReader(stream))
{
foreach (var obj in JsonExtensions.FromNewlineDelimitedJson<object, RecordedDataHeader, PressureMap>(textReader))
{
if (obj is RecordedDataHeader)
{
var header = (RecordedDataHeader)obj;
// Process the header
Console.WriteLine(JsonConvert.SerializeObject(header));
}
else
{
var row = (PressureMap)obj;
// Process the row.
Console.WriteLine(JsonConvert.SerializeObject(row));
}
}
}