C# 在处理有异步调用和无异步调用之前移动GCS文件
我已经编写了以下代码,这些代码是在云运行的API中调用的。对于通过pubsub通知在GCS bucket中创建的每个文件,该API将被调用一次。今天,我添加了将文件移动到已处理或失败的存储桶的代码,这取决于我的文件是否得到完全处理。但是,查看我的StackDriver日志,在很多情况下,文件似乎会从传入文件夹中删除,并在其余代码有机会读取之前移动到“已处理”。我将看到一行内容是开始读取文件xyz,但紧接着它会抛出一个错误,因为文件被移动了。我的代码最初并没有使用async和await,即使这样,我也看到了相同的行为。知道为什么会这样吗?如果有帮助,我可以附加StackDriver日志C# 在处理有异步调用和无异步调用之前移动GCS文件,c#,async-await,google-cloud-storage,C#,Async Await,Google Cloud Storage,我已经编写了以下代码,这些代码是在云运行的API中调用的。对于通过pubsub通知在GCS bucket中创建的每个文件,该API将被调用一次。今天,我添加了将文件移动到已处理或失败的存储桶的代码,这取决于我的文件是否得到完全处理。但是,查看我的StackDriver日志,在很多情况下,文件似乎会从传入文件夹中删除,并在其余代码有机会读取之前移动到“已处理”。我将看到一行内容是开始读取文件xyz,但紧接着它会抛出一个错误,因为文件被移动了。我的代码最初并没有使用async和await,即使这样,
private async Task ReadMyFile(MyFile file)
{
FileStream downloadStream= null;
StorageClient storageClient = StorageClient.Create();
try
{
var gcsURI = @"gs://" + file.bucket + @"/" + file.fileName;
Console.WriteLine("Beginning to Read My File: " + gcsURI);
using (downloadStream= System.IO.File.Create(file.fileName))
{
storageClient.DownloadObject(file.bucket, file.fileName, downloadStream);
downloadStream.Position = 0;
}
using (var f = System.IO.File.OpenRead(file.fileName))
using (var zip = new ZipArchive(f, ZipArchiveMode.Read))
using (var stream = zip.Entries[0].Open())
{
Console.WriteLine("Successfully downloaded My File: " + gcsURI);
var fna = GetFileAttributesFromFileName(file.fileName);
int type1Counter = 0;
int type2Counter = 0;
int type3Counter = 0;
StringBuilder type1s = new StringBuilder();
StringBuilder type2s = new StringBuilder();
StringBuilder type3Points = new StringBuilder();
using (StreamReader sr = new StreamReader(stream))
{
while ((line = sr.ReadLine()) != null)
{
var tokens = line.Split(',');
int dataType = Convert.ToInt32(tokens[1]);
int oid = Convert.ToInt32(tokens[0]);
DateTime dateTimeUTC = Convert.ToDateTime(tokens[2]);
switch (dataType)
{
case 0:
var type1Line = taName + "|" + bt + "|" + fna.vId + "|" + fna.dd + "|" + dateTimeUTC.ToString("yyyy-MM-dd") + "|" + dateTimeUTC.ToString("HH") + "|" + line.Replace(",", "|");
type1s.AppendLine(type1Line);
type1Counter++;
break;
case 1:
var type2Line = taName + "|" + bt + "|" + fna.vId + "|" + fna.dd + "|" + dateTimeUTC.ToString("yyyy-MM-dd") + "|" + dateTimeUTC.ToString("HH") + "|" + line.Replace(",", "|");
type2s.AppendLine(type2Line);
type2Counter++;
break;
case 2:
var type3_Line = taName + "|" + bt + "|" + fna.vId + "|" + fna.dd + "|" + dateTimeUTC.ToString("yyyy-MM-dd") + "|" + dateTimeUTC.ToString("HH") + "|" + line.Replace(",", "|");
type3Points.AppendLine(type3_Line);
type3Counter++;
break;
detype1:
break;
}
}
}
if (type1Counter > 0)
{
using (var memoryStream = new MemoryStream())
using (var writer = new StreamWriter(memoryStream))
{
writer.Write(type1s);
writer.Flush();
memoryStream.Position = 0;
await storageClient.UploadObjectAsync
(file.readyforingestionbucket,
"f_" + file.fileName,
"text/plain",
memoryStream);
}
}
if (type2Counter > 0)
{
using (var memoryStream = new MemoryStream())
using (var writer = new StreamWriter(memoryStream))
{
writer.Write(type2s);
writer.Flush();
memoryStream.Position = 0;
await storageClient.UploadObjectAsync
(file.readyforingestionbucket,
"n_" + file.fileName,
"text/plain",
memoryStream);
}
}
if (type3Counter > 0)
{
using (var memoryStream = new MemoryStream())
using (var writer = new StreamWriter(memoryStream))
{
writer.Write(type3Points);
writer.Flush();
memoryStream.Position = 0;
await storageClient.UploadObjectAsync
(file.readyforingestionbucket,
"a_" + file.fileName,
"text/plain",
memoryStream);
}
}
// File completed processing, move to processed folder
Console.WriteLine("Copying {0} to processed folder {1}", file.fileName, file.processedBucket);
await storageClient.CopyObjectAsync(file.bucket, file.fileName, file.processedBucket, file.fileName);
Console.WriteLine("Copied {0} to processed folder {1}", file.fileName, file.processedBucket);
Console.WriteLine("Deleting {0}", file.fileName);
await storageClient.DeleteObjectAsync(file.bucket, file.fileName);
Console.WriteLine("Deleted {0}", file.fileName);
}
}
catch (Exception ex)
{
Console.WriteLine("Error reading file: " + file.fileName + " - " + ex.ToString());
Console.WriteLine(ex.InnerException.ToString());
// File failed processing, move to failed folder
Console.WriteLine("Copying {0} to failed folder {1}", file.fileName, file.failedbucket);
var copied = storageClient.CopyObjectAsync(file.bucket, file.fileName, file.failedbucket, file.fileName);
copied.Wait();
Console.WriteLine("Copied {0} to processed folder {1}", file.fileName, file.failedbucket);
Console.WriteLine("Deleting {0}", file.fileName);
storageClient.DeleteObject(file.bucket, file.fileName);
Console.WriteLine("Deleted {0}", file.fileName);
}
finally
{
try
{
downloadStream.Dispose();
// Delete the file to not run out of disk space on the image.
Console.Write("Deleting local file from image: " + file.fileName);
System.IO.File.Delete(file.fileName);
Console.Write("Deleted local file from image: " + file.fileName);
}
catch (Exception ex)
{
Console.WriteLine("Error deleting file: " + file.fileName);
Console.WriteLine(ex.InnerException.ToString());
}
}
}
这是一个调试问题,您应该大量记录这一点(您不确定的每一行),并了解它实际上在做什么。要求我们调试它并不是真正的目的,在日志中,我看到文件在剩下的代码之前被复制和删除。我不明白这是为什么。它是否应该在等待storageClient.UploadObjectAsync调用时等待?我想我已经正确地编写了代码,但没有看到这种行为。这不是调试问题。或者我想一切都是这样。