Asp.net 更快地创建CSV文件

我创建了一个CSV文件，方法是获取一个数据表，然后在该数据表中循环并写入CSV文件的每一行。我的数据源中通常有大约65000行。从浏览器下载此过程需要几分钟。问题是，在本地和dev上，这不会花费太长时间，但在客户机上，它们通常会超时

有没有更快的方法生成此文件

Function GenerateCSVFile() As String
    Dim stuPro As New studentProvider.StudentProvider
    Dim emailCenter As New EmailCenter
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "")
    Dim strResult As String = ""

    Dim dtStudent As Data.DataTable
    Dim paymentYear As String = ""
    dtStudent = stuPro.generateDataFile()

    If dtStudent.Rows.Count > 0 Then

        Using sw As New System.IO.StreamWriter(Server.MapPath("Temp/" + strFileName + ".csv"))
            Try
                Dim lineValue As String = ""

                lineValue += "Academic Year, StudentID, SSN, First, Middle, Last"

                sw.WriteLine(lineValue)

                For i As Integer = 0 To dtStudent.Rows.Count - 1

                    lineValue = dtStudent.Rows(i)("fy").ToString
                    lineValue += "," & dtStudent.Rows(i)("uniq_stu_id").ToString
                    lineValue += "," & dtStudent.Rows(i)("ssn").ToString
                    lineValue += "," & dtStudent.Rows(i)("fname").ToString
                    lineValue += "," & dtStudent.Rows(i)("mname").ToString
                    lineValue += "," & dtStudent.Rows(i)("lname").ToString
                    sw.WriteLine(lineValue)

                Next
            Catch ex As Exception
                strResult += ex.ToString
            Finally
                sw.Close()
            End Try

        End Using

        Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv"

        If String.IsNullOrEmpty(strResult) Then

            Dim fs As System.IO.FileStream = Nothing

            fs = System.IO.File.Open(Server.MapPath("Temp/" + strFileName + ".csv"), System.IO.FileMode.Open)
            Dim btFile(fs.Length) As Byte
            fs.Read(btFile, 0, fs.Length)
            fs.Close()

            With Response
                .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName)
                .ContentType = "application/octet-stream"
                .BinaryWrite(btFile)
                .End()
            End With
        End If
    Else
        strResult = "No records found for specified academic year"
    End If

    Return strResult
End Function

更新代码

Function GenerateCSVFile() As String
    Dim startDate As Date = Date.Now
    Dim enddate As Date = Nothing
    Dim stuPro As New studentProvider.StudentProvider
    Dim emailCenter As New EmailCenter
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "")
    Dim strResult As String = ""

    Dim dtStudent As Data.DataTable
    Dim paymentYear As String = ""
    dtStudent = stuPro.generateDataFile(Session("VendorID"), txtAcademicYear.Text.Trim)

    If dtStudent.Rows.Count > 0 Then

        With Response

            Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv"
            .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName)
            .ContentType = "application/octet-stream"

            Dim lineValue As StringBuilder = New StringBuilder

            lineValue.Append("Academic Year, StudentID, SSN, First, Middle, Last")

            .Write(lineValue.ToString)

            For i As Integer = 0 To dtStudent.Rows.Count - 1

                lineValue = New StringBuilder
                lineValue.Append(dtStudent.Rows(i)("fy").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("uniq_stu_id").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("ssn").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("fname").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("mname").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("lname").ToString)

                .Write(lineValue.ToString)

            Next
            enddate = Date.Now

            MsgBox(DateDiff(DateInterval.Second, startDate, enddate))

            .End()
        End With
    Else
        strResult = "No records found for specified academic year"
    End If
    Return strResult
End Function

---

您正在写入一个临时文件，读取该文件，然后他们将该文件的内容写入响应。跳过临时文件，一次一行直接写入响应。这将防止浏览器认为您的服务器超时，加快速度，并减少应用程序消耗的内存量

---

之后，研究如何在此web请求上启用缓存，以便在多个用户在短时间内请求CSV时，ASP.NET不必重新创建CSV。

您应该使用StringBuilder而不是串联

有一些选项可以加快速度：

• 不要从StreamWriter写入到响应的文件，直接写入到响应的页面
• 寻找一种使用datareader而不是datatable来循环数据的解决方案，使用这么多的数据可能会更快。它还将降低内存使用率（不会在内存中加载整个表）
• 如果连接多个字符串，请使用StringBuilder，或者在本例中使用String.Join轻松创建整行

String.Join示例：

For Each row in dtStudent.Rows
    Dim line as new List(of String)
    line.Add(row("fy").ToString)
    line.Add(row("uniq_stu_id").ToString)
    line.Add(-etc-)

    Response.Write(String.Join(",", line.ToArray) & vbcrlf )
Next

---

除了@Robert Levy的建议外，还要注意如何使用字符串变量。在以下几行中使用stringbuilder会更好：

        dim sbTemp as new StringBuilder()
        For i As Integer = 0 To dtStudent.Rows.Count - 1

            sbTemp.Append(dtStudent.Rows(i)("fy").ToString)
            sbTemp.Append(",") 
            sbTemp.Append(dtStudent.Rows(i)("uniq_stu_id").ToString)

            'etc
            sw.WriteLine(lineValue)

        Next

---

您可以研究的一件事是生产者/消费者设计模式。这样做可以让一个（或多个）线程向包含需要写入csv文件的数据的队列发送数据，另一个（或多个）线程执行实际写入操作。

我使用什么方法写入响应？Response.write或Response.WriteFile.@guanome-.write（正如您在更新的代码中所做的那样）。这是否解决了超时问题？我使用什么方法写入响应？Response.write或Response.WriteFile.Use Response.write（字符串）用于创建的每一行，并在执行此操作之前设置标题，以便浏览器知道所需的数据类型。我添加了建议的更改，但它仍在运行。生成文件需要1分钟。这是65000多行所能得到的最好结果吗？生成的文件是28MB。A for each循环可能更快，它不像循环那样依赖于行索引，请参阅更新的示例代码。（另外，如果是28MB，下载可能需要一段时间，具体取决于网络速度和服务器位置，但我怀疑它都是本地主机/本地网络）我会尝试为每一个下载一次，但实际下载速度很快，这一切都是在服务器端进行的，需要一段时间。