Sql SSIS:将记录集写入文件的脚本任务

我正在使用SQL Server Data Tools 2013创建SSIS包。此包有一个执行SQL任务，该任务带有完整的Resultset选项，用于将查询结果推送到Object类型的SSIS变量中

我在脚本任务中使用以下命令获取存储在对象变量中的记录集并将其写入CSV：

    Public Sub Main()

    Dim fileName As String = Dts.Variables("vFileName").Value.ToString
    Dim destinationPath As String = Dts.Variables("vDestinationPath").Value.ToString
    Dim destinationPathAndFileName As String = destinationPath + fileName
    Dim fileContents As String = ""

    Dim oleDB As OleDbDataAdapter = New OleDbDataAdapter()
    Dim table As DataTable = New DataTable()
    Dim rs As System.Object = Dts.Variables("vResultSet").Value

    ' Populate DataTable with vResultSet data
    oleDB.Fill(table, rs)

    ' Loop through columns and concatenate with commas
    For Each col As DataColumn In table.Columns
        fileContents &= col.ColumnName & ","
    Next

    ' Remove final comma from columns string and append line break
    fileContents = fileContents.Substring(0, fileContents.Length - 1)
    fileContents &= Environment.NewLine

    ' Loop through rows and concatenate with commas
    Dim i As Integer
    For Each row As DataRow In table.Rows
        For i = 1 To table.Columns.Count
            fileContents &= row(i - 1).ToString() & ","
        Next

        ' Remove final comma from row string and append line break
        fileContents = fileContents.Substring(0, fileContents.Length - 1)
        fileContents &= Environment.NewLine

    Next

    ' Write all text to destination file. If file exists, this step will overwrite it.
    System.IO.File.WriteAllText(destinationPathAndFileName, fileContents)

    Dts.TaskResult = ScriptResults.Success
End Sub

---

这是可行的，但速度非常慢，将一个14k行数据集写入CSV需要25分钟以上。我不能使用数据流，因为这个过程存在于一个循环中，并且每个要导出的表的元数据都不同。我很确定脚本任务是唯一的选择，但是有没有比循环遍历数据集的每一行更快的方法呢？如果我能提供更多信息，请告诉我。

如果您认为合适，请随时翻译到VB.NET。鉴于我已经为另一个项目编写了这段代码，我将您的请求与我的工作方式混为一谈

传入3个SSIS变量：vFileName、vDestinationPath和vResultSet，Main中的代码将ado记录集转换为DataTable，然后将其添加到DataSet并传递给Persist方法

Persist

具有

的

分隔符的默认参数

此实现根本不尝试处理任何角落案例。它不使用限定符转义文本列，不转义嵌入的限定符，对提要中的换行符执行任何操作，
OleDbDataAdapter
的fill方法中的某些操作会因二进制数据而失败，等等

    public void Main()
    {
        string fileName = Dts.Variables["User::vFileName"].Value.ToString();
        DataSet ds = null;
        DataTable dt = null;
        string outputFolder = Dts.Variables["User::vDestinationPath"].Value.ToString();
        string fileMask = string.Empty;
        string sheetName = string.Empty;
        string outSubFolder = string.Empty;
        string message = string.Empty;
        bool fireAgain = true;
        try
        {

            ds = new DataSet();
            dt = new DataTable();

            System.Data.OleDb.OleDbDataAdapter adapter = new System.Data.OleDb.OleDbDataAdapter();
            adapter.Fill(dt, Dts.Variables["User::vResultSet"].Value);

            string baseFileName = System.IO.Path.GetFileNameWithoutExtension(fileName);
            baseFileName = System.IO.Path.GetFileName(fileName);

            ds.Tables.Add(dt);
            //foreach (DataTable dt in ds.Tables)
            {
                Persist(ds, fileName, outputFolder);
            }
        }
        catch (Exception ex)
        {
            Dts.Events.FireInformation(0, "Data Dumper", string.Format("{0}|{1}", "fileName", fileName), string.Empty, 0, ref fireAgain);
            Dts.Events.FireInformation(0, "Data Dumper", string.Format("{0}|{1}", "outputFolder", outputFolder), string.Empty, 0, ref fireAgain);
            Dts.Events.FireInformation(0, "Data Dumper", string.Format("{0}|{1}", "ExceptionDetails", ex.ToString()), string.Empty, 0, ref fireAgain);
            Dts.Events.FireInformation(0, "Data Dumper", string.Format("{0}|{1}", "InnerExceptionDetails", ex.InnerException), string.Empty, 0, ref fireAgain);
        }

        Dts.TaskResult = (int)ScriptResults.Success;
    }

    public static void Persist(System.Data.DataSet ds, string originalFileName, string outputFolder, string delimiter = "|")
    {
        // Enumerate through all the tables in the dataset
        // Save it out as sub versions of the 
        if (ds == null)
        {
            return;
        }

        string baseFileName = System.IO.Path.GetFileNameWithoutExtension(originalFileName);
        string baseFolder = System.IO.Path.GetDirectoryName(originalFileName);
        System.Collections.Generic.List<string> header = null;            

        foreach (System.Data.DataTable table in ds.Tables)
        {
            string outFilePath = System.IO.Path.Combine(outputFolder, string.Format("{0}.{1}.csv", baseFileName, table.TableName));
            System.Text.Encoding e = System.Text.Encoding.Default;

            if (table.ExtendedProperties.ContainsKey("Unicode") && (bool)table.ExtendedProperties["Unicode"])
            {
                e = System.Text.Encoding.Unicode;
            }

            using (System.IO.StreamWriter file = new System.IO.StreamWriter(System.IO.File.Open(outFilePath, System.IO.FileMode.Create), e))
            {
                table.ExtendedProperties.Add("Path", outFilePath);

                // add header row
                header = new System.Collections.Generic.List<string>(table.Columns.Count);
                foreach (System.Data.DataColumn item in table.Columns)
                {
                    header.Add(item.ColumnName);
                }

                file.WriteLine(string.Join(delimiter, header));

                foreach (System.Data.DataRow row in table.Rows)
                {
                    // TODO: For string based fields, capture the max length
                    IEnumerable<string> fields = (row.ItemArray).Select(field => field.ToString());

                    file.WriteLine(string.Join(delimiter, fields));
                }
            }
        }
    }

为什么最初的速度很慢？
我的假设是，慢可以归结为所有的连接。字符串在.Net中是不可变的，每次向该字符串添加列时，都会创建该字符串的新版本。当我构建行时，我使用String.Join方法将数组中的每个元素压缩成单个字符串。这还简化了附加字段分隔符所需的逻辑

我还立即将当前行写入一个文件，而不是通过调用
writealText
将其转换为VB.NET，而只是为了转储内存。鉴于我已经为另一个项目编写了这段代码，我将您的请求与我的工作方式混为一谈

传入3个SSIS变量：vFileName、vDestinationPath和vResultSet，Main中的代码将ado记录集转换为DataTable，然后将其添加到DataSet并传递给Persist方法
Persist
具有
的
分隔符的默认参数

此实现根本不尝试处理任何角落案例。它不使用限定符转义文本列，不转义嵌入的限定符，对提要中的换行符执行任何操作，
OleDbDataAdapter
的fill方法中的某些操作会因二进制数据而失败，等等

    public void Main()
    {
        string fileName = Dts.Variables["User::vFileName"].Value.ToString();
        DataSet ds = null;
        DataTable dt = null;
        string outputFolder = Dts.Variables["User::vDestinationPath"].Value.ToString();
        string fileMask = string.Empty;
        string sheetName = string.Empty;
        string outSubFolder = string.Empty;
        string message = string.Empty;
        bool fireAgain = true;
        try
        {

            ds = new DataSet();
            dt = new DataTable();

            System.Data.OleDb.OleDbDataAdapter adapter = new System.Data.OleDb.OleDbDataAdapter();
            adapter.Fill(dt, Dts.Variables["User::vResultSet"].Value);

            string baseFileName = System.IO.Path.GetFileNameWithoutExtension(fileName);
            baseFileName = System.IO.Path.GetFileName(fileName);

            ds.Tables.Add(dt);
            //foreach (DataTable dt in ds.Tables)
            {
                Persist(ds, fileName, outputFolder);
            }
        }
        catch (Exception ex)
        {
            Dts.Events.FireInformation(0, "Data Dumper", string.Format("{0}|{1}", "fileName", fileName), string.Empty, 0, ref fireAgain);
            Dts.Events.FireInformation(0, "Data Dumper", string.Format("{0}|{1}", "outputFolder", outputFolder), string.Empty, 0, ref fireAgain);
            Dts.Events.FireInformation(0, "Data Dumper", string.Format("{0}|{1}", "ExceptionDetails", ex.ToString()), string.Empty, 0, ref fireAgain);
            Dts.Events.FireInformation(0, "Data Dumper", string.Format("{0}|{1}", "InnerExceptionDetails", ex.InnerException), string.Empty, 0, ref fireAgain);
        }

        Dts.TaskResult = (int)ScriptResults.Success;
    }

    public static void Persist(System.Data.DataSet ds, string originalFileName, string outputFolder, string delimiter = "|")
    {
        // Enumerate through all the tables in the dataset
        // Save it out as sub versions of the 
        if (ds == null)
        {
            return;
        }

        string baseFileName = System.IO.Path.GetFileNameWithoutExtension(originalFileName);
        string baseFolder = System.IO.Path.GetDirectoryName(originalFileName);
        System.Collections.Generic.List<string> header = null;            

        foreach (System.Data.DataTable table in ds.Tables)
        {
            string outFilePath = System.IO.Path.Combine(outputFolder, string.Format("{0}.{1}.csv", baseFileName, table.TableName));
            System.Text.Encoding e = System.Text.Encoding.Default;

            if (table.ExtendedProperties.ContainsKey("Unicode") && (bool)table.ExtendedProperties["Unicode"])
            {
                e = System.Text.Encoding.Unicode;
            }

            using (System.IO.StreamWriter file = new System.IO.StreamWriter(System.IO.File.Open(outFilePath, System.IO.FileMode.Create), e))
            {
                table.ExtendedProperties.Add("Path", outFilePath);

                // add header row
                header = new System.Collections.Generic.List<string>(table.Columns.Count);
                foreach (System.Data.DataColumn item in table.Columns)
                {
                    header.Add(item.ColumnName);
                }

                file.WriteLine(string.Join(delimiter, header));

                foreach (System.Data.DataRow row in table.Rows)
                {
                    // TODO: For string based fields, capture the max length
                    IEnumerable<string> fields = (row.ItemArray).Select(field => field.ToString());

                    file.WriteLine(string.Join(delimiter, fields));
                }
            }
        }
    }

为什么最初的速度很慢？
我的假设是，慢可以归结为所有的连接。字符串在.Net中是不可变的，每次向该字符串添加列时，都会创建该字符串的新版本。当我构建行时，我使用String.Join方法将数组中的每个元素压缩成单个字符串。这还简化了附加字段分隔符所需的逻辑

我还立即将当前行写入一个文件，而不是通过调用
writealText
将其全部转储，这是@billinkc的VB.NET版本的优秀答案，以防对任何人都有用：

导入系统

Imports System.Data 
Imports System.Math 
Imports System.Collections 
Imports System.Collections.Generic 
Imports Microsoft.SqlServer.Dts.Runtime 
Imports System.Linq 
Imports System.Text 
Imports System.Windows.Forms

Public Sub Main()
    Dim fileName As String = Dts.Variables("User::vFileName").Value.ToString()
    Dim ds As DataSet = Nothing
    Dim dt As DataTable = Nothing
    Dim outputFolder As String = Dts.Variables("User::vDestinationPath").Value.ToString()
    Dim fileMask As String = String.Empty
    Dim sheetName As String = String.Empty
    Dim outSubFolder As String = String.Empty
    Dim message As String = String.Empty
    Dim fireAgain As Boolean = True
    Try

        ds = New DataSet()
        dt = New DataTable()

        Dim adapter As New System.Data.OleDb.OleDbDataAdapter()
        adapter.Fill(dt, Dts.Variables("User::vResultSet").Value)

        Dim baseFileName As String = System.IO.Path.GetFileNameWithoutExtension(fileName)
        baseFileName = System.IO.Path.GetFileName(fileName)

        ds.Tables.Add(dt)
        'foreach (DataTable dt in ds.Tables)
        If True Then
            Persist(ds, fileName, outputFolder)
        End If
    Catch ex As Exception
        Dts.Events.FireInformation(0, "Data Dumper", String.Format("{0}|{1}", "fileName", fileName), String.Empty, 0, fireAgain)
        Dts.Events.FireInformation(0, "Data Dumper", String.Format("{0}|{1}", "outputFolder", outputFolder), String.Empty, 0, fireAgain)
        Dts.Events.FireInformation(0, "Data Dumper", String.Format("{0}|{1}", "ExceptionDetails", ex.ToString()), String.Empty, 0, fireAgain)
        Dts.Events.FireInformation(0, "Data Dumper", String.Format("{0}|{1}", "InnerExceptionDetails", ex.InnerException), String.Empty, 0, fireAgain)
    End Try

    Dts.TaskResult = CInt(ScriptResults.Success)
End Sub

Public Shared Sub Persist(ds As System.Data.DataSet, originalFileName As String, outputFolder As String, Optional delimiter As String = ",")

    ' Enumerate through all the tables in the dataset
    ' Save it out as sub versions of the 
    If ds Is Nothing Then
        Return
    End If

    Dim baseFileName As String = System.IO.Path.GetFileNameWithoutExtension(originalFileName)
    Dim baseFolder As String = System.IO.Path.GetDirectoryName(originalFileName)
    Dim header As System.Collections.Generic.List(Of String) = Nothing

    For Each table As System.Data.DataTable In ds.Tables
        Dim outFilePath As String = System.IO.Path.Combine(outputFolder, String.Format("{0}.csv", baseFileName, table.TableName))
        Dim e As System.Text.Encoding = System.Text.Encoding.[Default]

        If table.ExtendedProperties.ContainsKey("Unicode") AndAlso CBool(table.ExtendedProperties("Unicode")) Then
            e = System.Text.Encoding.Unicode
        End If

        Using file As New System.IO.StreamWriter(System.IO.File.Open(outFilePath, System.IO.FileMode.Create), e)
            table.ExtendedProperties.Add("Path", outFilePath)

            ' add header row
            header = New System.Collections.Generic.List(Of String)(table.Columns.Count)
            For Each item As System.Data.DataColumn In table.Columns
                header.Add(item.ColumnName)
            Next

            file.WriteLine(String.Join(delimiter, header))

            For Each row As System.Data.DataRow In table.Rows
                ' TODO: For string based fields, capture the max length
                Dim fields As IEnumerable(Of String) = (row.ItemArray).[Select](Function(field) field.ToString())

                file.WriteLine(String.Join(delimiter, fields))
            Next
        End Using
    Next
End Sub

这是@billinkc的优秀答案的VB.NET版本，以防对任何人有用：

导入系统

Imports System.Data 
Imports System.Math 
Imports System.Collections 
Imports System.Collections.Generic 
Imports Microsoft.SqlServer.Dts.Runtime 
Imports System.Linq 
Imports System.Text 
Imports System.Windows.Forms

Public Sub Main()
    Dim fileName As String = Dts.Variables("User::vFileName").Value.ToString()
    Dim ds As DataSet = Nothing
    Dim dt As DataTable = Nothing
    Dim outputFolder As String = Dts.Variables("User::vDestinationPath").Value.ToString()
    Dim fileMask As String = String.Empty
    Dim sheetName As String = String.Empty
    Dim outSubFolder As String = String.Empty
    Dim message As String = String.Empty
    Dim fireAgain As Boolean = True
    Try

        ds = New DataSet()
        dt = New DataTable()

        Dim adapter As New System.Data.OleDb.OleDbDataAdapter()
        adapter.Fill(dt, Dts.Variables("User::vResultSet").Value)

        Dim baseFileName As String = System.IO.Path.GetFileNameWithoutExtension(fileName)
        baseFileName = System.IO.Path.GetFileName(fileName)

        ds.Tables.Add(dt)
        'foreach (DataTable dt in ds.Tables)
        If True Then
            Persist(ds, fileName, outputFolder)
        End If
    Catch ex As Exception
        Dts.Events.FireInformation(0, "Data Dumper", String.Format("{0}|{1}", "fileName", fileName), String.Empty, 0, fireAgain)
        Dts.Events.FireInformation(0, "Data Dumper", String.Format("{0}|{1}", "outputFolder", outputFolder), String.Empty, 0, fireAgain)
        Dts.Events.FireInformation(0, "Data Dumper", String.Format("{0}|{1}", "ExceptionDetails", ex.ToString()), String.Empty, 0, fireAgain)
        Dts.Events.FireInformation(0, "Data Dumper", String.Format("{0}|{1}", "InnerExceptionDetails", ex.InnerException), String.Empty, 0, fireAgain)
    End Try

    Dts.TaskResult = CInt(ScriptResults.Success)
End Sub

Public Shared Sub Persist(ds As System.Data.DataSet, originalFileName As String, outputFolder As String, Optional delimiter As String = ",")

    ' Enumerate through all the tables in the dataset
    ' Save it out as sub versions of the 
    If ds Is Nothing Then
        Return
    End If

    Dim baseFileName As String = System.IO.Path.GetFileNameWithoutExtension(originalFileName)
    Dim baseFolder As String = System.IO.Path.GetDirectoryName(originalFileName)
    Dim header As System.Collections.Generic.List(Of String) = Nothing

    For Each table As System.Data.DataTable In ds.Tables
        Dim outFilePath As String = System.IO.Path.Combine(outputFolder, String.Format("{0}.csv", baseFileName, table.TableName))
        Dim e As System.Text.Encoding = System.Text.Encoding.[Default]

        If table.ExtendedProperties.ContainsKey("Unicode") AndAlso CBool(table.ExtendedProperties("Unicode")) Then
            e = System.Text.Encoding.Unicode
        End If

        Using file As New System.IO.StreamWriter(System.IO.File.Open(outFilePath, System.IO.FileMode.Create), e)
            table.ExtendedProperties.Add("Path", outFilePath)

            ' add header row
            header = New System.Collections.Generic.List(Of String)(table.Columns.Count)
            For Each item As System.Data.DataColumn In table.Columns
                header.Add(item.ColumnName)
            Next

            file.WriteLine(String.Join(delimiter, header))

            For Each row As System.Data.DataRow In table.Rows
                ' TODO: For string based fields, capture the max length
                Dim fields As IEnumerable(Of String) = (row.ItemArray).[Select](Function(field) field.ToString())

                file.WriteLine(String.Join(delimiter, fields))
            Next
        End Using
    Next
End Sub

这必须是vb吗？换一种方式问，你是在2005年还是2008年+？不必是VB；我只是更熟悉VB而不是C#，所以这是我的默认设置。我正在使用Visual Studio 2013的SQL Server数据工具。最后一个问题，您要传递的记录集，它只是从一个执行SQL任务中填充的，该任务将完整的结果集传递给Object类型的SSIS变量，是吗？是的，没错！我有一个错误的印象，一旦我把数据输入到完整的resultset中，只需将其转储到文件中就很简单了。但是我发现唯一的方法就是这个循环过程，它对于这个过程要处理的数据量来说太慢了。这必须是vb吗？换一种方式问，你是在2005年还是2008年+？不必是VB；我只是更熟悉VB而不是C#，所以这是我的默认设置。我正在使用Visual Studio 2013的SQL Server数据工具。最后一个问题，您要传递的记录集，它只是从一个执行SQL任务中填充的，该任务将完整的结果集传递给Object类型的SSIS变量，是吗？是的，没错！我错误地认为，一旦我将数据放入完整的结果集中，就可以简单地将其转储到一个文件中。但是我发现唯一的方法就是这个循环过程，它对于这个过程要处理的数据量来说太慢了。你太棒了，非常感谢你的帮助。我快到了。除了以下行之外，一切似乎都正常：
IEnumerable fields=（row.ItemArray）.Select（field=>field.ToString（））它似乎不喜欢“”部分；我得到一个错误，上面写着“System.Collections.IEnumerable没有类型参数，因此不能有类型参数。”我正试图弄清楚这一点，但我对IEnumerable并不十分熟悉，所以我欢迎任何指导。再次感谢你的帮助@凯尔更新-确保你有使用声明中添加你是一个绝对的救命恩人，非常感谢你！此解决方案在几秒钟内将140k+记录集写入CSV。我将在单独的评论中发布VB.NET版本的代码，以防将来对任何人都有帮助。你是最棒的！！你真棒，非常感谢你的帮助。我快到了。除了以下行之外，一切似乎都正常：
IEnumerable fields=（row.ItemArray）.Select（field=>field.ToString（））；