改进python csv处理循环

我有一个csv文件，需要从中导出多个列的数据。第4列包含行名。每当此列中的值发生更改时，我导出的数据都需要写入一个新的单独文件。下面的代码可以工作，但速度相当慢。关于如何改进这一点有什么建议吗

补充：数据样本：

Altitude,Date,Db,Depth,Linename,Qmag,TF,TF_HP,X,X_ob,X_org,Y,Y_ob,Y_org
10.87,10/2/2015,148,21.8342,10,1316,48831.936,0.060026123,506479.5515,506479.46,506479.46,5726744.3,5726743.73,5726743.73
10.84,10/2/2015,148,21.8342,11,1316,48831.969,0.092713686,506479.7927,506479.77,506479.77,5726744.443,5726744.2,5726744.2
10.85,10/2/2015,148,21.8669,11,1313,48832.014,0.137400275,506479.9672,506479.77,506479.77,5726744.741,5726744.2,5726744.2
10.82,10/2/2015,148,21.8342,12,1311,48831.969,0.092093953,506480.1677,506479.92,506479.92,5726744.945,5726744.44,5726744.44
10.83,10/2/2015,148,21.8669,12,1309,48831.969,0.091807708,506480.326,506480.08,506480.08,5726745.195,5726744.68,5726744.68

Python代码：

import glob,csv,os, itertools

list_of_files = glob.glob('C:/test/*.csv')

directory = 'C:/test/conv/'
if not os.path.exists(directory):
     os.makedirs(directory)

for filename in list_of_files:

    with open(filename,"r") as source:
        header_line = next(source)
        rdr= csv.reader( source, delimiter = ',',lineterminator='\n' )

        x=0

        for row in itertools.islice(rdr,0,None):
            itemRow4 = row[4]
            outfileName=directory+itemRow4+'.csv'

            with open(outfileName,"a") as result:

                wtr=csv.writer(result, lineterminator='\n')

                if x == 0:
                    previousitemRow4 = row[4]
                    x = x+1                 

                if previousitemRow4 == itemRow4: 
                    wtr.writerow((row[8],row[11],row[6],row[0]))
                    previousitemRow4 = itemRow4
                if previousitemRow4 != itemRow4:
                    wtr.writerow((row[8],row[11],row[6],row[0]))
                    print 'next line'
                    previousitemRow4 = itemRow4

---

使用标准Unix shell工具、和：

$cut-d，-f5out-${lineno}.csv
完成
$ls out-*.csv
out-10.csv out-11.csv out-12.csv out-Linename.csv

---

也许

grep

表达式不够复杂，因为行号可能出现在第5列以外的其他列中。在这种情况下，可以使用一个简单的正则表达式使

grep

仅在第5列中查找lineno。

根据Eumiro的建议，我提出了这个解决方案。我试着使用一个列表的目录，但无法使它工作。下面的解决方案有效且速度非常快。谢谢大家的帮助

import glob,csv,os, itertools

list_of_files = glob.glob('C:/test/*.csv')
print 'By the Power of the Python, Split these here files!'  

directory = 'C:/test/conv/'
if not os.path.exists(directory):
    os.makedirs(directory)

for filename in list_of_files:
    storage = []
    specialStorage = []
    with open(filename,"r") as source:
        header_line = next(source)
        rdr= csv.reader( source, delimiter = ',',lineterminator='\n' )
        x=0
        resetValue = 0

        for row in itertools.islice(rdr,0,None):
            itemRow4 = row[4]

            if x == 0:
                previousitemRow4 = row[4]
                x = x+1

            outfileName=directory+previousitemRow4+'.csv'        

            if previousitemRow4 == itemRow4:                 

                storage.append((row[8],row[11],row[6],row[0]))
                previousitemRow4 = itemRow4

            if previousitemRow4 != itemRow4:

                with open(outfileName,"a") as result:

                    wtr=csv.writer(result, lineterminator='\n')
                    previousitemRow4 = itemRow4

                    if len(specialStorage) !=0:
                        wtr.writerow(specialStorage)
                    wtr.writerows(storage)
                    storage = []

                specialStorage=(row[8],row[11],row[6],row[0])

        else:
            with open(outfileName,"a") as result:

                    wtr=csv.writer(result, lineterminator='\n')
                    previousitemRow4 = itemRow4
                    wtr.writerow(specialStorage)
                    wtr.writerows(storage)
                    storage = []

            print 'end of file reached'

---

您经常打开

outfileName

。将输出行附加到列表中，并在末尾写出所有内容。这不是对您的问题的直接解决方案/答案，而是更一般的建议：如果您想查看代码中（每个方法）花费了多少处理时间，请使用探查器。然后，尝试优化大部分处理时间的使用位置：提高调用的执行时间，或者将调用数量限制在较慢的部分。请参阅Python中的本文档页。@eumiro，显然@Michiel希望每行有一个单独的

outfileName

。因此，他不会多次打开同一个文件（除非

itemrrow4

包含相同的值——他没有包含一个包含数据的示例CSV文件）。@Michiel，你能在问题中包含一个（简短但有代表性的）CSV文件示例吗？@jschulenklopper：我已经包含了该文件的一个小示例。在实际文件中，可能有10000行或更多行属于同一数据行（即第4行中的值相同），谢谢。不过，我使用的是windows机器，只有python可供我使用。

import glob,csv,os, itertools

list_of_files = glob.glob('C:/test/*.csv')
print 'By the Power of the Python, Split these here files!'  

directory = 'C:/test/conv/'
if not os.path.exists(directory):
    os.makedirs(directory)

for filename in list_of_files:
    storage = []
    specialStorage = []
    with open(filename,"r") as source:
        header_line = next(source)
        rdr= csv.reader( source, delimiter = ',',lineterminator='\n' )
        x=0
        resetValue = 0

        for row in itertools.islice(rdr,0,None):
            itemRow4 = row[4]

            if x == 0:
                previousitemRow4 = row[4]
                x = x+1

            outfileName=directory+previousitemRow4+'.csv'        

            if previousitemRow4 == itemRow4:                 

                storage.append((row[8],row[11],row[6],row[0]))
                previousitemRow4 = itemRow4

            if previousitemRow4 != itemRow4:

                with open(outfileName,"a") as result:

                    wtr=csv.writer(result, lineterminator='\n')
                    previousitemRow4 = itemRow4

                    if len(specialStorage) !=0:
                        wtr.writerow(specialStorage)
                    wtr.writerows(storage)
                    storage = []

                specialStorage=(row[8],row[11],row[6],row[0])

        else:
            with open(outfileName,"a") as result:

                    wtr=csv.writer(result, lineterminator='\n')
                    previousitemRow4 = itemRow4
                    wtr.writerow(specialStorage)
                    wtr.writerows(storage)
                    storage = []

            print 'end of file reached'