Php 将巨大的CSV文件上载到MySQL数据库脚本

我从一家公司收到大型CSV文件。我想设计一个PHP脚本，将这些文件上传到MySQL数据库，这样我就不必在MySQL b/c中手动执行这些文件每天都会更新

是否有人知道或有我如何创建脚本的例子，该脚本将删除数据库中的旧文件，然后使用CSV文件中的新文件更新数据库。CSV文件包含超过200000行的信息。我需要的东西，将继续和网页不会超时

3/19新增数据

---

我为我的客户提供了相同的解决方案。第一个用户上载一个文件，然后我

截断表并使用
加载数据本地填充。它非常快。我用它来记录10万张唱片。我在测量插入时间，对于60466条记录，通常的
insert
为65071s，而
LOAD
0510s。所以你可以看到差别很大。这是它的参考资料

例如，我有这样一个例子：

$insert='LOAD DATA LOCAL INFILE "'.$csv_file.'" 
INTO TABLE im_serialnumbers CHARACTER SET latin1
FIELDS TERMINATED BY ";"
IGNORE 1 LINES
(sn,l,gdatum1,artik,cust,lag,nr,freetext,
chnr,snr,idatum2,odatum2,vdatum)';

并使用.csv文件的绝对路径。
我使用了以下Javascript和php代码，而不是使用上述信息。基本上这是一个表单，它调用下面的javascript函数，然后调用processupload.php。php然后将文件上载到服务器

Index.php

echo '<div id="upload-wrapper">'; 
echo '<div align="center">'; 
echo '<h3>File Uploader</h3>'; 
echo '<form action="processupload.php" method="post" enctype="multipart/form-data" id="MyUploadForm">'; 
echo '<select name="CSVFileData" id="CSVFileData"><option value="default">Select CSV Upload</option><option value="NovationRoster">Novation Roster CSV File</option><option value="MembershipRoster">Membership Roster CSV File</option><option value="GLNExport">GLN Text File</option></select><br/>'; 
echo '<input name="FileInput" id="FileInput" type="file" />'; 
echo '<input type="submit"  id="submit-btn" value="Upload" />'; 
//echo '<img src="images/loading.gif" width="30px" id="loading-img" style="display:none;" alt="Please Wait"/>'; 
echo '</form>'; 
echo '<div id="progressbox" ><div id="statustxt">0%</div></div>'; 
echo '<div id="output"></div>'; 
echo '<div><img src="images/loading.gif" width="50px" id="loading-img" style="display:none;" alt="Please Wait"/></div>'; 
echo '</div>'; 
echo '</div>'; 

?> 
<script type="text/javascript" src="js/jquery-1.10.2.min.js"></script> 
<script type="text/javascript" src="js/jquery.form.min.js"></script> 
<script type="text/javascript"> 
$(document).ready(function() {  
    var options = {  
        target:   '#output',   // target element(s) to be updated with server response  
        beforeSubmit:  beforeSubmit,  // pre-submit callback  
        success:       afterSuccess,  // post-submit callback  
        uploadProgress: OnProgress, //upload progress callback  
        resetForm: true        // reset the form after successful submit  
    };  

 $('#MyUploadForm').submit(function() {  
        $(this).ajaxSubmit(options);             
        // always return false to prevent standard browser submit and page navigation  
        return false;  
    });  


//function after successful file upload (when server response) 
function afterSuccess() 
{ 
    $('#submit-btn').show();  
    $('#loading-img').hide(); 
    $('#progressbox').delay( 1000 ).fadeOut(); //hide progress bar 
    $('#output').html("Data added to database"); 
} 

//function to check file size before uploading. 
function beforeSubmit(){ 
    //check whether browser fully supports all File API 
   if (window.File && window.FileReader && window.FileList && window.Blob) 
    { 

        if( !$('#FileInput').val()) //check empty input filed 
        { 
            $("#output").html("No file, please select file!"); 
            return false 

        if ($('#CSVFileData').val() == 'default') 
        { 
            $("#output").html("Select Value in Dropdown"); 
            return false 
        } 

        var fsize = $('#FileInput')[0].files[0].size; //get file size 
        var ftype = $('#FileInput')[0].files[0].type; // get file type 


        //allow file types  
        switch(ftype) 
        { 
            //case 'image/png':  
            //case 'image/gif':  
            //case 'image/jpeg':  
            //case 'image/pjpeg': 
            case 'text/plain': 
            //case 'text/html': 
            //case 'application/x-zip-compressed': 
            //case 'application/pdf': 
            //case 'application/msword': 
            case 'application/vnd.ms-excel': 
            //case 'video/mp4': 
                break; 
            default: 
                $("#output").html("<b>"+ftype+"</b> Unsupported file type, please enter csv file type."); 
            return false 
        } 

        //Allowed file size is less than 5 MB (1048576) 
        //if(fsize>30000000)  
        if(fsize > 65000000) 
        { 
            $("#output").html("<b>"+bytesToSize(fsize) +"</b> Too big file! <br />File is too big, it should be less than 5 MB."); 
            return false 
        } 

        $('#submit-btn').hide();  
        $('#loading-img').show();  
        $("#output").html("");   
    } 
    else
    { 
        //Output error to older unsupported browsers that doesn't support HTML5 File API 
        $("#output").html("Please upgrade your browser, because your current browser lacks some new features we need!"); 
    return false; 
    } 
} 

//progress bar function 
function OnProgress(event, position, total, percentComplete) 
{ 
/    /Progress bar 
$('#progressbox').show(); 
//$('#progressbar').width(percentComplete + '%') //update progressbar percent complete 
$('#statustxt').html(percentComplete + '%'); //update status text 
if(percentComplete>50) 
{ 
    $('#statustxt').css('color','#000'); //change status text to white after 50% 
} 
if (percentComplete == 100) 
{ 
    $('#output').html("File upload successful, adding data to database."); 
    $('#progressbox').delay( 1000 ).fadeOut(); 
} 
} 

//function to format bites bit.ly/19yoIPO 
function bytesToSize(bytes) { 
   var sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB']; 
   if (bytes == 0) return '0 Bytes'; 
   var i = parseInt(Math.floor(Math.log(bytes) / Math.log(1024))); 
   return Math.round(bytes / Math.pow(1024, i), 2) + ' ' + sizes[i]; 
} 

});  
</script>

你不需要一个复杂的临时脚本。您只需将该文件发送到运行MySQL的服务器，然后运行以下命令：我添加的代码仍然不起作用。该文件是否位于运行MySQL的同一服务器上？你犯了什么样的错误？尝试手动执行该查询，查看生成了什么错误消息。是否还有其他需要添加的内容？我的文件保存在服务器上php执行的同一位置。但是，不要加载到数据库中。显示完整的代码，如果您使用了此答案，请接受它
echo '<div id="upload-wrapper">'; 
echo '<div align="center">'; 
echo '<h3>File Uploader</h3>'; 
echo '<form action="processupload.php" method="post" enctype="multipart/form-data" id="MyUploadForm">'; 
echo '<select name="CSVFileData" id="CSVFileData"><option value="default">Select CSV Upload</option><option value="NovationRoster">Novation Roster CSV File</option><option value="MembershipRoster">Membership Roster CSV File</option><option value="GLNExport">GLN Text File</option></select><br/>'; 
echo '<input name="FileInput" id="FileInput" type="file" />'; 
echo '<input type="submit"  id="submit-btn" value="Upload" />'; 
//echo '<img src="images/loading.gif" width="30px" id="loading-img" style="display:none;" alt="Please Wait"/>'; 
echo '</form>'; 
echo '<div id="progressbox" ><div id="statustxt">0%</div></div>'; 
echo '<div id="output"></div>'; 
echo '<div><img src="images/loading.gif" width="50px" id="loading-img" style="display:none;" alt="Please Wait"/></div>'; 
echo '</div>'; 
echo '</div>'; 

?> 
<script type="text/javascript" src="js/jquery-1.10.2.min.js"></script> 
<script type="text/javascript" src="js/jquery.form.min.js"></script> 
<script type="text/javascript"> 
$(document).ready(function() {  
    var options = {  
        target:   '#output',   // target element(s) to be updated with server response  
        beforeSubmit:  beforeSubmit,  // pre-submit callback  
        success:       afterSuccess,  // post-submit callback  
        uploadProgress: OnProgress, //upload progress callback  
        resetForm: true        // reset the form after successful submit  
    };  

 $('#MyUploadForm').submit(function() {  
        $(this).ajaxSubmit(options);             
        // always return false to prevent standard browser submit and page navigation  
        return false;  
    });  


//function after successful file upload (when server response) 
function afterSuccess() 
{ 
    $('#submit-btn').show();  
    $('#loading-img').hide(); 
    $('#progressbox').delay( 1000 ).fadeOut(); //hide progress bar 
    $('#output').html("Data added to database"); 
} 

//function to check file size before uploading. 
function beforeSubmit(){ 
    //check whether browser fully supports all File API 
   if (window.File && window.FileReader && window.FileList && window.Blob) 
    { 

        if( !$('#FileInput').val()) //check empty input filed 
        { 
            $("#output").html("No file, please select file!"); 
            return false 

        if ($('#CSVFileData').val() == 'default') 
        { 
            $("#output").html("Select Value in Dropdown"); 
            return false 
        } 

        var fsize = $('#FileInput')[0].files[0].size; //get file size 
        var ftype = $('#FileInput')[0].files[0].type; // get file type 


        //allow file types  
        switch(ftype) 
        { 
            //case 'image/png':  
            //case 'image/gif':  
            //case 'image/jpeg':  
            //case 'image/pjpeg': 
            case 'text/plain': 
            //case 'text/html': 
            //case 'application/x-zip-compressed': 
            //case 'application/pdf': 
            //case 'application/msword': 
            case 'application/vnd.ms-excel': 
            //case 'video/mp4': 
                break; 
            default: 
                $("#output").html("<b>"+ftype+"</b> Unsupported file type, please enter csv file type."); 
            return false 
        } 

        //Allowed file size is less than 5 MB (1048576) 
        //if(fsize>30000000)  
        if(fsize > 65000000) 
        { 
            $("#output").html("<b>"+bytesToSize(fsize) +"</b> Too big file! <br />File is too big, it should be less than 5 MB."); 
            return false 
        } 

        $('#submit-btn').hide();  
        $('#loading-img').show();  
        $("#output").html("");   
    } 
    else
    { 
        //Output error to older unsupported browsers that doesn't support HTML5 File API 
        $("#output").html("Please upgrade your browser, because your current browser lacks some new features we need!"); 
    return false; 
    } 
} 

//progress bar function 
function OnProgress(event, position, total, percentComplete) 
{ 
/    /Progress bar 
$('#progressbox').show(); 
//$('#progressbar').width(percentComplete + '%') //update progressbar percent complete 
$('#statustxt').html(percentComplete + '%'); //update status text 
if(percentComplete>50) 
{ 
    $('#statustxt').css('color','#000'); //change status text to white after 50% 
} 
if (percentComplete == 100) 
{ 
    $('#output').html("File upload successful, adding data to database."); 
    $('#progressbox').delay( 1000 ).fadeOut(); 
} 
} 

//function to format bites bit.ly/19yoIPO 
function bytesToSize(bytes) { 
   var sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB']; 
   if (bytes == 0) return '0 Bytes'; 
   var i = parseInt(Math.floor(Math.log(bytes) / Math.log(1024))); 
   return Math.round(bytes / Math.pow(1024, i), 2) + ' ' + sizes[i]; 
} 

});  
</script>

set_time_limit(0); 
ini_set('memory_limit','2048M'); 

if(isset($_FILES["FileInput"]) && $_FILES["FileInput"]["error"]== UPLOAD_ERR_OK) 
{ 
    ############ Edit settings ############## 
    $UploadDirectory    = '**************************'; 
    //specify upload directory ends with / (slash) 
    ########################################## 


    //check if this is an ajax request 
    if (!isset($_SERVER['HTTP_X_REQUESTED_WITH'])) 
    { 
         die(); 
    } 


    //Is file size is less than allowed size. 
    //if ($_FILES["FileInput"]["size"] > 30000000)  
    if ($_FILES["FileInput"]["size"] > 65000000)  
    { 
        die("File size is too big!"); 
    } 

    //allowed file type Server side check 
    switch(strtolower($_FILES['FileInput']['type'])) 
    {  
        case 'text/plain': 
        case 'application/vnd.ms-excel': 
        //case 'video/mp4': 
        break; 
        default: 
        { 
            die('Unsupported File!'); //output error 
        } 
    } 

    $File_Name          = strtolower($_FILES['FileInput']['name']); 
    // Get File Extension 
    $File_Ext           = substr($File_Name, strrpos($File_Name, '.')); 
    //$NewFileName      = $_FILES['FileInput']['name']; 

    if ($_POST['CSVFileData'] == 'NovationRoster') 
    { 
        $NewFileName = 'NovationRoster.csv'; 
    } 
    else if ($_POST['CSVFileData'] == 'MembershipRoster') 
    { 
        $NewFileName = 'MembershipRoster.csv'; 
    } 
    else if ($_POST['CSVFileData'] == 'GLNExport') 
    { 
        $NewFileName = 'export.txt'; 
    } 

    if(move_uploaded_file($_FILES['FileInput']['tmp_name'], $UploadDirectory.$NewFileName )) 
   { 
        header ("location: index.php?upload=".urlencode($_FILES['FileInput']['name'])."&d=".$_POST['CSVFileData']); 
    } 
    else
    { 
        die('error uploading File!'); 
    } 

} 
else
{ 
    die('Something wrong with upload! Is "upload_max_filesize" set correctly?'); 
}