如何让PHP运行得更快?
我目前正试图将大约2000万行数据从一个文本文件导入到表中。它的速度太慢了(加上2000万不是全部) 有什么方法可以让这个过程更快吗?顺便说一句,我从phpmyadmin运行它如何让PHP运行得更快?,php,mysql,performance,import,ram,Php,Mysql,Performance,Import,Ram,我目前正试图将大约2000万行数据从一个文本文件导入到表中。它的速度太慢了(加上2000万不是全部) 有什么方法可以让这个过程更快吗?顺便说一句,我从phpmyadmin运行它 <?php //connection to the database $dbhandle = mysql_connect($hostname, $username, $password) or die("Unable to connect to MySQL"); echo "Connected to My
<?php
//connection to the database
$dbhandle = mysql_connect($hostname, $username, $password)
or die("Unable to connect to MySQL");
echo "Connected to MySQL<br>";
$selected = mysql_select_db("data",$dbhandle)
or die("Could not select data");
$handle = fopen ('text.txt', 'rt');
while (!feof ($handle))
{
ini_set('max_execution_time',10800);
$buffer = fgets($handle, 4096);
list($a,$b,$c)=explode(" ",$buffer);
$lol = explode(".",$c);
$rest = substr($c,-5,3);
$date = date('Y-m-d H:i:s');
if($rest == 'COM'){
echo $a." | ".$b." | ".$c."<br>";
$sqlquery = "INSERT INTO zonenet (date, domainname, dnstype, nameserver) VALUES('".$date."','".$a."','".$b."','".$c."')";
mysql_query($sqlquery,$dbhandle) or die(mysql_error());
}
else {
//$dnstype = array("a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", ".",
// "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z");
$str = str_replace($lol[1],".NET",$lol[1]);
//$str = str_replace(end($lol),".NET",end($lol));
echo $a." | ".$b." | ".$lol[0]."".$str."<br>";
$sqlquery = "INSERT INTO zonenet (date, domainname, dnstype, nameserver) VALUES('".$date."','".$a."','".$b."','".$lol[0]."".$str."')";
mysql_query($sqlquery,$dbhandle) or die(mysql_error());
}
//dnstype reference @http://en.wikipeda.org/wiki/List_of_DNS_record_types
$sqlquery1 = "DELETE FROM zonenet WHERE dnstype NOT IN ('A', 'AAAA', 'AFSDB', 'APL', 'CAA', 'CERT', 'CNAME', 'DHCID'`enter code here`, 'DLV', 'DNAME', 'DNSKEY', 'DS', 'HIP', 'IPSECKEY', 'KEY', 'KX', 'LOC', 'MX', 'NAPTR', 'NS', 'NSEC', 'NSEC3', 'NSEC3PARAM', 'PTR', 'RRSIG', 'RP', 'SIG', 'SOA', 'SPF', 'SRV', 'SSHFP', 'TA', 'TKEY', 'TLSA', 'TSIG', 'TXT')";
mysql_query($sqlquery1,$dbhandle) or die(mysql_error());
}
fclose ($handle);
//close the connection
mysql_close($dbhandle);
//$net=explode(".",$c);
//echo $a."-".$b."-".$c."<br>";
//
//
//mysql_query($sqlquery,$dbhandle) or die(mysql_error());
?>
2000万行需要时间,因为磁盘限制了在mysql中实际可以插入的数量
我的建议是使用事先准备好的语句。根据您的mysql驱动程序,它可能类似于:
$driver->prepareStatement("insert into table (`field`) values (?)");
foreach ($rows as $row) {
$driver->bindParam('s', $row->value);
$driver->execute();
}
取决于服务器上的内存,上述操作可能会崩溃,因为您的内存中同时有所有行。这可以通过逐行流式传输文件来解决,但使用类似于20M记录这样的函数应该可以很快地解决,因此这应该很容易优化:
在循环中不需要ini_集合,一次就足够了
使用批插入。现在,您要为每一条记录执行一次插入
在插入错误记录之前进行检查,而不是“删除”错误记录
代码更改:(未测试)
您当时实施了什么?你尝试了什么?我刚刚编辑了我的文章,是的,当导入大量数据时它发生了很多崩溃。不要使用这种方法。让数据库做它设计的最好的事情。使用MySQL命令加载数据填充
,速度会快得多。我同意这一点,这取决于您试图实现的目标。如果您只是想直接导入大量数据,那么Load data方法或mysqlimport是更好的解决方案。然而,这个问题被标记为“PHP”,基于此,我假设在导入过程中可能最终会发生一些PHP处理,但仍然会更快直接从PHP级别对文件进行处理,然后直接将其导入MySQL。
<?php
//connection to the database
$dbhandle = mysql_connect($hostname, $username, $password) or die("Unable to connect to MySQL");
echo "Connected to MySQL<br>";
$selected = mysql_select_db("data",$dbhandle) or die("Could not select data");
ini_set('max_execution_time',10800);
$allowedDnsType = array('A', 'AAAA', 'AFSDB', 'APL', 'CAA', 'CERT', 'CNAME', 'DHCID', 'DLV', 'DNAME', 'DNSKEY', 'DS', 'HIP', 'IPSECKEY', 'KEY', 'KX', 'LOC', 'MX', 'NAPTR', 'NS', 'NSEC', 'NSEC3', 'NSEC3PARAM', 'PTR', 'RRSIG', 'RP', 'SIG', 'SOA', 'SPF', 'SRV', 'SSHFP', 'TA', 'TKEY', 'TLSA', 'TSIG', 'TXT');
$lookup = array_flip($allowedDnsType);
$values = array();
$handle = fopen ('text.txt', 'rt');
while (!feof ($handle)) {
$buffer = fgets($handle, 4096);
list($domain,$dnstype,$nameserver)=explode(" ",$buffer);
$lol = explode(".",$nameserver);
$rest = substr($nameserver,-5,3);
$date = date('Y-m-d H:i:s');
if (isset($lookup[$dnstype])) {
if($rest == 'COM'){
echo $domain." | ".$dnstype." | ".$c."<br>";
$values[] = "'".$date."','".$domain."','".$dnstype."','".$nameserver."'";
}
else {
$str = str_replace($lol[1],".NET",$lol[1]);
//$str = str_replace(end($lol),".NET",end($lol));
echo $domain." | ".$dnstype." | ".$lol[0]."".$str."<br>";
$values[] = "'".$date."','".$domain."','".$dnstype."','".$lol[0].$str."'";
}
}
// insert per 200
if (count($values) > 200) {
$sqlquery = "INSERT INTO zonenet (date, domainname, dnstype, nameserver) VALUES(".implode('),(', $values).")";
mysql_query($sqlquery,$dbhandle) or die(mysql_error());
$values = array();
}
//dnstype reference @http://en.wikipeda.org/wiki/List_of_DNS_record_types
}
fclose ($handle);
if (count($values) > 0) {
$sqlquery = "INSERT INTO zonenet (date, domainname, dnstype, nameserver) VALUES(".implode('),(', $values).")";
mysql_query($sqlquery,$dbhandle) or die(mysql_error());
}
//close the connection
mysql_close($dbhandle);