MySQL查询在插入文件时速度会减慢_Mysql_Sql_Performance_Optimization_Query Optimization

MySQL查询在插入文件时速度会减慢

mysql sql performance optimization

MySQL查询在插入文件时速度会减慢,mysql,sql,performance,optimization,query-optimization,Mysql,Sql,Performance,Optimization,Query Optimization,这是我的第一个问题，因为我遇到的每一个问题在这里都有答案。请原谅格式错误查询本身在1ms内运行，这很好。它从大约300万个条目中产生大约60万个结果，而数据库每秒插入大约10个条目。我知道这对于数据库来说不是很重要，所以我假设负载不是问题。我还有其他大的查询，可以很好地插入到文件中。具体来说，当添加“SELECT*INTO OUTFILE”时，此操作将在大约11小时内运行。这对于运行查询来说太长了，我不知道为什么表：容器表 -主键：containerID（bigint）、mapID（int）

这是我的第一个问题，因为我遇到的每一个问题在这里都有答案。请原谅格式错误

查询本身在1ms内运行，这很好。它从大约300万个条目中产生大约60万个结果，而数据库每秒插入大约10个条目。我知道这对于数据库来说不是很重要，所以我假设负载不是问题。我还有其他大的查询，可以很好地插入到文件中。具体来说，当添加“SELECT*INTO OUTFILE”时，此操作将在大约11小时内运行。这对于运行查询来说太长了，我不知道为什么

表：容器表

主键：containerID（bigint）、mapID（int）、cavityID（int）

索引：时间戳（datetime）

表：续表

主键：containerID（bigint）、box（int）、probe（int）、inspectionID（int）、measurementID（int）

表：空洞图

主键：mapID（int）、gob（char）、section（int）、cave（int）

查询：

(SELECT  'containerID','timestamp','mapID','lineID','fp','fpSequence','pocket','cavityID', 'location','inspResult',
     'otgMinThickMeasValuePrb2_1','otgMaxThickMeasValuePrb2_1','RatioPrb2_1','otgOORMeasValuePrb2_1',
     'otgMinThickMeasValuePrb2_2','otgMaxThickMeasValuePrb2_2','RatioPrb2_2','otgOORMeasValuePrb2_2',
     'otgMinThickMeasValuePrb2_3','otgMaxThickMeasValuePrb2_3','RatioPrb2_3')
UNION
(SELECT * INTO OUTFILE 'testcsv.csv'
   FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
   LINES TERMINATED BY '\n'
 FROM
(SELECT          containerID, timestamp, groupmeas.mapID, lineID, fp, fpSequence, pocket,     cavityID, CONCAT(MIN(section), MIN(gob)) AS location,
             inspResult, otgMinThickMeasValuePrb2_1, otgMaxThickMeasValuePrb2_1, 
             (COALESCE(otgMaxThickMeasValuePrb2_1/NULLIF(CAST(otgMinThickMeasValuePrb2_1 AS DECIMAL(10,5)), 0), 0)) AS RatioPrb2_1,
             otgOORMeasValuePrb2_1, otgMinThickMeasValuePrb2_2, otgMaxThickMeasValuePrb2_2,
             (COALESCE(otgMaxThickMeasValuePrb2_2/NULLIF(CAST(otgMinThickMeasValuePrb2_2 AS DECIMAL(10,5)), 0), 0)) AS RatioPrb2_2,
             otgOORMeasValuePrb2_2, otgMinThickMeasValuePrb2_3, otgMaxThickMeasValuePrb2_3,
             (COALESCE(otgMaxThickMeasValuePrb2_3/NULLIF(CAST(otgMinThickMeasValuePrb2_3 AS DECIMAL(10,5)), 0), 0)) AS RatioPrb2_3
FROM 
(SELECT   dbad.container_table.containerID, dbad.container_table.timestamp, dbad.container_table.mapID, dbad.container_table.lineID, dbad.container_table.fp, 
      dbad.container_table.fpSequence, dbad.container_table.pocket, dbad.container_table.cavityID, dbad.container_table.inspResult, 
      CASE WHEN aggMeas.otgMinThickMeasValuePrb2_1 IS NULL
         THEN - 1 ELSE aggMeas.otgMinThickMeasValuePrb2_1 END AS otgMinThickMeasValuePrb2_1, 
      CASE WHEN aggMeas.otgMaxThickMeasValuePrb2_1 IS NULL 
         THEN - 1 ELSE aggMeas.otgMaxThickMeasValuePrb2_1 END AS otgMaxThickMeasValuePrb2_1, 
      CASE WHEN aggMeas.otgOORMeasValuePrb2_1 IS NULL 
         THEN - 1 ELSE aggMeas.otgOORMeasValuePrb2_1 END AS otgOORMeasValuePrb2_1, 
      CASE WHEN aggMeas.otgMinThickMeasValuePrb2_2 IS NULL 
         THEN - 1 ELSE aggMeas.otgMinThickMeasValuePrb2_2 END AS otgMinThickMeasValuePrb2_2, 
      CASE WHEN aggMeas.otgMaxThickMeasValuePrb2_2 IS NULL 
         THEN - 1 ELSE aggMeas.otgMaxThickMeasValuePrb2_2 END AS otgMaxThickMeasValuePrb2_2, 
      CASE WHEN aggMeas.otgOORMeasValuePrb2_2 IS NULL 
         THEN - 1 ELSE aggMeas.otgOORMeasValuePrb2_2 END AS otgOORMeasValuePrb2_2, 
      CASE WHEN aggMeas.otgMinThickMeasValuePrb2_3 IS NULL 
         THEN - 1 ELSE aggMeas.otgMinThickMeasValuePrb2_3 END AS otgMinThickMeasValuePrb2_3, 
      CASE WHEN aggMeas.otgMaxThickMeasValuePrb2_3 IS NULL 
         THEN - 1 ELSE aggMeas.otgMaxThickMeasValuePrb2_3 END AS otgMaxThickMeasValuePrb2_3, 
      CASE WHEN aggMeas.otgOORMeasValuePrb2_3 IS NULL 
         THEN - 1 ELSE aggMeas.otgOORMeasValuePrb2_3 END AS otgOORMeasValuePrb2_3
 FROM   dbad.container_table 
      LEFT OUTER JOIN
      (SELECT     containerID, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 0) AND (meas.probe = 0) THEN meas.value END), - 1) AS otgMinThickMeasValuePrb2_1, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 1) AND (meas.probe = 0) THEN meas.value END), - 1) AS otgMaxThickMeasValuePrb2_1, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 2) AND (meas.probe = 0) THEN meas.value END), - 1) AS otgOORMeasValuePrb2_1, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 0) AND (meas.probe = 1) THEN meas.value END), - 1) AS otgMinThickMeasValuePrb2_2, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 1) AND (meas.probe = 1) THEN meas.value END), - 1) AS otgMaxThickMeasValuePrb2_2, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 2) AND (meas.probe = 1) THEN meas.value END), - 1) AS otgOORMeasValuePrb2_2, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 0) AND (meas.probe = 2) THEN meas.value END), - 1) AS otgMinThickMeasValuePrb2_3, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 1) AND (meas.probe = 2) THEN meas.value END), - 1) AS otgMaxThickMeasValuePrb2_3, 
       COALESCE(MIN(CASE WHEN (meas.inspectionID = 1) AND (meas.measurementID = 2) AND (meas.probe = 2) THEN meas.value END), - 1) AS otgOORMeasValuePrb2_3
       FROM  (SELECT  containerID, inspectionID, measurementID, probe, value, threshold, calibration FROM  dbad.cont_meas_table AS a) AS meas
       GROUP BY containerID) AS aggMeas 
    ON dbad.container_table.containerID = aggMeas.containerID) AS groupmeas
INNER JOIN
dbad.cavity_map
  ON groupmeas.mapID=dbad.cavity_map.mapID  AND
  groupmeas.cavityID=dbad.cavity_map.cavity
  WHERE timestamp LIKE '2014-08-29%'
    AND otgMinThickMeasValuePrb2_1 BETWEEN 1 AND 499
    AND otgMinThickMeasValuePrb2_2 BETWEEN 1 AND 499
    AND otgMinThickMeasValuePrb2_3 BETWEEN 1 AND 499
    AND otgMaxThickMeasValuePrb2_1 BETWEEN 1 AND 499
    AND otgMaxThickMeasValuePrb2_2 BETWEEN 1 AND 499
    AND otgMaxThickMeasValuePrb2_3 BETWEEN 1 AND 499
GROUP BY containerID) AS outside)

我已经去掉了任何

COUNT（）

或

DISTINCT

，并删除了我的

中时间戳前面的“%”，如“2014-08-29%”

，以便可以使用时间戳的索引。不幸的是，这没有帮助

编辑：添加后

其中时间戳>='2014-08-29'
和时间戳<'2014-08-29'+间隔1天

查询实际上需要更长的时间。我知道情况不应该是这样，所以我在这个查询中肯定犯了严重错误。

为了确保您的数据库正确配置为处理这种工作负载，请运行开源工具mysqltuner并查看建议

您的问题描述听起来像是您可能希望在my.cnf中使用不同的tmp_table_大小和max_heap_table_大小

您可以在此处找到该工具：

这里有一样东西跳起来打我的脸：

WHERE timestamp LIKE '2014-08-29%'  /* slow! */

这会阻止在

timestamp

列上使用索引，因为它隐式地将

timestamp

强制转换为字符串

请尝试使用此选项：

WHERE timestamp >= '2014-08-29'
  AND timestamp <  '2014-08-29' + INTERVAL 1 DAY

其中时间戳>='2014-08-29'
时间戳<'2014-08-29'+间隔1天

这将允许查询在

时间戳上使用索引范围扫描，这可能会有很大帮助。它之所以有效，是因为它将常量日期强制转换为与timestamp
相同的数据类型，而不是相反的方式
索引的目的是避免所谓的完全表扫描，在这种扫描中，MySQL服务器必须快速浏览表中的每一行以查找匹配的数据。省略WHERE
子句也会使服务器查看表中的每一行
 您需要并可以优化查询：替换
LIKE '% 29-08-2014'

为了
=“08-29-2014”并且您正在从4个级别的派生表/子查询中进行选择，当然性能会很差。天哪，这是一个复杂的查询。要对其进行性能故障排除，您可能需要将其分解为多个部分。我假设要写入的文件与数据库引擎位于同一驱动器/位置？（它与托管数据库的物理计算机没有什么不同，对吗？）不要忘记快乐的小树。通过首先自行解决查询性能来丢弃文件i/o问题。@sebas他不是说查询本身在1毫秒内运行吗？我想我假设600000是在1毫秒内生成的……我听说过这个，但公司政策阻止我使用它。“公司政策阻止我使用它。”——很好。调整脚本是个糟糕的主意。只需专注于这个问题。在一些评论中提供了很好的建议。由于查询缓存，进入输出文件
无法访问，因此，如果查询没有进入输出文件
，可能会比您想象的慢<代码>选择SQL\u NO\u缓存…

（不带输出文件）将确认这一点<代码>解释选择…将向您显示查询计划，这对于性能故障排除非常有价值。但当数据库是全新的时，我投了更高的票(

>= "08-29-2014" and <'2014-08-30'