Performance 联合需要太长时间_Performance_Postgresql_Query Optimization_Union All

Performance 联合需要太长时间

performance postgresql

Performance 联合需要太长时间,performance,postgresql,query-optimization,union-all,Performance,Postgresql,Query Optimization,Union All,我在多个表上有群集数据，通常它们看起来如下： CREATE TABLE 2012_03_09 ( guid_key integer, property_key integer, instance_id_key integer, time_stamp timestamp without time zone, "value" double precision ) 使用这些索引： CREATE INDEX 2012_03_09_a ON 2012_03_0

我在多个表上有群集数据，通常它们看起来如下：

CREATE TABLE 2012_03_09 (
    guid_key integer,
    property_key integer,
    instance_id_key integer,
    time_stamp timestamp without time zone,
    "value" double precision
)

使用这些索引：

CREATE INDEX 2012_03_09_a
  ON 2012_03_09
  USING btree
  (guid_key, property_key, time_stamp);

CREATE INDEX 2012_03_09_b
  ON 2012_03_09
  USING btree
  (time_stamp, property_key);

当我分析查询时，追加操作的总时间让我感到困扰。你能解释一下为什么查询运行得太长吗？有没有办法优化这样的查询

Sort  (cost=262.50..262.61 rows=47 width=20) (actual time=1918.237..1918.246 rows=100 loops=1)    
  Output: 2012_04_26.time_stamp, 2012_04_26.value, 2012_04_26.instance_id_key    
  Sort Key: 2012_04_26.instance_id_key, 2012_04_26.time_stamp    
  Sort Method:  quicksort  Memory: 32kB    
  ->  Append  (cost=0.00..261.19 rows=47 width=20) (actual time=69.817..1917.848 rows=100 loops=1)    
        ->  Index Scan using 2012_04_26_a on 2012_04_26  (cost=0.00..8.28 rows=1 width=20) (actual time=14.909..14.909 rows=0 loops=1)    
              Output: 2012_04_26.time_stamp, 2012_04_26.value, 2012_04_26.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_04_27_a on 2012_04_27  (cost=0.00..8.28 rows=1 width=20) (actual time=1.535..1.535 rows=0 loops=1)    
              Output: 2012_04_27.time_stamp, 2012_04_27.value, 2012_04_27.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_02_a on 2012_05_02  (cost=0.00..12.50 rows=2 width=20) (actual time=53.370..121.894 rows=6 loops=1)    
              Output: 2012_05_02.time_stamp, 2012_05_02.value, 2012_05_02.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_03_a on 2012_05_03  (cost=0.00..24.74 rows=5 width=20) (actual time=59.136..170.215 rows=11 loops=1)    
              Output: 2012_05_03.time_stamp, 2012_05_03.value, 2012_05_03.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_04_a on 2012_05_04  (cost=0.00..12.47 rows=2 width=20) (actual time=67.458..125.172 rows=5 loops=1)    
              Output: 2012_05_04.time_stamp, 2012_05_04.value, 2012_05_04.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_05_a on 2012_05_05  (cost=0.00..8.28 rows=1 width=20) (actual time=14.112..14.112 rows=0 loops=1)    
              Output: 2012_05_05.time_stamp, 2012_05_05.value, 2012_05_05.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_07_a on 2012_05_07  (cost=0.00..12.46 rows=2 width=20) (actual time=60.549..99.999 rows=4 loops=1)    
              Output: 2012_05_07.time_stamp, 2012_05_07.value, 2012_05_07.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_08_a on 2012_05_08  (cost=0.00..24.71 rows=5 width=20) (actual time=63.367..197.296 rows=12 loops=1)    
              Output: 2012_05_08.time_stamp, 2012_05_08.value, 2012_05_08.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_09_a on 2012_05_09  (cost=0.00..28.87 rows=6 width=20) (actual time=59.596..224.685 rows=15 loops=1)    
              Output: 2012_05_09.time_stamp, 2012_05_09.value, 2012_05_09.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_10_a on 2012_05_10  (cost=0.00..28.85 rows=6 width=20) (actual time=56.995..196.590 rows=13 loops=1)    
              Output: 2012_05_10.time_stamp, 2012_05_10.value, 2012_05_10.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_11_a on 2012_05_11  (cost=0.00..20.59 rows=4 width=20) (actual time=62.761..134.313 rows=8 loops=1)    
              Output: 2012_05_11.time_stamp, 2012_05_11.value, 2012_05_11.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_12_a on 2012_05_12  (cost=0.00..8.28 rows=1 width=20) (actual time=12.018..12.018 rows=0 loops=1)    
              Output: 2012_05_12.time_stamp, 2012_05_12.value, 2012_05_12.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_13_a on 2012_05_13  (cost=0.00..8.28 rows=1 width=20) (actual time=12.286..12.286 rows=0 loops=1)    
              Output: 2012_05_13.time_stamp, 2012_05_13.value, 2012_05_13.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_14_a on 2012_05_14  (cost=0.00..16.58 rows=3 width=20) (actual time=92.161..156.802 rows=6 loops=1)    
              Output: 2012_05_14.time_stamp, 2012_05_14.value, 2012_05_14.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_15_a on 2012_05_15  (cost=0.00..25.03 rows=5 width=20) (actual time=73.636..263.537 rows=14 loops=1)    
              Output: 2012_05_15.time_stamp, 2012_05_15.value, 2012_05_15.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
        ->  Index Scan using 2012_05_16_a on 2012_05_16  (cost=0.00..12.56 rows=2 width=20) (actual time=100.893..172.404 rows=6 loops=1)    
              Output: 2012_05_16.time_stamp, 2012_05_16.value, 2012_05_16.instance_id_key    
              Index Cond: ((guid_key = 2105) AND (property_key = 67) AND (time_stamp >= '2012-04-16 00:00:00'::timestamp without time zone) AND (time_stamp <= '2012-05-16 06:25:50.172'::timestamp without time zone))    
Total runtime: 1918.745 ms

看起来你应该检查一下。您的查询会更简单，而且可能会执行得更好—不是100%确定，但我认为值得一试

看起来您应该检查一下。您的查询将更简单，而且可能会执行得更好（不是100%确定），但我认为值得一试

除了追加之外，所有行似乎都是第一种类型的索引扫描获得的。我想知道这是否是最好的指数。由于您似乎正在选择有效的时间范围，因此唯一的其他选择是guid\u键和property\u键。哪个更有选择性？更具选择性的列应该是第一列，也就是说，如果您不担心排序问题（我认为您不应该为100行排序）。其次，您添加这些索引是为了这个查询还是其他查询？如果它们在其他任何地方都没有用处，那么丢弃它们可能是有意义的。索引实际上会降低性能，尤其是在表记录大部分时间都已在内存中的情况下，因为它们可能需要数据库从内存中卸载记录以加载索引，然后在完成索引扫描后将表记录加载回内存

我能给出的唯一真正的建议就是玩它

编辑：

当然还有另外一个问题，为什么这些记录没有某种主键，并且我忽略了表本身上有/没有集群，但它们也在这里发挥作用。

我能给出的唯一真正的建议就是玩它

编辑：

当然还有其他问题，为什么这些记录没有某种主键，并且表本身有/没有集群，我忽略了这些问题，但它们也在这里发挥作用。

联合不是您的计时问题，它报告的运行时间基本上是每个分区的索引扫描时间之和。您的_a索引看起来对查询谓词具有适当的选择性。我在解释分析中看到的实时罪魁祸首是，在每个分区上进行索引扫描时，仅检索几行需要很长时间。例如：2012年5月4日5行125ms。索引扫描可能会根据缓存状态和表大小调用0-5次搜索，如果数据未聚集，则每个数据行将有一次搜索。速度较慢的单轴磁盘应能在约10毫秒内进行寻道和块提取，因此，使用蹩脚的存储系统进行扫描的最坏情况约为100毫秒，但对于更常见的7200或10K rpm磁盘以及多轴磁盘，假设没有缓存命中，最坏情况应在50毫秒以下。有了良好的缓存保留率，我预计每个分区的索引扫描不会超过几十毫秒

此查询在第一次之后的第二次尝试中是否运行得更快？如果是这样的话，这就意味着存储速度慢，而冷缓存是问题所在。数据库运行在哪种类型的存储上？如果我们谈论的是低速笔记本电脑驱动器，或是高延迟的网络挂载，这就可以解释IO性能差的原因。索引扫描也会受到极端索引膨胀的影响。如果您有几十个或数百个死索引项，这是由于数据的更新/删除操作不当造成的，那么这可能是罪魁祸首。这些表格是否定期进行真空吸尘和分析

正如Adrian Serafin所建议的，您应该研究Pg的表分区特性。

联合不是您的计时问题，它报告的运行时间基本上是每个分区的索引扫描时间的总和。您的_a索引看起来对查询谓词具有适当的选择性。我在解释分析中看到的实时罪魁祸首是，在每个分区上进行索引扫描时，仅检索几行需要很长时间。例如：2012年5月4日5行125ms。索引扫描可能会根据缓存状态和表大小调用0-5次搜索，如果数据未聚集，则会出现每个数据行一次搜索。速度较慢的单轴磁盘应能在约10毫秒内进行寻道和块提取，因此，使用蹩脚的存储系统进行扫描的最坏情况约为100毫秒，但对于更常见的7200或10K rpm磁盘以及多轴磁盘，假设没有缓存命中，最坏情况应在50毫秒以下。有了良好的缓存保留率，我预计每个分区的索引扫描不会超过几十毫秒

正如Adrian Serafin所建议的，您应该研究Pg的表分区功能。

我已经更新了原始帖子，感谢您提到查询返回了多少记录？另外，查询是否真的运行太长，或者只是解释中的高数字让您感到困扰？261的成本真的很低。真的很慢吗？这看起来也像是将所有成本加起来的成本。有没有一个原因可以让您期望append比它下面所有工作的总和都要快？查询返回大约100行，但它被多次调用，这大大降低了业务逻辑的速度。Append中的高数字也让我感到困扰。我已经更新了原始帖子，感谢您提到查询返回了多少记录？另外，查询是否真的运行太长，或者只是解释中的高数字让您感到困扰？261的成本真的很低。真的很慢吗？这看起来也像是将所有成本加起来的成本。有没有一个原因可以让您期望append比它下面所有工作的总和都要快？查询返回大约100行，但它被多次调用，这大大降低了业务逻辑的速度。Append中的高数字也困扰着我。。

select time_stamp, value, instance_id_key as segment from perf_hourly_2012_04_26 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_04_27 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_02 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_03 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_04 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_05 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_07 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_08 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_09 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_10 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_11 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_12 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_13 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_14 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_15 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
UNION ALL
select time_stamp, value, instance_id_key as segment from 2012_05_16 where guid_key = 2105 and property_key=67 and time_stamp between '2012-04-16 00:00:00.0'::timestamp without time zone and '2012-05-16 06:25:50.172'::timestamp without time zone
ORDER BY 3 ASC, 1 ASC