Python 多GPU的TensorFlow

如果有人能帮我了解情况，那就太好了。提前谢谢。 我的设置： 操作系统：Ubuntu 16.04，2个Titan X GPU。TensorFlow（版本0.12.1）安装在conda环境中，在TF文档上使用pip as。Python 3.5

代码： 我运行以下代码来测试我的2GPU设置。一次使用

random\u matrix=tf.zero（…）

和

random\u matrix=tf.random\u uniform（…）

。输出如下所示

问题: 1） 当我使用

tf.zero运行时

。CPU和GPU上的计时是相同的。但是使用

tf.random\u uniform

我发现GPU速度更快（正如我预期的那样）。为什么

tf.zeros

在GPU上速度较慢？我错过了什么？ 2） 我已经修复了全球种子和本地种子。为什么在

tf.random\u uniform

情况下，GPU内的输出不同

非常感谢您提前提供的任何见解

import sys
import numpy as np
import tensorflow as tf
from datetime import datetime

device_names = ["/cpu:0", "/gpu:0", "/gpu:1"]
shapes = [(3000, 3000), (6000, 6000), (9000, 9000), (12000, 12000)]

all_timings = []
tf.set_random_seed(1234)
for device_name in device_names:
    device_timings = []
    for shape in shapes:
        print("device_name:::::::::{}".format(device_name))
        with tf.device(device_name):
            # random_matrix = tf.zeros(shape)
            random_matrix = tf.random_uniform(shape=shape, 
                                              minval=0, 
                                              maxval=1, 
                                              seed=1234)
            result_op = tf.reduce_sum(tf.matmul(random_matrix,tf.transpose(random_matrix)))

        start_time = datetime.now()
        result = -1.0
        with tf.Session(config=tf.ConfigProto(log_device_placement=False)) as session:
            result = session.run(result_op)
            time_diff = datetime.now() - start_time
            device_timings.append((device_name,
                                   shape,
                                   "time_taken (secs): {}".format(time_diff.total_seconds()),
                                   "result: {}".format(result)))
            print("++++++++++++++++++++++++++++++++++++++++++++++++++++++\n\n")
all_timings.append(device_timings)

print("\n\n")
for device_timings in all_timings:
    for t in device_timings:
    print(t)
    print("---------------------------------------------------------\n\n")

使用tf.random_uniform（）的计时：

使用tf.zeros（）的计时：

---

我怀疑这与GPU内核优化有关。如果您通过运行相同的计算形状来“预热”您的GPU，那么下一次执行会快得多。有一个PTX编译，它是GPU上的内核在进程中的第一次使用，但奇怪的是，您的运行时随着矩阵的大小而增加，也许还有一些评测正在进行

请注意，如果没有

tf.OptimizerOptions.L0

，它的速度会变得难以置信，因此也会发生一些缓存

shape = (6000, 6000)
with tf.device("/gpu:0"):
    random_matrix_gpu = tf.zeros(shape)
    result_op_gpu = tf.reduce_sum(tf.matmul(random_matrix_gpu,tf.transpose(random_matrix_gpu)))
with tf.device("/cpu:0"):
    random_matrix_cpu = tf.zeros(shape)
    result_op_cpu = 

tf.reduce_sum(tf.matmul(random_matrix_cpu,tf.transpose(random_matrix_cpu)))
config = tf.ConfigProto(graph_options=tf.GraphOptions(optimizer_options=tf.OptimizerOptions(opt_level=tf.OptimizerOptions.L0)))
sess = tf.Session(config=config)

def profile(op, msg):
    start_time = time.time()
    sess.run(op)
    print(msg, time.time()-start_time)

profile(result_op_cpu, "cpu1")
profile(result_op_cpu, "cpu2")
profile(result_op_gpu, "gpu1")
profile(result_op_gpu, "gpu2")

我看到：

cpu1 1.716048240661621
cpu2 1.509080171585083
gpu1 4.192790746688843
gpu2 0.13361549377441406

---

我怀疑这与GPU内核优化有关。如果您通过运行相同的计算形状来“预热”您的GPU，那么下一次执行会快得多。有一个PTX编译，它是GPU上的内核在进程中的第一次使用，但奇怪的是，您的运行时随着矩阵的大小而增加，也许还有一些评测正在进行

请注意，如果没有

tf.OptimizerOptions.L0

，它的速度会变得难以置信，因此也会发生一些缓存

shape = (6000, 6000)
with tf.device("/gpu:0"):
    random_matrix_gpu = tf.zeros(shape)
    result_op_gpu = tf.reduce_sum(tf.matmul(random_matrix_gpu,tf.transpose(random_matrix_gpu)))
with tf.device("/cpu:0"):
    random_matrix_cpu = tf.zeros(shape)
    result_op_cpu = 

tf.reduce_sum(tf.matmul(random_matrix_cpu,tf.transpose(random_matrix_cpu)))
config = tf.ConfigProto(graph_options=tf.GraphOptions(optimizer_options=tf.OptimizerOptions(opt_level=tf.OptimizerOptions.L0)))
sess = tf.Session(config=config)

def profile(op, msg):
    start_time = time.time()
    sess.run(op)
    print(msg, time.time()-start_time)

profile(result_op_cpu, "cpu1")
profile(result_op_cpu, "cpu2")
profile(result_op_gpu, "gpu1")
profile(result_op_gpu, "gpu2")

我看到：

cpu1 1.716048240661621
cpu2 1.509080171585083
gpu1 4.192790746688843
gpu2 0.13361549377441406

---

谢谢亚罗斯拉夫！我提供代码和运行结果，以防其他人感兴趣。如果您尝试了代码，请耐心等待几分钟

代码：

总结：

matrix=random_uniform+config=simple+warmup=False
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.428429 seconds', 'result = 6754431488.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.806464 seconds', 'result = 54023852032.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.36232 seconds', 'result = 184425938944.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 22.376751 seconds', 'result = 439655661568.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.392957 seconds', 'result = 6754390016.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.082889 seconds', 'result = 54006833152.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.221844 seconds', 'result = 182251814912.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.438476 seconds', 'result = 431995879424.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.483864 seconds', 'result = 6754393088.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.097571 seconds', 'result = 54006833152.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.250176 seconds', 'result = 182252044288.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.473314 seconds', 'result = 431996567552.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=random_uniform+config=simple+warmup=True
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.475717 seconds', 'result = 6754431488.0')
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.379542 seconds', 'result = 6754431488.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.856803 seconds', 'result = 54023852032.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.798967 seconds', 'result = 54023852032.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.447787 seconds', 'result = 184425938944.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.385646 seconds', 'result = 184425938944.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.752967 seconds', 'result = 439655661568.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.832136 seconds', 'result = 439655661568.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.067066 seconds', 'result = 6754394624.0')
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.008072 seconds', 'result = 6754394624.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.123611 seconds', 'result = 54006833152.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.057391 seconds', 'result = 54006833152.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.248432 seconds', 'result = 182251913216.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.18535 seconds', 'result = 182251913216.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.48081 seconds', 'result = 431996043264.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.412447 seconds', 'result = 431996043264.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.105071 seconds', 'result = 6754395648.0')
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.008107 seconds', 'result = 6754395648.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.137264 seconds', 'result = 54006849536.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.064462 seconds', 'result = 54006849536.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.280302 seconds', 'result = 182251831296.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.191399 seconds', 'result = 182251831296.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.509208 seconds', 'result = 431996534784.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.4263 seconds', 'result = 431996534784.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=random_uniform+config=optim+warmup=False
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.552631 seconds', 'result = 6754431488.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.894024 seconds', 'result = 54023852032.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.394226 seconds', 'result = 184425938944.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.870817 seconds', 'result = 439655661568.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.107416 seconds', 'result = 6754392576.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.163633 seconds', 'result = 54006804480.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.304741 seconds', 'result = 182251667456.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.526494 seconds', 'result = 431995944960.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.119625 seconds', 'result = 6754394624.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.203158 seconds', 'result = 54006800384.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.317646 seconds', 'result = 182251978752.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.544184 seconds', 'result = 431996076032.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=zeros+config=simple+warmup=False
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.632157 seconds', 'result = 0.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.901679 seconds', 'result = 0.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.345713 seconds', 'result = 0.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.707619 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.498451 seconds', 'result = 0.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 2.900121 seconds', 'result = 0.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 9.4296 seconds', 'result = 0.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 21.750406 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.523286 seconds', 'result = 0.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 2.887522 seconds', 'result = 0.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 9.377383 seconds', 'result = 0.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 21.639043 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=zeros+config=simple+warmup=True
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.520212 seconds', 'result = 0.0')
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.000172 seconds', 'result = 0.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.914485 seconds', 'result = 0.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 0.000166 seconds', 'result = 0.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.346122 seconds', 'result = 0.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 0.000207 seconds', 'result = 0.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.715376 seconds', 'result = 0.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 0.0002 seconds', 'result = 0.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.556841 seconds', 'result = 0.0')
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.000234 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (6000, 6000)', 'times = 2.936608 seconds', 'result = 0.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.000244 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (9000, 9000)', 'times = 9.34956 seconds', 'result = 0.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.000246 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (12000, 12000)', 'times = 21.634354 seconds', 'result = 0.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.000221 seconds', 'result = 0.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.562244 seconds', 'result = 0.0')
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.000255 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (6000, 6000)', 'times = 2.961658 seconds', 'result = 0.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.000237 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (9000, 9000)', 'times = 9.308582 seconds', 'result = 0.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.000239 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (12000, 12000)', 'times = 21.707127 seconds', 'result = 0.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.000261 seconds', 'result = 0.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=zeros+config=optim+warmup=False
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.560451 seconds', 'result = 0.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.978946 seconds', 'result = 0.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.3279 seconds', 'result = 0.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.694664 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.249778 seconds', 'result = 0.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.365332 seconds', 'result = 0.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.663667 seconds', 'result = 0.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 1.032716 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.299856 seconds', 'result = 0.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.294592 seconds', 'result = 0.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.55067 seconds', 'result = 0.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.806868 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++

---

谢谢亚罗斯拉夫！我提供代码和运行结果，以防其他人感兴趣。如果您尝试了代码，请耐心等待几分钟

代码：

总结：

matrix=random_uniform+config=simple+warmup=False
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.428429 seconds', 'result = 6754431488.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.806464 seconds', 'result = 54023852032.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.36232 seconds', 'result = 184425938944.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 22.376751 seconds', 'result = 439655661568.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.392957 seconds', 'result = 6754390016.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.082889 seconds', 'result = 54006833152.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.221844 seconds', 'result = 182251814912.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.438476 seconds', 'result = 431995879424.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.483864 seconds', 'result = 6754393088.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.097571 seconds', 'result = 54006833152.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.250176 seconds', 'result = 182252044288.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.473314 seconds', 'result = 431996567552.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=random_uniform+config=simple+warmup=True
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.475717 seconds', 'result = 6754431488.0')
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.379542 seconds', 'result = 6754431488.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.856803 seconds', 'result = 54023852032.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.798967 seconds', 'result = 54023852032.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.447787 seconds', 'result = 184425938944.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.385646 seconds', 'result = 184425938944.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.752967 seconds', 'result = 439655661568.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.832136 seconds', 'result = 439655661568.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.067066 seconds', 'result = 6754394624.0')
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.008072 seconds', 'result = 6754394624.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.123611 seconds', 'result = 54006833152.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.057391 seconds', 'result = 54006833152.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.248432 seconds', 'result = 182251913216.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.18535 seconds', 'result = 182251913216.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.48081 seconds', 'result = 431996043264.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.412447 seconds', 'result = 431996043264.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.105071 seconds', 'result = 6754395648.0')
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.008107 seconds', 'result = 6754395648.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.137264 seconds', 'result = 54006849536.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.064462 seconds', 'result = 54006849536.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.280302 seconds', 'result = 182251831296.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.191399 seconds', 'result = 182251831296.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.509208 seconds', 'result = 431996534784.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.4263 seconds', 'result = 431996534784.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=random_uniform+config=optim+warmup=False
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.552631 seconds', 'result = 6754431488.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.894024 seconds', 'result = 54023852032.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.394226 seconds', 'result = 184425938944.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.870817 seconds', 'result = 439655661568.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.107416 seconds', 'result = 6754392576.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.163633 seconds', 'result = 54006804480.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.304741 seconds', 'result = 182251667456.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.526494 seconds', 'result = 431995944960.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.119625 seconds', 'result = 6754394624.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.203158 seconds', 'result = 54006800384.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.317646 seconds', 'result = 182251978752.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.544184 seconds', 'result = 431996076032.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=zeros+config=simple+warmup=False
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.632157 seconds', 'result = 0.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.901679 seconds', 'result = 0.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.345713 seconds', 'result = 0.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.707619 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.498451 seconds', 'result = 0.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 2.900121 seconds', 'result = 0.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 9.4296 seconds', 'result = 0.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 21.750406 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.523286 seconds', 'result = 0.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 2.887522 seconds', 'result = 0.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 9.377383 seconds', 'result = 0.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 21.639043 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=zeros+config=simple+warmup=True
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.520212 seconds', 'result = 0.0')
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.000172 seconds', 'result = 0.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.914485 seconds', 'result = 0.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 0.000166 seconds', 'result = 0.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.346122 seconds', 'result = 0.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 0.000207 seconds', 'result = 0.0', '*****WARMED UP*****')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.715376 seconds', 'result = 0.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 0.0002 seconds', 'result = 0.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.556841 seconds', 'result = 0.0')
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.000234 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (6000, 6000)', 'times = 2.936608 seconds', 'result = 0.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.000244 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (9000, 9000)', 'times = 9.34956 seconds', 'result = 0.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.000246 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:0', 'shape = (12000, 12000)', 'times = 21.634354 seconds', 'result = 0.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 0.000221 seconds', 'result = 0.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.562244 seconds', 'result = 0.0')
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.000255 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (6000, 6000)', 'times = 2.961658 seconds', 'result = 0.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.000237 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (9000, 9000)', 'times = 9.308582 seconds', 'result = 0.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.000239 seconds', 'result = 0.0', '*****WARMED UP*****')
('/gpu:1', 'shape = (12000, 12000)', 'times = 21.707127 seconds', 'result = 0.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.000261 seconds', 'result = 0.0', '*****WARMED UP*****')
++++++++++++++++++++++++++++++++++++++++++++++++++++



matrix=zeros+config=optim+warmup=False
('/cpu:0', 'shape = (3000, 3000)', 'times = 0.560451 seconds', 'result = 0.0')
('/cpu:0', 'shape = (6000, 6000)', 'times = 2.978946 seconds', 'result = 0.0')
('/cpu:0', 'shape = (9000, 9000)', 'times = 9.3279 seconds', 'result = 0.0')
('/cpu:0', 'shape = (12000, 12000)', 'times = 21.694664 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:0', 'shape = (3000, 3000)', 'times = 0.249778 seconds', 'result = 0.0')
('/gpu:0', 'shape = (6000, 6000)', 'times = 0.365332 seconds', 'result = 0.0')
('/gpu:0', 'shape = (9000, 9000)', 'times = 0.663667 seconds', 'result = 0.0')
('/gpu:0', 'shape = (12000, 12000)', 'times = 1.032716 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++
('/gpu:1', 'shape = (3000, 3000)', 'times = 0.299856 seconds', 'result = 0.0')
('/gpu:1', 'shape = (6000, 6000)', 'times = 0.294592 seconds', 'result = 0.0')
('/gpu:1', 'shape = (9000, 9000)', 'times = 0.55067 seconds', 'result = 0.0')
('/gpu:1', 'shape = (12000, 12000)', 'times = 0.806868 seconds', 'result = 0.0')
++++++++++++++++++++++++++++++++++++++++++++++++++++

---

谢谢你的建议，雅罗斯拉夫。我将尝试每种尺寸的热启动，并报告。这种热启动效应是否应该取决于操作，例如tf.zeros vs tf.random_uniform？从您的计时数字来看，它似乎取决于操作感谢您的建议Yaroslav。我将尝试每种尺寸的热启动，并报告。这种热启动效应是否应该取决于操作，例如tf.zeros vs tf.random_uniform？从您的计时数字来看，它似乎取决于操作