Java 我应该如何并行化计算代价高昂的for循环并整理迭代结果?
我在一台8核机器上工作,正在执行一项计算繁重的任务。然而,任务的每次执行(即for循环的迭代)都与前一个任务相当独立。从一次执行到下一次执行,只有一些变量被“汇总”。我猜这是并行化/线程化的一个很好的例子,但我不知道该怎么做 下面是代码的外观。现在,它只是我的主executor类中主方法的一部分:Java 我应该如何并行化计算代价高昂的for循环并整理迭代结果?,java,multithreading,performance,parallel-processing,Java,Multithreading,Performance,Parallel Processing,我在一台8核机器上工作,正在执行一项计算繁重的任务。然而,任务的每次执行(即for循环的迭代)都与前一个任务相当独立。从一次执行到下一次执行,只有一些变量被“汇总”。我猜这是并行化/线程化的一个很好的例子,但我不知道该怎么做 下面是代码的外观。现在,它只是我的主executor类中主方法的一部分: double testerPayoffSum = 0.0, developerPayoffSum = 0.0; Random seed = new Random(); try {
double testerPayoffSum = 0.0, developerPayoffSum = 0.0;
Random seed = new Random();
try {
for (int i = 0; i < GameConstants.MAX_GAMES; i++) {
EraserSimulator eraser = new EraserSimulator(GameConstants.MAX_TARGETS, GameConstants.MAX_RESOURCES, GameConstants.NUM_ATTACKER_TYPES, seed.nextInt());
Map<Set<SingleObjectiveTarget>, Double> gameStrategy = eraser.run();
assert (gameStrategy != null);
TestingGameSimulator testingGame = new TestingGameSimulator(GameConstants.MAX_TARGETS, gameStrategy, GameConstants.NUM_GAMES_TO_STORE_FOR_HISTORY, GameConstants.NUM_TESTING_GAMES_TO_PLAY);
PlayerPayoffs payoffs = testingGame.run(eraser.getEraserInstance());
testerPayoffSum += payoffs.getAverageTesterPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
developerPayoffSum += payoffs.getAverageDeveloperPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
System.out.print("Output: ERASER Games played; Number of developers caught");
System.out.print(", " + GameConstants.NUM_TESTING_GAMES_TO_PLAY + ", " + payoffs.getNumTimesCaught() + "\n");
} catch(Exception e){sendEmailAlert("Execution Failed with Exception");}
double testerPayoffSum=0.0,developerPayoffSum=0.0;
随机种子=新随机();
试一试{
对于(int i=0;i
如果可能的话,我想将for循环
计算并行化,并不断总结testerPayoffSum
和developerPayOffsum
变量。我该如何实现这一点
注意:for循环的每次执行大约需要20-30分钟,具体取决于输入大小(由各种
游戏常数设置)。即使是少量的MAX_游戏
上述操作也需要2-3个小时。创建一个线程对象,实现可调用
,该线程对象返回一个未来
对象,该对象包含测试人员支付额
和开发人员支付额
,开始计算并对从未来获得的结果求和e
s(另请参见)。创建一个线程对象,实现Callable
,该线程对象返回一个Future
对象,其中包含testerPayoffSum
和developerPayoffSum
,开始计算并对从Future
中获得的结果求和(另请参见)
1.使用的类不得共享任何变量
- 如果是这样,那么您必须添加锁
- 但它会影响性能
- 如果广泛使用某些共享变量
- 即使在非并行执行下,性能也会显著下降
2.使用过的类不得使用任何类型的机器学习
- 对此没有解决办法
- 因为并行化会破坏你的结果
现在如何做(我不是java编码器,所以我坚持C++代码)。
请注意,锁有时会导致应用程序冻结,尤其是在重载使用时
不管它是自己的锁还是操作系统锁
这主要发生在线程内部而不是主线程混合可视化内容或一些操作系统调用时
在这种情况下,有时适当的睡眠有助于避免线程内部的操作系统调用(如果可以的话)
因为它会引起很多其他问题
也要尽可能短的时间锁定,因为一旦发生冲突,冲突的线程就会停止
因此,不能只在循环开始时添加锁,在循环结束时添加解锁
因为并行加速将丢失
你绝对确定自己没有依赖关系吗
1.使用的类不得共享任何变量
- 如果是这样,那么您必须添加锁
- 但它会影响性能
- 如果广泛使用某些共享变量
- 即使在非并行执行下,性能也会显著下降
2.使用过的类不得使用任何类型的机器学习
- 对此没有解决办法
- 因为并行化会破坏你的结果
现在如何做(我不是java编码器,所以我坚持C++代码)。
请注意,锁有时会导致应用程序冻结,尤其是在重载使用时
不管它是自己的锁还是操作系统锁
这主要发生在线程内部而不是主线程混合可视化内容或一些操作系统调用时
在这种情况下,有时适当的睡眠有助于避免线程内部的操作系统调用(如果可以的话)
因为它会引起很多其他问题
也要尽可能短的时间锁定,因为一旦发生冲突,冲突的线程就会停止
因此,不能只在循环开始时添加锁,在循环结束时添加解锁
因为
//--- globals and headers -----------------------------------------------------
unsigned long __stdcall function(LPVOID p);
Random seed = new Random();
const int N=8; // threads count (<=CPU count)
int id[N]; // thread id
int max[N]; // number of games per thread
double testerPayoffSum[N]; // sum to separate variables to avoid locks need
double developerPayoffSum[N];
volatile int run=0,stop=0; // thread control variables run is number of running threads and stop force stop...
//--- main code ---------------------------------------------------------------
// init some variables ... may be the seed init will be better here too
int i;
for (i = 0; i < N; i++)
{
id[i]=i;
max[i]=GameConstants.MAX_GAMES / N;
testerPayoffSum[i]=0.0;
developerPayoffSum[i]=0.0;
}
max[0]=GameConstants.MAX_GAMES % N;
// create threads
for (i = 0; i < N; i++)
{
HANDLE hnd=CreateThread(0,0,function,&id[i],0,0);
if (hnd!=NULL) CloseHandle(hnd); // this line is important !!!
// because if you do not close Handle it will be allocated until the end of app
// handle leaks are nasty and cause weird OS behaviour
// I saw many times this bug in commercial drivers
// it is a nightmare for 24/7 software
}
// wait for them
while (run) Sleep(200);
// sum the results to [0]
for (i = 1; i < N; i++)
{
testerPayoffSum[0] +=testerPayoffSum[i];
developerPayoffSum[0]+=developerPayoffSum[i];
}
// here do what you need to do with the results
//--- thread function ---------------------------------------------------------
unsigned long __stdcall function(LPVOID p)
{
run++;
int ix=((int*)p)[0];
for (i = 0; i < max[ix]; i++)
{
if (stop) break;
EraserSimulator eraser = new EraserSimulator(GameConstants.MAX_TARGETS, GameConstants.MAX_RESOURCES, GameConstants.NUM_ATTACKER_TYPES, seed.nextInt());
Map<Set<SingleObjectiveTarget>, Double> gameStrategy = eraser.run();
assert (gameStrategy != null);
TestingGameSimulator testingGame = new TestingGameSimulator(GameConstants.MAX_TARGETS, gameStrategy, GameConstants.NUM_GAMES_TO_STORE_FOR_HISTORY, GameConstants.NUM_TESTING_GAMES_TO_PLAY);
PlayerPayoffs payoffs = testingGame.run(eraser.getEraserInstance());
testerPayoffSum[ix] += payoffs.getAverageTesterPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
developerPayoffSum[ix] += payoffs.getAverageDeveloperPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
// do not call any visual stuff from thread !!! sometimes it can cause a lot of problems ...
// instead cretae some global string variable and set it to what shoud be printed out
// and inside wait while loop in main code add if string != "" then System.out.print(string);
// but in that case you should add lock to it.
// System.out.print("Output: ERASER Games played; Number of developers caught");
// System.out.print(", " + GameConstants.NUM_TESTING_GAMES_TO_PLAY + ", " + payoffs.getNumTimesCaught() + "\n");
//Sleep(100); // well placed sleep
}
run--;
}
class _lock
{
public:
volatile bool locked;
_lock() { locked=false; }
void lock() { while(locked) Sleep(1); locked=true; }
void unlock() { locked=false; }
};
// now for each shared variable (or group of variables) add one global _lock variable
_lock l1; int sv1; // shared variable 1 and her lock
// any write access and sometimes also read access needs lock
l1.lock();
sv1++;
l1.unlock();
final ArrayBloclingQueue<PlayerPayoffs> queue=new ArrayBloclingQueue<PlayerPayoffs>();
Executor exec=new Executors.newFixedThreadPool(N); // number of threads depends on hardware
for (int i = 0; i < GameConstants.MAX_GAMES; i++) {
exec.execute(new Runnable(){
EraserSimulator eraser = new EraserSimulator(GameConstants.MAX_TARGETS, GameConstants.MAX_RESOURCES, GameConstants.NUM_ATTACKER_TYPES, seed.nextInt());
Map<Set<SingleObjectiveTarget>, Double> gameStrategy = eraser.run();
assert (gameStrategy != null);
TestingGameSimulator testingGame = new TestingGameSimulator(GameConstants.MAX_TARGETS, gameStrategy, GameConstants.NUM_GAMES_TO_STORE_FOR_HISTORY, GameConstants.NUM_TESTING_GAMES_TO_PLAY);
PlayerPayoffs payoffs = testingGame.run(eraser.getEraserInstance());
queue.put(payoffs);
});
}
double testerPayoffSum = 0.0, developerPayoffSum = 0.0;
for (int i = 0; i < GameConstants.MAX_GAMES; i++) {
PlayerPayoffs payoffs=queue.take();
testerPayoffSum += payoffs.getAverageTesterPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
developerPayoffSum += payoffs.getAverageDeveloperPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
System.out.print("Output: ERASER Games played; Number of developers caught");
System.out.print(", " + GameConstants.NUM_TESTING_GAMES_TO_PLAY + ", " + payoffs.getNumTimesCaught() + "\n");
}