Parallel processing 如何并行化此代码?
我最初是用ruby写的,后来发现MRI根本不支持并行执行。所以我用rust重新写了一遍,但在不同作品的所有权上遇到了麻烦Parallel processing 如何并行化此代码?,parallel-processing,rust,Parallel Processing,Rust,我最初是用ruby写的,后来发现MRI根本不支持并行执行。所以我用rust重新写了一遍,但在不同作品的所有权上遇到了麻烦 extern crate rand; extern crate csv; extern crate num_cpus; use std::fs::File; use csv::Reader; use rand::{thread_rng, sample}; use std::thread; use std::str::from_utf8; use std::io::{self
extern crate rand;
extern crate csv;
extern crate num_cpus;
use std::fs::File;
use csv::Reader;
use rand::{thread_rng, sample};
use std::thread;
use std::str::from_utf8;
use std::io::{self, Write};
use csv::index::{Indexed, create_index};
fn met_n_in_common(n:usize,csv:&mut Reader<File>)->usize{
csv.byte_records().map(|r| if(from_utf8(r.unwrap().get(n).unwrap()).unwrap() == "TRUE"){1}else{0}).fold(0usize, |sum, i| sum + i)
}
fn mets_in_common(csv:&mut Reader<File>,current_set_length:usize)->usize {
(0..csv.headers().unwrap().len()).map(|i| if(i == 0){0}else{met_n_in_common(i,csv)} ).filter(|&e| e==current_set_length ).count()
}
fn main() {
let csv_s = || csv::Reader::from_file("/Users/camdennarzt/Documents/All 7000 series-Table 1-1-1-3.csv").unwrap();
let mut csv = csv_s();
let mut index_data = io::Cursor::new(Vec::new());
create_index(csv_s(), index_data.by_ref()).unwrap();
let mut index = Indexed::open(csv_s(), index_data).unwrap();
let mut tried_indices = Vec::new();
let mut threads : Vec<_> = (0..num_cpus::get()).map(|i|{
thread::spawn(move || {
let mut best_set : Vec<Vec<String>> = Vec::new();
let mut best_count = 0;
let mut rng = thread_rng();
let mut indices = Vec::new();
let limit = 2usize.pow(10)/num_cpus::get();
for _ in (0..limit) {
while {
let count = *sample(&mut rng, 13..83, 1).first().unwrap();
indices = sample(&mut rng, 1..83, count);
tried_indices.contains(&indices)
}{}
tried_indices.push(indices.to_owned());
let current_set:Vec<_> = indices.iter().map(|&i|{
index.seek(i).unwrap();
index.records().next().unwrap().unwrap()
}).collect();
let current_count = mets_in_common(&mut csv,current_set.len());
if (current_count > best_count){
best_count = current_count;
best_set = current_set;
}
}
(best_count,best_set.iter().map(|r| *r.first().unwrap()).collect::<Vec<String>>())
})
}).collect();
}
extern板条箱兰德;
外部板条箱;
外部机箱数量(CPU);
使用std::fs::File;
使用csv::Reader;
使用rand::{thread\rng,sample};
使用std::线程;
使用std::str::from_utf8;
使用std::io::{self,Write};
使用csv::index::{index,create_index};
fn符合通用标准(n:usize、csv:&mut Reader)->usize{
csv.byte_records().map(| r|if(从_utf8(r.unwrap().get(n.unwrap()).unwrap()=“TRUE”){1}否则{0})。折叠(0usize,| sum,i | sum+i)
}
fn mets_in_common(csv:&mut读取器,当前设置_长度:usize)->usize{
(0..csv.headers().unwrap().len()).map(|i | if(i==0){0}else{met_n_in_common(i,csv)}).filter(|&e | e==当前设置长度).count()
}
fn main(){
让csv|s=|csv::Reader::from|u文件(“/Users/camdennarzt/Documents/All 7000 series Table 1-1-1-3.csv”).unwrap();
让mut csv=csv_s();
让mut index_data=io::Cursor::new(Vec::new());
创建索引(csv_s(),索引_data.by_ref()).unwrap();
让mut index=index::open(csv_s(),index_data).unwrap();
让mut\u index=Vec::new();
让mut线程:Vec=(0..num_CPU::get()).map(| i|{
线程::生成(移动| |{
设mut best_set:Vec=Vec::new();
让mut best_count=0;
设mut rng=thread_rng();
让mut index=Vec::new();
让limit=2usize.pow(10)/num_CPU::get();
对于uuin(0..limit){
当{
let count=*sample(&mut rng,13..83,1).first().unwrap();
指数=样本(&mut rng,1..83,计数);
已尝试索引。包含(&索引)
}{}
已尝试将索引推送(index.to_owned());
让当前设置:Vec=index.iter().map(|&i)|{
index.seek(i.unwrap();
index.records().next().unwrap().unwrap()
}).收集();
让current_count=mets_in_common(&mut csv,current_set.len());
如果(当前计数>最佳计数){
最佳计数=当前计数;
最佳_集=当前_集;
}
}
(最佳计数,最佳集合.iter().map(| r |*r.first().unwrap()).collect::())
})
}).收集();
}
具体来说,当我编译这个(rust 1.2 stable)时,我得到:
-*-模式:编译;默认目录:“~/Developer/Rust/optimal\u subset\u finder/src/”-*-
编译于8月19日星期三14:55:10开始
货物建造
编译最佳_子集_查找器v0.1.0(file:///Users/camdennarzt/Developer/Rust/optimal_subset_finder)
main.rs:31:23:56:10错误:无法移出'FnMut'闭包中捕获的外部变量
main.rs:31线程::繁殖(移动| |{
main.rs:32让mut-best_-set:Vec=Vec::new();
main.rs:33让mut最佳计数=0;
main.rs:34让mut rng=thread_rng();
rs:35让mut index=Vec::new();
main.rs:36 let limit=2usize.pow(10)/num_CPU::get();
...
注:在闭合膨胀的膨胀中
main.rs:31:23:56:10注:扩建场地
注:在闭合膨胀的膨胀中
主体:rs:30:57:57:6注:扩建场地
main.rs:31:23:56:10错误:无法移出'FnMut'闭包中捕获的外部变量
main.rs:31线程::繁殖(移动| |{
main.rs:32让mut-best_-set:Vec=Vec::new();
main.rs:33让mut最佳计数=0;
main.rs:34让mut rng=thread_rng();
rs:35让mut index=Vec::new();
main.rs:36 let limit=2usize.pow(10)/num_CPU::get();
...
注:在闭合膨胀的膨胀中
main.rs:31:23:56:10注:扩建场地
注:在闭合膨胀的膨胀中
主体:rs:30:57:57:6注:扩建场地
main.rs:31:23:56:10错误:无法移出'FnMut'闭包中捕获的外部变量
main.rs:31线程::繁殖(移动| |{
main.rs:32让mut-best_-set:Vec=Vec::new();
main.rs:33让mut最佳计数=0;
main.rs:34让mut rng=thread_rng();
rs:35让mut index=Vec::new();
main.rs:36 let limit=2usize.pow(10)/num_CPU::get();
...
注:在闭合膨胀的膨胀中
main.rs:31:23:56:10注:扩建场地
注:在闭合膨胀的膨胀中
主体:rs:30:57:57:6注:扩建场地
main.rs:55:49:55:68错误:无法移出借用内容
main.rs:55(最佳计数,最佳集合.iter().map(|r |*r.first().unwrap())。收集:())
^~~~~~~~~~~~~~~~~~~
注:在闭合膨胀的膨胀中
main.rs:55:45:55:68注:扩建场地
注:在闭合膨胀的膨胀中
main.rs:31:23:56:10注:扩建场地
注:在闭合膨胀的膨胀中
主体:rs:30:57:57:6注:扩建场地
错误:由于之前的4个错误而中止
无法编译“最优子集查找器”。
要了解更多信息,请使用--verbose再次运行该命令。
编译在8月19日星期三14:55:10异常退出,代码为101
如果我完全注释掉线程,那么它将编译并运行。但是我无法解析文档以使线程正常工作。而且,当需要知道如何解决此问题时,这些错误也没有特别大的帮助。您的代码中有几个问题 首先,您显示的实际错误是由于您试图从
&Vec
中获取字符串
。如果不进行克隆,这是不可能的,因为这需要将字符串
从&Vec
中移出,但您无法移出引用的数据。因为最佳设置
在它是一个内部线程,在您从它返回数据后立即被删除,您可以安全地使用它
-*- mode: compilation; default-directory: "~/Developer/Rust/optimal_subset_finder/src/" -*-
Compilation started at Wed Aug 19 14:55:10
cargo build
Compiling optimal_subset_finder v0.1.0 (file:///Users/camdennarzt/Developer/Rust/optimal_subset_finder)
main.rs:31:23: 56:10 error: cannot move out of captured outer variable in an `FnMut` closure
main.rs:31 thread::spawn(move || {
main.rs:32 let mut best_set : Vec<Vec<String>> = Vec::new();
main.rs:33 let mut best_count = 0;
main.rs:34 let mut rng = thread_rng();
main.rs:35 let mut indices = Vec::new();
main.rs:36 let limit = 2usize.pow(10)/num_cpus::get();
...
note: in expansion of closure expansion
main.rs:31:23: 56:10 note: expansion site
note: in expansion of closure expansion
main.rs:30:57: 57:6 note: expansion site
main.rs:31:23: 56:10 error: cannot move out of captured outer variable in an `FnMut` closure
main.rs:31 thread::spawn(move || {
main.rs:32 let mut best_set : Vec<Vec<String>> = Vec::new();
main.rs:33 let mut best_count = 0;
main.rs:34 let mut rng = thread_rng();
main.rs:35 let mut indices = Vec::new();
main.rs:36 let limit = 2usize.pow(10)/num_cpus::get();
...
note: in expansion of closure expansion
main.rs:31:23: 56:10 note: expansion site
note: in expansion of closure expansion
main.rs:30:57: 57:6 note: expansion site
main.rs:31:23: 56:10 error: cannot move out of captured outer variable in an `FnMut` closure
main.rs:31 thread::spawn(move || {
main.rs:32 let mut best_set : Vec<Vec<String>> = Vec::new();
main.rs:33 let mut best_count = 0;
main.rs:34 let mut rng = thread_rng();
main.rs:35 let mut indices = Vec::new();
main.rs:36 let limit = 2usize.pow(10)/num_cpus::get();
...
note: in expansion of closure expansion
main.rs:31:23: 56:10 note: expansion site
note: in expansion of closure expansion
main.rs:30:57: 57:6 note: expansion site
main.rs:55:49: 55:68 error: cannot move out of borrowed content
main.rs:55 (best_count,best_set.iter().map(|r| *r.first().unwrap()).collect::<Vec<String>>())
^~~~~~~~~~~~~~~~~~~
note: in expansion of closure expansion
main.rs:55:45: 55:68 note: expansion site
note: in expansion of closure expansion
main.rs:31:23: 56:10 note: expansion site
note: in expansion of closure expansion
main.rs:30:57: 57:6 note: expansion site
error: aborting due to 4 previous errors
Could not compile `optimal_subset_finder`.
To learn more, run the command again with --verbose.
Compilation exited abnormally with code 101 at Wed Aug 19 14:55:10
best_set.into_iter()
.flat_map(|r| r.take(1))
.collect::<Vec<String>>()
let index = Arc::new(Mutex::new(Indexed::open(csv_s(), index_data).unwrap()));
...
let index = index.clone();
thread::spawn(move || {
...
let current_set:Vec<_> = {
let index = index.lock();
indices.iter().map(|&i| {
index.seek(i).unwrap();
index.records().next().unwrap().unwrap()
}).collect()
};
...
});
let mut csv = csv_s();
thread::spawn(move || {
...
});