Haskell-使用来自真实世界Haskell的mapreduce框架(Control.Parallel.Strategies)进行并行字计数
我是一名学生,在Haskell做一个关于并行性和并发性的作业。作为作业的一部分,我们收到了这段代码(源于真实世界Haskell的第24章),我们被要求接受这段代码并编写一个并行字数计算程序:Haskell-使用来自真实世界Haskell的mapreduce框架(Control.Parallel.Strategies)进行并行字计数,haskell,concurrency,parallel-processing,mapreduce,Haskell,Concurrency,Parallel Processing,Mapreduce,我是一名学生,在Haskell做一个关于并行性和并发性的作业。作为作业的一部分,我们收到了这段代码(源于真实世界Haskell的第24章),我们被要求接受这段代码并编写一个并行字数计算程序: -- file: ch24/MapReduce.hs mapReduce :: Strategy b -- evaluation strategy for mapping -> (a -> b) -- map function -> Strategy
-- file: ch24/MapReduce.hs
mapReduce
:: Strategy b -- evaluation strategy for mapping
-> (a -> b) -- map function
-> Strategy c -- evaluation strategy for reduction
-> ([b] -> c) -- reduce function
-> [a] -- list to map over
-> c
-- file: ch24/MapReduce.hs
mapReduce mapStrat mapFunc reduceStrat reduceFunc input =
mapResult `pseq` reduceResult
where mapResult = parMap mapStrat mapFunc input
reduceResult = reduceFunc mapResult `using` reduceStrat
顺序版本:
我编写了一个程序的顺序版本,它可以工作:
import System.Environment
import System.IO
import System.Directory
import Data.Char (toLower)
import Data.List (sort, group)
import Control.Arrow ((&&&))
import Data.Map as Map
simpleMapReduce
:: (a -> b) -- map function
-> ([b] -> c) -- reduce function
-> [a] -- list to map over
-> c -- result
simpleMapReduce mapFunc reduceFunc = reduceFunc . Prelude.map mapFunc
stringToWordCountMap :: String -> Map.Map String Int
stringToWordCountMap = Map.fromList . Prelude.map (head &&& length) . group . sort . words . Prelude.map toLower
combineWordCountMaps :: Map.Map String Int -> Map.Map String Int -> Map.Map String Int
combineWordCountMaps map1 map2 = Map.unionWith (+) map1 map2
reduceWordCountMaps :: [Map.Map String Int] -> Map.Map String Int
reduceWordCountMaps (x:[]) = x
reduceWordCountMaps (x:xs) = combineWordCountMaps x (reduceWordCountMaps xs)
main = do (fileName:_) <- getArgs
fileExists <- doesFileExist fileName
if fileExists
then do contents <- readFile fileName
let fileInLines = lines contents
result = simpleMapReduce stringToWordCountMap reduceWordCountMaps fileInLines
putStrLn $ "The file has " ++ show (length (lines contents)) ++ " lines!"
putStrLn $ "result = " ++ show result ++ "."
else do putStrLn "The file doesn't exist!"
我想知道是否有人可以看看这个,并帮助我让这个工作?我对haskell还不是很有经验,所以如果我遗漏了一些明显的东西,我深表歉意。我发现这些策略有点混乱,因此任何链接/资源都将不胜感激。非常感谢
最终编辑:
根据user5402的回答:
在复制粘贴代码时,我也遇到了一些缩进错误。此处提供了仅提供警告的版本
只需将此粘贴到下面,以防其消失:
import System.Environment
import System.IO
import System.Directory
import Data.Char (toLower)
import Data.List (sort, group)
import Control.Arrow ((&&&))
import Data.Map as Map
import Control.Parallel
import Control.Parallel.Strategies
mapReduce
:: Strategy b -- evaluation strategy for mapping
-> (a -> b) -- map function
-> Strategy c -- evaluation strategy for reduction
-> ([b] -> c) -- reduce function
-> [a] -- list to map over
-> c
-- file: ch24/MapReduce.hs
mapReduce mapStrat mapFunc reduceStrat reduceFunc input =
mapResult `pseq` reduceResult
where mapResult = parMap mapStrat mapFunc input
reduceResult = reduceFunc mapResult `using` reduceStrat
stringToWordCountMap :: String -> Map.Map String Int
stringToWordCountMap = Map.fromList . Prelude.map (head &&& length) . group . sort . words . Prelude.map toLower
combineWordCountMaps :: Map.Map String Int -> Map.Map String Int -> Map.Map String Int
combineWordCountMaps map1 map2 = Map.unionWith (+) map1 map2
reduceWordCountMaps :: [ Map.Map String Int] -> Map.Map String Int
reduceWordCountMaps (x:[]) = x
reduceWordCountMaps (x:xs) = combineWordCountMaps x (reduceWordCountMaps xs)
main = do (fileName:_) <- getArgs
fileExists <- doesFileExist fileName
if fileExists
then do contents <- readFile fileName
let fileInLines = lines contents
result = mapReduce rpar stringToWordCountMap rpar reduceWordCountMaps fileInLines
putStrLn $ "The file has " ++ show (length (lines contents)) ++ " lines!"
putStrLn $ "result = " ++ show result ++ "."
else do putStrLn "The file doesn't exist!"
导入系统环境
导入系统.IO
导入系统目录
导入Data.Char(toLower)
导入数据列表(排序、组)
导入控件.Arrow(&&&&))
导入数据。映射为映射
进口管制
进口管制.平行战略
地图还原
::策略b——映射的评估策略
->(a->b)--映射功能
->战略c——减排评估战略
->([b]->c)--减少功能
->[a]--要映射的列表
->c
--文件:ch24/MapReduce.hs
mapReduce mapStrat mapFunc reduceStrat reduceFunc输入=
mapResult`pseq`reduceResult
其中mapResult=parMap mapStrat mapFunc input
reduceResult=reduceFunc mapResult`using`reduceResult`
StringToOrdCountMap::String->Map.Map字符串Int
StringToOrdCountMap=Map.fromList。Prelude.map(头部和长度)。小组。分类话。前奏曲.托洛尔地图
combineWordCountMaps::Map.Map字符串Int->Map.Map字符串Int->Map.Map字符串Int->Map.Map字符串Int
combineWordCountMaps map1 map2=Map.Union与(+)map1 map2
reduceWordCountMaps::[Map.Map字符串Int]->Map.Map字符串Int
reduceWordCountMaps(x:[])=x
reduceWordCountMaps(x:xs)=combineWordCountMaps x(reduceWordCountMaps xs)
main=do(文件名:)在此行中:
result = mapReduce Control.Parallel.Strategies.parMap stringToWordCountMap Control.Parallel.Strategies.parList reduceWordCountMaps fileInLines
对于策略
应提供类似rpar
或rseq
的值。有关其他已定义的策略,请参阅。所以上面的一行应该是这样的:
result = mapReduce rpar stringToWordCountMap rpar reduceWordCountMaps fileInLines
在复制粘贴代码时,我也遇到了一些缩进错误。仅提供警告的版本可用
注:不需要完全限定名称<代码> PARMAP ,<代码> RPAR < /代码>等。如果有冲突,请考虑使用.< /P>“这是我尝试编写一个上面的并行版本,我甚至无法编译。”你得到什么错误消息?对不起,我应该澄清,我得到这个(很长)。错误信息:对不起,我应该澄清一下,我收到了一条很长的错误信息,我已经添加到了问题中。我发布的代码实际上是我尝试了很多事情的结果,但是,我真的认为它不应该像现在这样工作。嗨,非常感谢你的回答,很抱歉直到现在才回复。我刚试过,效果很好,非常感谢你的帮助!
result = mapReduce Control.Parallel.Strategies.parMap stringToWordCountMap Control.Parallel.Strategies.parList reduceWordCountMaps fileInLines
result = mapReduce rpar stringToWordCountMap rpar reduceWordCountMaps fileInLines