Performance 解析haskell中的二进制文件时性能不佳

Performance 解析haskell中的二进制文件时性能不佳,performance,haskell,binaryfiles,Performance,Haskell,Binaryfiles,我将一组二进制记录打包到一个文件中,并使用Data.ByteString.Lazy和Data.binary.Get读取它们。在我当前的实现中,一个8Mb的文件需要6秒钟才能解析 import qualified Data.ByteString.Lazy as BL import Data.Binary.Get data Trade = Trade { timestamp :: Int, price :: Int , qty :: Int } deriving (Show) getTrade

我将一组二进制记录打包到一个文件中,并使用Data.ByteString.Lazy和Data.binary.Get读取它们。在我当前的实现中,一个8Mb的文件需要6秒钟才能解析

import qualified Data.ByteString.Lazy as BL
import Data.Binary.Get

data Trade = Trade { timestamp :: Int, price :: Int ,  qty :: Int } deriving (Show)

getTrades = do
  empty <- isEmpty
  if empty
    then return []
    else do
      timestamp <- getWord32le          
      price <- getWord32le
      qty <- getWord16le          
      rest <- getTrades
      let trade = Trade (fromIntegral timestamp) (fromIntegral price) (fromIntegral qty)
      return (trade : rest)

main :: IO()
main = do
  input <- BL.readFile "trades.bin" 
  let trades = runGet getTrades input
  print $ length trades
将限定数据.ByteString.Lazy导入为BL
导入Data.Binary.Get
数据交易=交易{时间戳::Int,价格::Int,数量::Int}衍生(显示)
getTrades=do

空您的代码在不到一秒钟的时间内解码8MB文件(ghc-7.4.1)-当然,我是用
-O2
编译的

然而,它需要过多的堆栈空间。你可以减少

  • 时间
  • 堆栈空间
  • 堆空间
需要在适当的位置增加更多的严格性,并使用累加器收集到目前为止解析的交易

{-# LANGUAGE BangPatterns #-}
module Main (main) where

import qualified Data.ByteString.Lazy as BL
import Data.Binary.Get

data Trade = Trade { timestamp :: {-# UNPACK #-} !Int
                   , price :: {-# UNPACK #-} !Int 
                   , qty :: {-# UNPACK #-} !Int
                   } deriving (Show)

getTrades :: Get [Trade]
getTrades = go []
  where
    go !acc = do
      empty <- isEmpty
      if empty
        then return $! reverse acc
        else do
          !timestamp <- getWord32le
          !price <- getWord32le
          !qty <- getWord16le
          let !trade = Trade (fromIntegral timestamp) (fromIntegral price) (fromIntegral qty)
          go (trade : acc)

main :: IO()
main = do
  input <- BL.readFile "trades.bin"
  let trades = runGet getTrades input
  print $ length trades
{-#语言模式}
主模块(Main),其中
将限定数据.ByteString.Lazy导入为BL
导入Data.Binary.Get
数据交易=交易{时间戳:{-#解包#-}!Int
,价格:{-#解包{-}!Int
,数量:{-#解包{-}!Int
}派生(显示)
getTrades::获取[交易]
getTrades=go[]
哪里
走!acc=do
empty稍微重构它(基本上是左折)可以提供更好的性能,并在解析8388600字节的文件时大大降低GC开销

{-# LANGUAGE BangPatterns #-}
module Main (main) where

import qualified Data.ByteString.Lazy as BL
import Data.Binary.Get

data Trade = Trade
  { timestamp :: {-# UNPACK #-} !Int
  , price     :: {-# UNPACK #-} !Int 
  , qty       :: {-# UNPACK #-} !Int
  } deriving (Show)

getTrade :: Get Trade
getTrade = do
  timestamp <- getWord32le
  price     <- getWord32le
  qty       <- getWord16le
  return $! Trade (fromIntegral timestamp) (fromIntegral price) (fromIntegral qty)

countTrades :: BL.ByteString -> Int
countTrades input = stepper (0, input) where
  stepper (!count, !buffer)
    | BL.null buffer = count
    | otherwise      =
        let (trade, rest, _) = runGetState getTrade buffer 0
        in stepper (count+1, rest)

main :: IO()
main = do
  input <- BL.readFile "trades.bin"
  let trades = countTrades input
  print trades
{-#语言模式}
主模块(Main),其中
将限定数据.ByteString.Lazy导入为BL
导入Data.Binary.Get
数据贸易=贸易
{时间戳::{-#解包#-}!Int
,价格:{-#解包{-}!Int
,数量:{-#解包{-}!Int
}派生(显示)
得到交易
getTrade=do

时间戳在真实世界的haskell中有一章是关于评测的,还有一些问题在so.com上标记为[haskell]+[performance]——也许这对你有帮助。@epsilonhalbe谢谢,我确实进行了很好的搜索,这个模式就是Data.Binary.Get文档中的模式。我怀疑这是一个“几乎是跟踪递归”的问题,但我有点想不出来。这是一个棘手的问题,因为Data.Binary.Get看起来很严格-我之前曾说过要尝试更好地懒散,但我已经删除了它,因为它不适用。Daniel Fischer的回答向您展示了如何更好地做到严格。非常感谢您的回答!你帮了一个noob一点忙。 426,003,680 bytes allocated in the heap 443,141,672 bytes copied during GC 99,305,920 bytes maximum residency (9 sample(s)) 203 MB total memory in use (0 MB lost due to fragmentation) Total time 0.62s ( 0.81s elapsed) %GC time 83.3% (86.4% elapsed) 357,851,536 bytes allocated in the heap 220,009,088 bytes copied during GC 40,846,168 bytes maximum residency (8 sample(s)) 85 MB total memory in use (0 MB lost due to fragmentation) Total time 0.24s ( 0.28s elapsed) %GC time 69.1% (71.4% elapsed) 290,725,952 bytes allocated in the heap 109,592 bytes copied during GC 78,704 bytes maximum residency (10 sample(s)) 2 MB total memory in use (0 MB lost due to fragmentation) Total time 0.06s ( 0.07s elapsed) %GC time 5.0% (6.0% elapsed)