使用boostxpressive降低性能
最近我一直在使用boostxpressive解析文件。这些文件每个都有10MB，将有几百个需要解析 Xpressive很好用，语法清晰，但性能会带来问题。令人难以置信的是，它在调试版本中是如何爬行的，而在发布版本中，它在每个文件上花费的时间超过一整秒钟。我已经针对旧的普通get_line（）、find（）和sscanf（）代码进行了测试，它可以轻松击败xpressive 我知道类型检查、回溯等都有成本，但这对我来说似乎太过分了。我多么奇怪，我做错了什么？以合适的速度运行是否有任何优化方法？它是否应该努力将代码迁移到boost:：spirit 我准备了一个lite版本的代码，其中嵌入了几行真实的文件，以防有人测试和帮助 注意-作为一项要求，必须使用VS2010（不幸的是，不完全符合c++11）使用boostxpressive降低性能,boost,boost-spirit,boost-xpressive,Boost,Boost Spirit,Boost Xpressive,最近我一直在使用boostxpressive解析文件。这些文件每个都有10MB，将有几百个需要解析 Xpressive很好用，语法清晰，但性能会带来问题。令人难以置信的是，它在调试版本中是如何爬行的，而在发布版本中，它在每个文件上花费的时间超过一整秒钟。我已经针对旧的普通get_line（）、find（）和sscanf（）代码进行了测试，它可以轻松击败xpressive 我知道类型检查、回溯等都有成本，但这对我来说似乎太过分了。我多么奇怪，我做错了什么？以合适的速度运行是否有任何优化方法？它是否
#包括
#包括
常量字符输入[]=“[2018-Mar-13 13:13:59.580482]-0.200秒=>驾驶员：0-速度：0.0-道路：BTN-1002-公里：90.0-打开：1\n\
[2018-Mar-13 13:13:59.580482]-0.200s=>驾驶员：0-速度：0.0-道路：A-11-公里：90.0-坡度：0\n\
[2018-Mar-13 13:14:01.170203]-1.790 s=>驾驶员：0-速度：0.0-道路：A-11-公里：90.0-档位：0\n\
[2018-Mar-13 13:14:01.170203]-1.790 s=>驾驶员：0-速度：0.1-道路：A-11-公里：90.0-档位：1\n\
[2018-Mar-13 13:14:01.819966]-2.440 s=>驾驶员：0-速度：0.1-道路：A-11-公里：90.0-顺序：1\n\
[2018-Mar-13 13:14:01.819966]-2.440 s=>驾驶员：0-速度：0.2-道路：A-11-公里：90.0-离合器：1\n\
[2018-Mar-13 13:14:01.819966]-2.540 s=>注册表备份\n\
[2018-Mar-13 13:14:02.409855]-3.030 s=>驾驶员：0-速度：0.2-道路：A-11-公里：90.0-顺序：4\n\
[2018-Mar-13 13:14:02.409855]-3.030 s=>驾驶员：0-速度：0.3-道路：A-11-公里：90.0-顺序：8\n\
[2018-Mar-13 13:14:01.819966]-3.110 s=>注册表备份\n\
[2018-Mar-13 13:14:02.620424]-3.240 s=>驾驶员：0-速度：0.4-道路：A-11-公里：90.1-顺序：15\n\
[2018-Mar-13 13:14:02.829983]-3.450秒=>驾驶员：0-速度：0.6-道路：B-302-公里：90.1-坡度：-5\n\
[2018-Mar-13 13:14:03.039600]-3.660 s=>驾驶员：0-速度：0.8-道路：B-302-公里：90.1-顺序：21\n\
[2018-Mar-13 13:14:03.250451]-3.870 s=>驾驶员：0-速度：1.2-道路：B-302-公里：90.2-档位：2\n\
[2018-Mar-13 13:14:03.460012]-4.080 s=>驾驶员：0-速度：1.7-道路：B-302-公里：90.3-顺序：29\n\
[2018-Mar-13 13:14:03.669448]-4.290 s=>驾驶员：0-速度：2.2-道路：B-302-公里：90.4-顺序：34\n\
[2018-Mar-13 13:14:03.880066]-4.500秒=>驾驶员：0-速度：2.8-道路：B-302-公里：90.5-离合器：1\n\
[2018-Mar-13 13:14:04.090444]-4.710 s=>驾驶员：0-速度：3.5-道路：B-302-公里：90.7-顺序：45\n\
[2018-Mar-13 13:14:04.300160]-4.920 s=>驾驶员：0-速度：4.2-道路：B-302-公里：90.9-坡度：10\n\
[2018-Mar-13 13:14:04.510025]-5.130秒=>驾驶员：0-速度：4.9-道路：B-302-公里：91.1-档位：3”；
const auto len=std:：distance（std:：begin（输入），std:：end（输入））；
结构序列
{
int-ms；
int驱动程序；
int序列；
双倍时间；
双水平；
双公里；
std：：字符串日期；
标准:弦道;；
};
命名空间xp=boost:：xpressive；
int main（）
{
序列数据；
std：：向量序列；
使用命名空间xp；
cregex real=（+\u d>'.>>+\u d）；
cregex关键字=“-SEQUENCE:”>（+_d）[xp:：ref（data.SEQUENCE）=as（_）]；
cregex date=repeat（_d）>>'-'>>repeat（alpha）>>'-'>>repeat（_d）>>>>repeat（_d）>>>'：'>>repeat（_d）>>>'：'>>repeat（_d）；
cregex头文件='['>>日期[xp:：ref（data.date）=\]>'.>>（+\[xp:：ref（data.ms）=as（\]>>“]-”
>>real[xp:：ref（data.time）=as（ux）]
>>“s=>Driver:”>>（+\ud）[xp:：ref（data.Driver）=as（\ud）]
>>“-Speed:”>>real[xp:：ref（data.vel）=as（u）]
>>“-Road:”>>（+set[alnum |'-']）[xp:：ref（data.Road）=]
>>“-Km:”>>real[xp:：ref（data.Km）=as（u3;）]；
xp:：cregex解析器=（标题>>关键字>>\u ln）；
cregex_迭代器cur（输入，输入+len，解析器）；
xp:：cregex_迭代器端；
对于（；cur！=end；++cur）
序列。安置（数据）；
返回0；
}
请注意VS 2010的限制。我大致看到了两个需要改进的方面：
- 您基本上解析所有行，包括您不感兴趣的行
- 你分配了很多字符串
cregex signature = -*~_n >> " - SEQUENCE: " >> (+_d) >> before(_ln|eos);
for (xp::cregex_iterator cur(b, e, signature), end; cur != end; ++cur) {
std::cout << "'" << cur->str() << "'\n";
}
template <typename It> struct QiParser : qi::grammar<It, Sequence()> {
QiParser() : QiParser::base_type(line) {
using namespace qi;
auto date_time = copy(
repeat(4)[digit] >> '-' >> repeat(3)[alpha] >> '-' >> repeat(2)[digit] >> ' ' >>
repeat(2)[digit] >> ':' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> '.' >> +digit);
line = '[' >> raw[date_time] >> "] - "
>> double_ >> " s"
>> " => Driver: " >> int_
>> " - Speed: " >> double_
>> " - Road: " >> raw[+graph]
>> " - Km: " >> double_
>> " - SEQUENCE: " >> int_
>> (eol|eoi);
}
private:
qi::rule<It, Sequence()> line;
};
boost:：string\u viewnamespace boost { namespace spirit { namespace traits {
template <typename It>
struct assign_to_attribute_from_iterators<boost::string_view, It, void> {
static inline void call(It f, It l, boost::string_view& attr) { attr = boost::string_view { &*f, size_t(std::distance(f,l)) }; }
};
} } }
- 完整的交互式图表如下：
- 使用
编译-DUSE_NONIUS - 使用
-DVERI编译namespace boost { namespace spirit { namespace traits { template <typename It> struct assign_to_attribute_from_iterators<boost::string_view, It, void> { static inline void call(It f, It l, boost::string_view& attr) { attr = boost::string_view { &*f, size_t(std::distance(f,l)) }; } }; } } }template <typename It> struct QiParser : qi::grammar<It, Sequence()> { QiParser() : QiParser::base_type(line) { using namespace qi; auto date_time = copy( repeat(4)[digit] >> '-' >> repeat(3)[alpha] >> '-' >> repeat(2)[digit] >> ' ' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> '.' >> +digit); line = '[' >> raw[date_time] >> "] - " >> double_ >> " s" >> " => Driver: " >> int_ >> " - Speed: " >> double_ >> " - Road: " >> raw[+graph] >> " - Km: " >> double_ >> " - SEQUENCE: " >> int_ >> (eol|eoi); } private: qi::rule<It, Sequence()> line; };#include <boost/fusion/adapted/struct.hpp> #include <boost/spirit/include/qi.hpp> #include <boost/spirit/repository/include/qi_seek.hpp> #include <boost/utility/string_view.hpp> #include <cstring> // strlen using It = char const*; struct Sequence { int driver; int sequence; double time; double vel; double km; boost::string_view date; boost::string_view road; }; BOOST_FUSION_ADAPT_STRUCT(::Sequence, date, time, driver, vel, road, km, sequence) namespace qi = boost::spirit::qi; namespace boost { namespace spirit { namespace traits { template <typename It> struct assign_to_attribute_from_iterators<boost::string_view, It, void> { static inline void call(It f, It l, boost::string_view& attr) { attr = boost::string_view { &*f, size_t(std::distance(f,l)) }; } }; } } } std::vector<Sequence> parse_spirit(It b, It e) { qi::rule<It, Sequence()> static const line = []{ using namespace qi; auto date_time = copy( repeat(4)[digit] >> '-' >> repeat(3)[alpha] >> '-' >> repeat(2)[digit] >> ' ' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> '.' >> +digit); qi::rule<It, Sequence()> r = '[' >> raw[date_time] >> "] - " >> double_ >> " s" >> " => Driver: " >> int_ >> " - Speed: " >> double_ >> " - Road: " >> raw[+graph] >> " - Km: " >> double_ >> " - SEQUENCE: " >> int_ >> (eol|eoi); return r; }(); std::vector<Sequence> sequences; parse(b, e, *boost::spirit::repository::qi::seek[line], sequences); return sequences; } static char input[] = /*... see question ...*/; static const size_t len = strlen(input); int main() { auto sequences = parse_spirit(input, input+len); std::cout << "Parsed: " << sequences.size() << " sequence lines\n"; }#include <boost/fusion/adapted/struct.hpp> #include <boost/spirit/include/qi.hpp> #include <boost/spirit/repository/include/qi_seek.hpp> #include <boost/utility/string_view.hpp> #include <cstring> // strlen using It = char const*; struct BaseEvent { int driver; int sequence; double time; double vel; double km; boost::string_view date; boost::string_view road; }; struct Sequence : BaseEvent{}; struct Clutch : BaseEvent{}; struct Gear : BaseEvent{}; BOOST_FUSION_ADAPT_STRUCT(::Sequence, date, time, driver, vel, road, km, sequence) BOOST_FUSION_ADAPT_STRUCT(::Clutch, date, time, driver, vel, road, km, sequence) BOOST_FUSION_ADAPT_STRUCT(::Gear, date, time, driver, vel, road, km, sequence) struct LogEvents { std::vector<Sequence> sequence; std::vector<Clutch> clutch; std::vector<Gear> gear; void add(Sequence const& s) { sequence.push_back(s); } void add(Clutch const& c) { clutch.push_back(c); } void add(Gear const& g) { gear.push_back(g); } }; namespace qi = boost::spirit::qi; namespace boost { namespace spirit { namespace traits { template <typename It> struct assign_to_attribute_from_iterators<boost::string_view, It, void> { static inline void call(It f, It l, boost::string_view& attr) { attr = boost::string_view { &*f, size_t(std::distance(f,l)) }; } }; template <> struct is_container<LogEvents> : std::true_type {}; template <> struct container_value<LogEvents> { using type = boost::variant<::Sequence, ::Clutch, ::Gear>; }; template <typename T> struct push_back_container<LogEvents, T> { struct Visitor { LogEvents& _log; template <typename U> void operator()(U const& ev) const { _log.add(ev); } using result_type = void; }; template <typename... U> static bool call(LogEvents& log, boost::variant<U...> const& attribute) { boost::apply_visitor(Visitor{log}, attribute); return true; } }; } } } namespace QiParsers { template <typename It, typename Attribute> struct BaseEventParser : qi::grammar<It, Attribute()> { BaseEventParser(std::string const& event_type) : BaseEventParser::base_type(start) { using namespace qi; auto date_time = copy( repeat(4)[digit] >> '-' >> repeat(3)[alpha] >> '-' >> repeat(2)[digit] >> ' ' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> '.' >> +digit); start = '[' >> raw[date_time] >> "] - " >> double_ >> " s" >> " => Driver: " >> int_ >> " - Speed: " >> double_ >> " - Road: " >> raw[+graph] >> " - Km: " >> double_ >> " - " >> lit(event_type) >> ": " >> int_ >> (eol|eoi); } private: qi::rule<It, Attribute()> start; }; } LogEvents parse_spirit(It b, It e) { QiParsers::BaseEventParser<It, ::Sequence> sequence("SEQUENCE"); QiParsers::BaseEventParser<It, ::Clutch> clutch("CLUTCH"); QiParsers::BaseEventParser<It, ::Gear> gear("GEAR"); LogEvents events; assert(parse(b, e, *boost::spirit::repository::qi::seek[sequence|clutch|gear], events)); return events; } static char input[] = /* see question */; static const size_t len = strlen(input); int main() { auto events = parse_spirit(input, input+len); std::cout << "Events: " << events.sequence.size() << " sequence, " << events.clutch.size() << " clutch, " << events.gear.size() << " gear events\n"; using boost::fusion::operator<<; for (auto& s : events.sequence) { std::cout << "SEQUENCE: " << s << "\n"; } for (auto& c : events.clutch) { std::cout << "CLUTCH: " << c << "\n"; } for (auto& g : events.gear) { std::cout << "GEAR: " << g << "\n"; } }#include <boost/fusion/adapted/struct.hpp> #include <boost/spirit/include/qi.hpp> #include <boost/spirit/repository/include/qi_seek.hpp> #include <boost/utility/string_view.hpp> #include <cstring> // strlen using It = char const*; namespace MyEvents { struct BaseEvent { int driver; int sequence; double time; double vel; double km; boost::string_view date; boost::string_view road; }; struct Sequence : BaseEvent{}; struct Clutch : BaseEvent{}; struct Gear : BaseEvent{}; using LogEvent = boost::variant<Sequence, Clutch, Gear>; using LogEvents = std::vector<LogEvent>; } BOOST_FUSION_ADAPT_STRUCT(MyEvents::Sequence, date, time, driver, vel, road, km, sequence) BOOST_FUSION_ADAPT_STRUCT(MyEvents::Clutch, date, time, driver, vel, road, km, sequence) BOOST_FUSION_ADAPT_STRUCT(MyEvents::Gear, date, time, driver, vel, road, km, sequence) namespace qi = boost::spirit::qi; namespace boost { namespace spirit { namespace traits { template <typename It> struct assign_to_attribute_from_iterators<boost::string_view, It, void> { static inline void call(It f, It l, boost::string_view& attr) { attr = boost::string_view { &*f, size_t(std::distance(f,l)) }; } }; } } } namespace QiParsers { template <typename It, typename Attribute> struct BaseEventParser : qi::grammar<It, Attribute()> { BaseEventParser(std::string const& event_type) : BaseEventParser::base_type(start) { using namespace qi; auto date_time = copy( repeat(4)[digit] >> '-' >> repeat(3)[alpha] >> '-' >> repeat(2)[digit] >> ' ' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> '.' >> +digit); start = '[' >> raw[date_time] >> "] - " >> double_ >> " s" >> " => Driver: " >> int_ >> " - Speed: " >> double_ >> " - Road: " >> raw[+graph] >> " - Km: " >> double_ >> " - " >> lit(event_type) >> ": " >> int_ >> (eol|eoi); } private: qi::rule<It, Attribute()> start; }; template <typename It> struct LogParser : qi::grammar<It, MyEvents::LogEvents()> { LogParser() : LogParser::base_type(start) { using namespace qi; using boost::spirit::repository::qi::seek; event = sequence | clutch | gear ; // TODO add types start = *seek[event]; } private: qi::rule<It, MyEvents::LogEvents()> start; qi::rule<It, MyEvents::LogEvent()> event; BaseEventParser<It, MyEvents::Sequence> sequence{"SEQUENCE"}; BaseEventParser<It, MyEvents::Clutch> clutch{"CLUTCH"}; BaseEventParser<It, MyEvents::Gear> gear{"GEAR"}; }; } MyEvents::LogEvents parse_spirit(It b, It e) { static QiParsers::LogParser<It> const parser {}; MyEvents::LogEvents events; parse(b, e, parser, events); return events; } static char input[] = /* see question */; static const size_t len = strlen(input); namespace MyEvents { // for debug/demo using boost::fusion::operator<<; static inline char const* kind(Sequence const&) { return "SEQUENCE"; } static inline char const* kind(Clutch const&) { return "CLUTCH"; } static inline char const* kind(Gear const&) { return "GEAR"; } struct KindVisitor : boost::static_visitor<char const*> { template <typename T> char const* operator()(T const& ev) const { return kind(ev); } }; static inline char const* kind(LogEvent const& ev) { return boost::apply_visitor(KindVisitor{}, ev); } } int main() { auto events = parse_spirit(input, input+len); std::cout << "Parsed: " << events.size() << " events\n"; for (auto& e : events) std::cout << kind(e) << ": " << e << "\n"; }#include <boost/fusion/adapted/struct.hpp> #include <boost/spirit/include/qi.hpp> #include <boost/spirit/repository/include/qi_seek.hpp> #include <boost/utility/string_view.hpp> #include <cstring> // strlen using It = char const*; namespace MyEvents { enum class Kind { Sequence, Clutch, Gear, Slope, Other }; struct CommonFields { boost::string_view date; double duration; }; struct BaseEvent { CommonFields common; int driver; int event_id; double vel; double km; boost::string_view road; Kind kind; }; struct OtherEvent { CommonFields common; std::string message; }; using LogEvent = boost::variant<BaseEvent, OtherEvent>; using LogEvents = std::vector<LogEvent>; } BOOST_FUSION_ADAPT_STRUCT(MyEvents::CommonFields, date, duration) BOOST_FUSION_ADAPT_STRUCT(MyEvents::BaseEvent, common, driver, vel, road, km, kind, event_id) BOOST_FUSION_ADAPT_STRUCT(MyEvents::OtherEvent, common, message) namespace qi = boost::spirit::qi; namespace boost { namespace spirit { namespace traits { template <typename It> struct assign_to_attribute_from_iterators<boost::string_view, It, void> { static inline void call(It f, It l, boost::string_view& attr) { attr = boost::string_view { &*f, size_t(std::distance(f,l)) }; } }; } } } namespace QiParsers { template <typename It> struct LogParser : qi::grammar<It, MyEvents::LogEvents()> { using Kind = MyEvents::Kind; LogParser() : LogParser::base_type(start) { using namespace qi; kind.add ("SEQUENCE", Kind::Sequence) ("CLUTCH", Kind::Clutch) ("GEAR", Kind::Gear) ("SLOPE", Kind::Slope) ; common_fields = '[' >> raw[ repeat(4)[digit] >> '-' >> repeat(3)[alpha] >> '-' >> repeat(2)[digit] >> ' ' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> '.' >> +digit ] >> "]" >> " - " >> double_ >> " s"; base_event = common_fields >> " => Driver: " >> int_ >> " - Speed: " >> double_ >> " - Road: " >> raw[+graph] >> " - Km: " >> double_ >> " - " >> kind >> ": " >> int_; other_event = common_fields >> " => " >> *~char_("\r\n"); event = (base_event | other_event) >> (eol|eoi); start = *boost::spirit::repository::qi::seek[event]; } private: qi::rule<It, MyEvents::LogEvents()> start; qi::rule<It, MyEvents::LogEvent()> event; qi::rule<It, MyEvents::CommonFields()> common_fields; qi::rule<It, MyEvents::BaseEvent()> base_event; qi::rule<It, MyEvents::OtherEvent()> other_event; qi::symbols<char, MyEvents::Kind> kind; }; } MyEvents::LogEvents parse_spirit(It b, It e) { static QiParsers::LogParser<It> const parser {}; MyEvents::LogEvents events; parse(b, e, parser, events); return events; } static char input[] = /* see question */; static const size_t len = strlen(input); namespace MyEvents { // for debug/demo using boost::fusion::operator<<; static inline Kind getKind(BaseEvent const& be) { return be.kind; } static inline Kind getKind(OtherEvent const&) { return Kind::Other; } struct KindVisitor : boost::static_visitor<Kind> { template <typename T> Kind operator()(T const& ev) const { return getKind(ev); } }; static inline Kind getKind(LogEvent const& ev) { return boost::apply_visitor(KindVisitor{}, ev); } static inline std::ostream& operator<<(std::ostream& os, Kind k) { switch(k) { case Kind::Sequence: return os << "SEQUENCE"; case Kind::Clutch: return os << "CLUTCH"; case Kind::Gear: return os << "GEAR"; case Kind::Slope: return os << "SLOPE"; case Kind::Other: return os << "(Other)"; } return os; } } int main() { auto events = parse_spirit(input, input+len); std::cout << "Parsed: " << events.size() << " events\n"; for (auto& e : events) std::cout << getKind(e) << ": " << e << "\n"; }Parsed: 37 events SLOPE: ((2018-Mar-13 13:13:59.580482 0.2) 0 0 A-11 90 SLOPE 0) GEAR: ((2018-Mar-13 13:14:01.170203 1.79) 0 0 A-11 90 GEAR 0) GEAR: ((2018-Mar-13 13:14:01.170203 1.79) 0 0.1 A-11 90 GEAR 1) SEQUENCE: ((2018-Mar-13 13:14:01.819966 2.44) 0 0.1 A-11 90 SEQUENCE 1) CLUTCH: ((2018-Mar-13 13:14:01.819966 2.44) 0 0.2 A-11 90 CLUTCH 1) (Other): ((2018-Mar-13 13:14:01.819966 2.54) Backup to regestry) [...]#include <boost/multi_index_container.hpp> #include <boost/multi_index/ordered_index.hpp> #include <boost/multi_index/composite_key.hpp> #include <boost/multi_index/global_fun.hpp> namespace Indexing { namespace bmi = boost::multi_index; using MyEvents::LogEvent; double getDuration(LogEvent const& ev) { return getCommon(ev).duration; } using Table = bmi::multi_index_container< std::reference_wrapper<LogEvent const>, //LogEvent, bmi::indexed_by< bmi::ordered_non_unique< bmi::tag<struct primary>, bmi::composite_key< LogEvent, bmi::global_fun<LogEvent const&, MyEvents::Kind, MyEvents::getKind>, bmi::global_fun<LogEvent const&, int, MyEvents::getEventId> > >, bmi::ordered_non_unique< bmi::tag<struct duration>, bmi::global_fun<LogEvent const&, double, getDuration> > > >; }Indexing::Table idx(events.begin(), events.end()); /* * // To print all events, grouped by by kind and event id: * for (MyEvents::LogEvent const& e : idx) * std::cout << getKind(e) << ": " << e << "\n"; * * // Ordered by duration: * for (MyEvents::LogEvent const& e : idx.get<Indexing::duration>()) * std::cout << getKind(e) << ": " << e << "\n"; */ std::cout << "\nAll GEAR events ordered by event id:\n"; for (MyEvents::LogEvent const& e : make_iterator_range(idx.equal_range(make_tuple(Kind::Gear)))) std::cout << getKind(e) << ": " << e << "\n"; std::cout << "\nOnly the SLOPE events with id 10:\n"; for (MyEvents::LogEvent const& e : make_iterator_range(idx.equal_range(make_tuple(Kind::Slope, 10)))) std::cout << getKind(e) << ": " << e << "\n"; std::cout << "\nEvents with durations in [2s..3s):\n"; auto& by_dur = idx.get<Indexing::duration>(); for (MyEvents::LogEvent const& e : make_iterator_range(by_dur.lower_bound(2), by_dur.upper_bound(3))) std::cout << getKind(e) << ": " << e << "\n";#include <boost/fusion/adapted/struct.hpp> #include <boost/spirit/include/qi.hpp> #include <boost/spirit/repository/include/qi_seek.hpp> #include <boost/utility/string_view.hpp> #include <cstring> // strlen using It = char const*; namespace MyEvents { enum class Kind { Sequence, Clutch, Gear, Slope, Other }; struct CommonFields { boost::string_view date; double duration; }; struct BaseEvent { CommonFields common; int driver; int event_id; double vel; double km; boost::string_view road; Kind kind; }; struct OtherEvent { CommonFields common; std::string message; }; using LogEvent = boost::variant<BaseEvent, OtherEvent>; using LogEvents = std::vector<LogEvent>; } BOOST_FUSION_ADAPT_STRUCT(MyEvents::CommonFields, date, duration) BOOST_FUSION_ADAPT_STRUCT(MyEvents::BaseEvent, common, driver, vel, road, km, kind, event_id) BOOST_FUSION_ADAPT_STRUCT(MyEvents::OtherEvent, common, message) namespace qi = boost::spirit::qi; namespace boost { namespace spirit { namespace traits { template <typename It> struct assign_to_attribute_from_iterators<boost::string_view, It, void> { static inline void call(It f, It l, boost::string_view& attr) { attr = boost::string_view { &*f, size_t(std::distance(f,l)) }; } }; } } } namespace QiParsers { template <typename It> struct LogParser : qi::grammar<It, MyEvents::LogEvents()> { using Kind = MyEvents::Kind; LogParser() : LogParser::base_type(start) { using namespace qi; kind.add ("SEQUENCE", Kind::Sequence) ("CLUTCH", Kind::Clutch) ("GEAR", Kind::Gear) ("SLOPE", Kind::Slope) ; common_fields = '[' >> raw[ repeat(4)[digit] >> '-' >> repeat(3)[alpha] >> '-' >> repeat(2)[digit] >> ' ' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> ':' >> repeat(2)[digit] >> '.' >> +digit ] >> "]" >> " - " >> double_ >> " s"; base_event = common_fields >> " => Driver: " >> int_ >> " - Speed: " >> double_ >> " - Road: " >> raw[+graph] >> " - Km: " >> double_ >> " - " >> kind >> ": " >> int_; other_event = common_fields >> " => " >> *~char_("\r\n"); event = (base_event | other_event) >> (eol|eoi); start = *boost::spirit::repository::qi::seek[event]; } private: qi::rule<It, MyEvents::LogEvents()> start; qi::rule<It, MyEvents::LogEvent()> event; qi::rule<It, MyEvents::CommonFields()> common_fields; qi::rule<It, MyEvents::BaseEvent()> base_event; qi::rule<It, MyEvents::OtherEvent()> other_event; qi::symbols<char, MyEvents::Kind> kind; }; } MyEvents::LogEvents parse_spirit(It b, It e) { static QiParsers::LogParser<It> const parser {}; MyEvents::LogEvents events; parse(b, e, parser, events); return events; } static char input[] = /* see question */; static const size_t len = strlen(input); namespace MyEvents { // for debug/demo using boost::fusion::operator<<; static inline CommonFields const& getCommon(BaseEvent const& be) { return be.common; } static inline CommonFields const& getCommon(OtherEvent const& oe) { return oe.common; } static inline Kind getKind(BaseEvent const& be) { return be.kind; } static inline Kind getKind(OtherEvent const&) { return Kind::Other; } static inline int getEventId(BaseEvent const& be) { return be.event_id; } static inline int getEventId(OtherEvent const&) { return 0; } #define IMPL_DISPATCH(name, T) \ struct name##Visitor : boost::static_visitor<T> { \ template <typename E> T operator()(E const &ev) const { return name(ev); } \ }; \ static inline T name(LogEvent const &ev) { return boost::apply_visitor(name##Visitor{}, ev); } IMPL_DISPATCH(getCommon, CommonFields const&) IMPL_DISPATCH(getKind, Kind) IMPL_DISPATCH(getEventId, int) static inline std::ostream& operator<<(std::ostream& os, Kind k) { switch(k) { case Kind::Sequence: return os << "SEQUENCE"; case Kind::Clutch: return os << "CLUTCH"; case Kind::Gear: return os << "GEAR"; case Kind::Slope: return os << "SLOPE"; case Kind::Other: return os << "(Other)"; } return os; } } #include <boost/multi_index_container.hpp> #include <boost/multi_index/ordered_index.hpp> #include <boost/multi_index/composite_key.hpp> #include <boost/multi_index/global_fun.hpp> namespace Indexing { namespace bmi = boost::multi_index; using MyEvents::LogEvent; double getDuration(LogEvent const& ev) { return getCommon(ev).duration; } using Table = bmi::multi_index_container< std::reference_wrapper<LogEvent const>, //LogEvent, bmi::indexed_by< bmi::ordered_non_unique< bmi::tag<struct primary>, bmi::composite_key< LogEvent, bmi::global_fun<LogEvent const&, MyEvents::Kind, MyEvents::getKind>, bmi::global_fun<LogEvent const&, int, MyEvents::getEventId> > >, bmi::ordered_non_unique< bmi::tag<struct duration>, bmi::global_fun<LogEvent const&, double, getDuration> > > >; } using boost::make_iterator_range; using boost::make_tuple; int main() { using MyEvents::LogEvent; using MyEvents::Kind; auto events = parse_spirit(input, input+len); std::cout << "Parsed: " << events.size() << " events\n"; Indexing::Table idx(events.begin(), events.end()); /* * // To print all events, grouped by by kind and event id: * for (MyEvents::LogEvent const& e : idx) * std::cout << getKind(e) << ": " << e << "\n"; * * // Ordered by duration: * for (MyEvents::LogEvent const& e : idx.get<Indexing::duration>()) * std::cout << getKind(e) << ": " << e << "\n"; */ std::cout << "\nAll GEAR events ordered by event id:\n"; for (MyEvents::LogEvent const& e : make_iterator_range(idx.equal_range(make_tuple(Kind::Gear)))) std::cout << getKind(e) << ": " << e << "\n"; std::cout << "\nOnly the SLOPE events with id 10:\n"; for (MyEvents::LogEvent const& e : make_iterator_range(idx.equal_range(make_tuple(Kind::Slope, 10)))) std::cout << getKind(e) << ": " << e << "\n"; std::cout << "\nEvents with durations in [2s..3s):\n"; auto& by_dur = idx.get<Indexing::duration>(); for (MyEvents::LogEvent const& e : make_iterator_range(by_dur.lower_bound(2), by_dur.upper_bound(3))) std::cout << getKind(e) << ": " << e << "\n"; }Parsed: 37 events All GEAR events ordered by event id: GEAR: ((2018-Mar-13 13:14:01.170203 1.79) 0 0 A-11 90 GEAR 0) GEAR: ((2018-Mar-13 13:14:01.170203 1.79) 0 0 A-11 90 GEAR 0) GEAR: ((2018-Mar-13 13:14:01.170203 1.79) 0 0.1 A-11 90 GEAR 1) GEAR: ((2018-Mar-13 13:14:01.170203 1.79) 0 0.1 A-11 90 GEAR 1) GEAR: ((2018-Mar-13 13:14:03.250451 3.87) 0 1.2 B-302 90.2 GEAR 2) GEAR: ((2018-Mar-13 13:14:03.250451 3.87) 0 1.2 B-302 90.2 GEAR 2) GEAR: ((2018-Mar-13 13:14:04.510025 5.13) 0 4.9 B-302 91.1 GEAR 3) Only the SLOPE events with id 10: SLOPE: ((2018-Mar-13 13:14:04.300160 4.92) 0 4.2 B-302 90.9 SLOPE 10) SLOPE: ((2018-Mar-13 13:14:04.300160 4.92) 0 4.2 B-302 90.9 SLOPE 10) Events with durations in [2s..3s): SEQUENCE: ((2018-Mar-13 13:14:01.819966 2.44) 0 0.1 A-11 90 SEQUENCE 1) CLUTCH: ((2018-Mar-13 13:14:01.819966 2.44) 0 0.2 A-11 90 CLUTCH 1) SEQUENCE: ((2018-Mar-13 13:14:01.819966 2.44) 0 0.1 A-11 90 SEQUENCE 1) CLUTCH: ((2018-Mar-13 13:14:01.819966 2.44) 0 0.2 A-11 90 CLUTCH 1) (Other): ((2018-Mar-13 13:14:01.819966 2.54) Backup to regestry) (Other): ((2018-Mar-13 13:14:01.819966 2.54) Backup to regestry)