Fatal编程技术网
  • java/
  • Java 如何以scala方式解析基于行的文本文件（.mht）？

Java 如何以scala方式解析基于行的文本文件（.mht）？

java parsing scala

Java 如何以scala方式解析基于行的文本文件（.mht）？,java,parsing,scala,line,Java,Parsing,Scala,Line,我想用scala解析一个.mht文件，但我发现我的代码和Java一模一样 以下是mht文件示例： From: <Save by Tencent MsgMgr> Subject: Tencent IM Message MIME-Version: 1.0 Content-Type:multipart/related; charset="utf-8" type="text/html"; boundary="----=_NextPart_20CAFF23_6090_4

我想用scala解析一个.mht文件，但我发现我的代码和Java一模一样

以下是
mht
文件示例：

From: <Save by Tencent MsgMgr>
Subject: Tencent IM Message
MIME-Version: 1.0
Content-Type:multipart/related;
    charset="utf-8"
    type="text/html";
    boundary="----=_NextPart_20CAFF23_6090_43fc_8C0A.EE179EE81D19"

------=_NextPart_20CAFF23_6090_43fc_8C0A.EE179EE81D19
Content-Type: text/html
Content-Transfer-Encoding:7bit

<html xmlns="http://www.w3.org/1999/xhtml"><head></head>...</html>

------=_NextPart_20CAFF23_6090_43fc_8C0A.EE179EE81D19
Content-Type:image/jpeg
Content-Transfer-Encoding:base64
Content-Location:{64172C34-99E7-40f6-A933-3DDCF670ACBA}.dat

/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMU
FRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQU
FBQUFBQUFBT/wAARCAJwA7sDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUF
BAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVW
V1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi

------=_NextPart_20CAFF23_6090_43fc_8C0A.EE179EE81D19
Content-Type:image/jpeg
Content-Transfer-Encoding:base64
Content-Location:{64172C34-99E7-40f6-A933-3DDCF670ACBA}.dat

/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMU
FRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQU
FBQUFBQUFBT/wAARCAJwA7sDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUF
BAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVW
V1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi

------=_NextPart_20CAFF23_6090_43fc_8C0A.EE179EE81D19
Content-Type:image/jpeg
Content-Transfer-Encoding:base64
Content-Location:{64172C34-99E7-40f6-A933-3DDCF670ACBA}.dat

/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMU
FRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQU
FBQUFBQUFBT/wAARCAJwA7sDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUF
BAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVW
V1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi

------=_NextPart_20CAFF23_6090_43fc_8C0A.EE179EE81D19
第一部分是关于这个文件的一些信息，可以忽略。下面是4个块，第一个是
html
文件，其他的是
jpg
带有
base64
编码文本的图像

如果我使用Java，代码如下：

BufferedReader reader = new BufferedReader(new FileInputStream(new File("test.mht")))
String line = null;

String boundary = null;

// for a block
String contentType = null;
String encoding = null;
String location = null;
List<String> data = null;

while((line=reader.readLine())!=null) {
    // first, get the boundary
    if(boundary==null) {
        if(line.trim().startsWith("boundary=\"") {
             boundary = substringBetween(line, "\"", "\"");
        }
        continue;
    }

    if(line.equals("--"+boundary) { // new block
        if(contentType!=null) {
           // save data to a file
        }
        encoding=null;
        contentType=null;
        location = null;
        data = new ArrayList<String>();
    } else {
        if(id==null || contentType==null || location ==null) {
            if(line.trim().startsWith("Content-Type:") { /* get content type */ }
            // else check encoding
            // else check location
        } else {
            data.add(line);
        }
    }
}

BufferedReader reader=new BufferedReader（新文件输入流（新文件（“test.mht”））
字符串行=null；
字符串边界=null；
//一个街区
字符串contentType=null；
字符串编码=空；
字符串位置=空；
列表数据=null；
而（（line=reader.readLine（））！=null）{
//首先，获取边界
如果（边界==null）{
if（line.trim（）.startsWith（“boundary=\”）{
边界=子字符串之间（行“\”，“\”）；
}
继续；
}
if（line.equals（“--”+边界）{//新块
if（contentType！=null）{
//将数据保存到文件中
}
编码=空；
contentType=null；
位置=空；
数据=新的ArrayList（）；
}否则{
if（id==null | | contentType==null | | location==null）{
if（line.trim（）.startsWith（“内容类型：”）{/*获取内容类型*/}
//else检查编码
//其他检查位置
}否则{
数据。添加（行）；
}
}
}
我试图使用scala重写代码，但我发现代码的结构几乎相同，只是我使用了scala语法而不是Java

有没有一种scala方法可以做同样的工作

PS：我不想把完整的文件加载到内存中，因为文件很大。相反，我想逐行读取和解析它

感谢您的帮助！

这可能是一个非常简单的状态机用例

import collection.mutable.ListBuffer
case class Part(contentType:Option[String], encoding:Option[String], location:Option[String], data:ListBuffer[String])

var boundary: String = null 

val Boundary = """.*boundary="(.*)"""".r
var state = 0
val IN_PART = 1
val IN_DATA = 2

var _contentType:Option[String] = None
var _encoding:Option[String] = None
var _location:Option[String] = None
var _data = new ListBuffer[String]()

Source.fromFile("test.mht").getLines.foreach{
  case Boundary(b) => boundary = b
  case `boundary` => 
    _contentType = None
    _encoding = None
    _location = None
    _data = new ListBuffer[String]()    
    state = IN_PART
  case "" => state match {
    case IN_PART => state = IN_DATA
    case IN_DATA => 
        var currentPart = Part(_contentType, _encoding, _location, _data)
        /* deal with current Part as allData.last */
    case _ =>
  }
  case line => state match {
    case IN_DATA => _data.append(line)            
    case IN_PART => line.split(":") match {
      case Array("Content-Type", t) => _contentType = Some(t)
      case Array("Content-Transfer-Encoding", e) => _encoding = Some(e)
      case Array("Content-Location", l) => _location = Some(l)
      case _ =>
    }
  }
}
我将解释如何使用解析器组合器以标准方式构建通用解决方案。提供的另一种解决方案速度要快得多，但是，一旦您了解了如何做到这一点，您就可以轻松地将其应用于其他任务

首先，您显示的是一封电子邮件。此类邮件的格式是在一堆RFC中定义的。定义标题和正文的基础知识，尽管它输入了关于标题的相当详细的信息，但没有提到正文。1522谈到MIME，而本身就是RFC 1341和1342的修订版。还有许多其他RFC ab把话题说清楚

有趣的是，他们提供了关于这些东西的语法，所以你可以编写解析器来正确分解它。让我们从RFC822的简化版本开始，几乎忽略所有已知字段及其格式，只需将所有内容都放在一个映射中。我这样做是因为语法相当长，这里只有几行可以与RFC中的进行比较

在Scala解析器组合器上，每个规则都由
~
分隔（在RFC中，只是用空格分隔），我有时使用
来丢弃其中一个不感兴趣的部分。此外，我还使用
^
将解析的内容转换为要使用的数据结构

import scala.util.parsing.combinator._

/** Object companion to RFC822, containing the Message class,
 *  and extending the trait so that it can be used as a parser
 */
object RFC822 extends RFC822 {
  case class Message(header: Map[String, String], text: String)
}

/**
 *  Parsers `message` according to RFC-822 (http://www.w3.org/Protocols/rfc822/),
 *  but without breaking up the contents for each field, 
 *  nor identifying particular fields.
 *
 *  Also, introduces "header" to convert all fields into a map.
 */
class RFC822 extends RegexParsers {
  import RFC822.Message

  override def skipWhitespace = false

  def message = (header <~ CRLF) ~ text ^^ {
    case hd ~ txt => Message(hd, txt)
  }

  // this isn't part of the RFC, but we use it to generate a map
  def header = field.* ^^ { _.toMap }

  def field = (fieldName <~ ":") ~ fieldBody <~ CRLF ^^ { case name ~ body => name -> body }
  def fieldName = """[^:\P{Graph}]+""".r

  // Recursive definition needs a type
  // Also, I use .+ on LWSPChar because it's specified for the lexer,
  // which we are not using
  def fieldBody: Parser[String] = fieldBodyContents ~ (CRLF ~> LWSPChar.+ ~> fieldBody).? ^^ { 
    case a ~ Some(b) => a + " " + b // reintroduces a single LWSPChar
    case a ~ None    => a
  }
  def fieldBodyContents = ".*".r

  def CRLF = """\n""".r  // this needs to be the regex \n pattern
  def LWSPChar = " " | "\t"  // these do not need to be regex

  def text = "(?s).*".r // (?s) makes . match newlines
}
我们现在可以使用它来解析文本

object Parser {
  def apply(email: String): Option[(Map[String, String], List[String])] = {
    import RFC822._

    parseAll (message, email) match {
      case Success(result, _) =>
        if (result.header get "Content-Type" nonEmpty) Some(getParts(result))
        else Some(result.header -> List(result.text))
      case _ => None
    }
  }

  def getParts(message: RFC822.Message): (Map[String, String], List[String]) = {
    import ContentType._

    parseAll (content, "Content-Type: " + message.header("Content-Type")) match {
      case Success(Content("multipart", _, parameters), _) =>
        // The ^.* part eats starting characters; it doesn't seem to be
        // as spec'ed, but the sample has two extra dashes at the start
        // of the line
        val parts = message.text split ("^.*?\\Q" + parameters("boundary") + "\\E")
        val bodies = flatMap this.apply flatMap (_._2)
        message.header -> bodies.toList
      case _ => message.header -> List(message.text)
    }
  }
}
然后您可以像
解析器（电子邮件）
一样使用它

再说一次，我不是建议你用这个方法来解决你目前的问题！但是学习这个方法可能会对你将来有所帮助。

你太好了，谢谢你！这个方法需要我花上几天的时间去理解，但我真的觉得它很重要，应该被掌握。对不起，我不明白
case`boundary`=>的意思是行的内容与变量
boundary
的内容匹配。如果不使用反引号，每行都将匹配，并且行的内容将设置为新的
boundary
变量。如果有第二行与boundary regexp匹配，则会出问题（原始代码检查边界是否为null以避免此情况）

object ContentType extends ContentType {
  case class Content(`type`: String, subtype: String, parameter: Map[String, String])
}

class ContentType extends RegexParsers {
  import ContentType.Content

  // case-insensitive matching of type and subtype
  def content =   ("Content-Type" ~> ":" ~> `type` <~ "/") ~ subtype ~ parameters ^^ {
    case t ~ s ~ p => Content(t, s, p)
  }

  // use this to generate a map
  // *** SEMI-COLON IS NOT OPTIONAL ***
  // I'm making it optional because the example is missing one
  def parameters = (";".? ~> parameter).* ^^ (_.toMap)

  // All values case-insensitive
  def `type` = ( "(?i)application".r | "(?i)audio".r
               | "(?i)image".r       | "(?i)message".r
               | "(?i)multipart".r   | "(?i)text".r
               | "(?i)video".r       | extensionToken
               )

  def extensionToken =  xToken | ianaToken
  def ianaToken = failure("IANA token not implemented")
  def xToken = """(?i)x-(?!\s)""".r ~ token ^^ { case a ~ b => a + b }

  def subtype = token

  def parameter = (attribute <~ "=") ~ value ^^ { case a ~ b => a -> b }
  def attribute = token // case-insensitive
  def value = token | quotedString

  def token: Parser[String] =  not(tspecials) ~> """\p{Graph}""".r ~ token.? ^^ {
    case a ~ Some(b) => a + b
    case a ~ None    => a
  }

  // Must be in quoted-string,
  // to use within parameter values
  def tspecials =  ( "(" | ")" | "<" | ">" | "@"
                   | "," | ";" | ":" | "\\" | "\""
                   | "/" | "[" | "]" | "?" | "="
                   )

  // These are part of RFC822
  def qtext = """[^\\"\n]""".r
  def quotedPair =  """\\.""".r
  def quotedString = "\"" ~> (qtext|quotedPair).* <~ "\"" ^^ { _.mkString }
}



object Parser {
  def apply(email: String): Option[(Map[String, String], List[String])] = {
    import RFC822._

    parseAll (message, email) match {
      case Success(result, _) =>
        if (result.header get "Content-Type" nonEmpty) Some(getParts(result))
        else Some(result.header -> List(result.text))
      case _ => None
    }
  }

  def getParts(message: RFC822.Message): (Map[String, String], List[String]) = {
    import ContentType._

    parseAll (content, "Content-Type: " + message.header("Content-Type")) match {
      case Success(Content("multipart", _, parameters), _) =>
        // The ^.* part eats starting characters; it doesn't seem to be
        // as spec'ed, but the sample has two extra dashes at the start
        // of the line
        val parts = message.text split ("^.*?\\Q" + parameters("boundary") + "\\E")
        val bodies = flatMap this.apply flatMap (_._2)
        message.header -> bodies.toList
      case _ => message.header -> List(message.text)
    }
  }
}