Regex 如何根据Spark中的列值匹配多个正则表达式模式？_Regex_Scala_Apache Spark_Dataframe_Pattern Matching

Regex 如何根据Spark中的列值匹配多个正则表达式模式？

regex scala apache-spark dataframe

Regex 如何根据Spark中的列值匹配多个正则表达式模式？,regex,scala,apache-spark,dataframe,pattern-matching,Regex,Scala,Apache Spark,Dataframe,Pattern Matching,我有一个专栏： val originalSqlLikePatternMap = Map("item (%) is blacklisted%" -> "BLACK_LIST", "%Testing%" -> "TESTING", "%purchase count % is too low %" -> "TOO_LOW_PURCHASE_COUNT") val javaPatternMap = originalSqlLikePatternMap.map(v =&g

我有一个专栏：

val originalSqlLikePatternMap = Map("item (%) is blacklisted%" -> "BLACK_LIST",
      "%Testing%" -> "TESTING",
  "%purchase count % is too low %" -> "TOO_LOW_PURCHASE_COUNT")

val javaPatternMap = originalSqlLikePatternMap.map(v => v._1.replaceAll("%", ".*") -> v._2)

val df = Seq(
  "Testing(2,4, (4,6,7) foo, Foo purchase count 1 is too low", 
  "Foo purchase count (12, 4) is too low ", "#!@", "item (mejwnw) is blacklisted",
   "item (1) is blacklisted, #!@" 
).toDF("raw_type")

val converter = (value: String) => javaPatternMap.find(v => value.matches(v._1)).map(_._2).getOrElse("Unknown")
val converterUDF = udf(converter)

val result = df.withColumn("updatedType", converterUDF($"raw_type"))

但它给出了：

+---------------------------------------------------------+----------------------+
|raw_type                                                 |updatedType           |
+---------------------------------------------------------+----------------------+
|Testing(2,4, (4,6,7) foo, Foo purchase count 1 is too low|TESTING               |
|Foo purchase count (12, 4) is too low                    |TOO_LOW_PURCHASE_COUNT|
|#!@                                                      |Unknown               |
|item (mejwnw) is blacklisted                             |BLACK_LIST            |
|item (1) is blacklisted, #!@                             |BLACK_LIST            |
+---------------------------------------------------------+----------------------+

但是我想要“测试（2,4，（4,6,7）foo，foo购买计数1太低”来给出2个值“测试，购买计数太低”，如下所示：

 +---------------------------------------------------------+--------------------------------+
|raw_type                                                 |updatedType                     |
+---------------------------------------------------------+--------------------------------+
|Testing(2,4, (4,6,7) foo, Foo purchase count 1 is too low|TESTING, TOO_LOW_PURCHASE_COUNT |
|Foo purchase count (12, 4) is too low                    |TOO_LOW_PURCHASE_COUNT          |
|#!@                                                      |Unknown                         |
|item (mejwnw) is blacklisted                             |BLACK_LIST                      |
|item (1) is blacklisted, #!@                             |BLACK_LIST, Unkown              |
+---------------------------------------------------------+--------------------------------+

有人能告诉我我做错了什么吗？

好的。这里有几件事

关于

find

，您需要对照每个正则表达式检查每个

行

，以获得所需的输出，因此find不是正确的选择

迭代器产生的满足谓词的第一个值，如果任何

注意正则表达式，low之后留下了一个空格，这就是它不匹配的原因。请您重新考虑是否也将

替换为

%purchase count%过低%

因此，随着更改，您的代码将类似于

 val originalSqlLikePatternMap = Map(
      "item (%) is blacklisted%" -> "BLACK_LIST",
      "%Testing%" -> "TESTING",
      "%purchase count % is too low%" -> "TOO_LOW_PURCHASE_COUNT")

    val javaPatternMap = originalSqlLikePatternMap.map(v => v._1.replaceAll("%", ".*").r -> v._2)

    val df = Seq(
      "Testing(2,4, (4,6,7) foo, Foo purchase count 1 is too low",
      "Foo purchase count (12, 4) is too low ", "#!@", "item (mejwnw) is blacklisted",
      "item (1) is blacklisted, #!@"
    ).toDF("raw_type")

    val converter = (value: String) => {
      val res = javaPatternMap.map(v => {
        v._1.findFirstIn(value) match {
          case Some(_) => v._2
          case None => ""
        }
      })
        .filter(_.nonEmpty).mkString(", ")

      if (res.isEmpty) "Unknown" else res
    }

    val converterUDF = udf(converter)

    val result = df.withColumn("updatedType", converterUDF($"raw_type"))

    result.show(false)

产出

+---------------------------------------------------------+-------------------------------+
|raw_type                                                 |updatedType                    |
+---------------------------------------------------------+-------------------------------+
|Testing(2,4, (4,6,7) foo, Foo purchase count 1 is too low|TESTING, TOO_LOW_PURCHASE_COUNT|
|Foo purchase count (12, 4) is too low                    |TOO_LOW_PURCHASE_COUNT         |
|#!@                                                      |Unknown                        |
|item (mejwnw) is blacklisted                             |BLACK_LIST                     |
|item (1) is blacklisted, #!@                             |BLACK_LIST                     |
+---------------------------------------------------------+-------------------------------+

希望这有帮助！