Scala 如何创建一个源来将度量从Spark导出到另一个接收器（Prometheus）？

我正试图从我用Scala编写的spark应用程序中创建一个度量源，以便将数据导出到另一个系统，最好是Prometheus。根据这个网站从我需要创建一个来源，扩展源特性。但是，源特征是私有[spark]特征源，我的源代码无法将其可视化。当我创建这个类时，我得到一个错误

符号源无法从此位置访问

package org.sense.spark.util

import org.apache.spark.metrics.source.Source
import com.codahale.metrics.{Counter, Histogram, MetricRegistry}

class MetricSource extends Source {
  override val sourceName: String = "MySource"

  override val metricRegistry: MetricRegistry = new MetricRegistry

  val FOO: Histogram = metricRegistry.histogram(MetricRegistry.name("fooHistory"))
  val FOO_COUNTER: Counter = metricRegistry.counter(MetricRegistry.name("fooCounter"))
}

我如何创建源以将数据导出到普罗米修斯？我想从
combineByKey
转换中的UDF导出监控值。这些值将是聚合的延迟和转换的吞吐量

这是我的
build.sbt
文件，以防需要检查我正在使用的库

name := "explore-spark"

version := "0.2"

scalaVersion := "2.12.3"

val sparkVersion = "3.0.0"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "com.twitter" %% "algebird-core" % "0.13.7",
  "joda-time" % "joda-time" % "2.5",
  "org.fusesource.mqtt-client" % "mqtt-client" % "1.16"
)

mainClass in(Compile, packageBin) := Some("org.sense.spark.app.App")
mainClass in assembly := Some("org.sense.spark.app.App")

assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
assemblyJarName in assembly := s"${name.value}_${scalaBinaryVersion.value}-fat_${version.value}.jar"

您需要将扩展源代码的类与源代码放在同一个包中

package org.apache.spark.metrics.source

import com.codahale.metrics.{Counter, Histogram, MetricRegistry}

class MetricsSource extends Source {
  override val sourceName: String = "MySource"

  override val metricRegistry: MetricRegistry = new MetricRegistry

    val FOO: Histogram = metricRegistry.histogram(MetricRegistry.name("fooHistory"))

  val FOO_COUNTER: Counter = metricRegistry.counter(MetricRegistry.name("fooCounter"))
}

您需要将扩展源代码的类与源代码放在同一个包中

package org.apache.spark.metrics.source

import com.codahale.metrics.{Counter, Histogram, MetricRegistry}

class MetricsSource extends Source {
  override val sourceName: String = "MySource"

  override val metricRegistry: MetricRegistry = new MetricRegistry

    val FOO: Histogram = metricRegistry.histogram(MetricRegistry.name("fooHistory"))

  val FOO_COUNTER: Counter = metricRegistry.counter(MetricRegistry.name("fooCounter"))
}

如果您需要使用prometheus接收器而不是控制台接收器，则可以使用为spark prometheus接收器编写的第三方库。这通过pushgateway实现-
如果您需要使用prometheus接收器而不是控制台接收器，则可以使用为spark prometheus接收器编写的第三方库。这项工作通过pushgateway-