将XML消息从Zerocode发布到Kafka主题

我正在尝试使用以下工具对流媒体平台（Kafka/spark）进行集成测试

零代码（发布XML消息）-->使用Apache AVRO序列化-->卡夫卡主题-->火花作业（处理）--->卡夫卡主题-->使用AVRO反序列化-->零代码使用者测试（断言消息内容）

当我试图用零代码将消息放在Kafka主题上时，零代码测试用例在XML消息的开头引入了特殊的字符，这导致在spark中解析XML消息时出错

当我使用字符串序列化程序和“hello world”时，它可以工作

我需要让它与ApacheAvro和XML消息一起工作

{
  "scenario Name": "Produce a message to Kafka topic - vanilla",
  "steps": [
    {
      "name": "produce_step",
      "URL": "Kafka-topic: my-topic ",
      "operation": "produce",
      "request": {
        "record Type" : "J SON",
        "records": [
          {
            "key":"123456789",
            "value": "Schema Test"
          }
        ]
      },
      "assertions": {
        "record Metadata" : "$NOT.NULL",
        "status" : "OK"
      }
    }
  ]
}

当零代码测试用例将消息放在Kafka主题上时，它不应该在消息的开头引入任何特殊字符

---

特殊字符位于xml的开头

如果我理解正确，您正在尝试将XML转换为Avro

根据我读到的关于zerocode的内容，这是不可能的，至少它可以将UTF-8字符串转换为Avro UTF-8字符串，您可以在zerocode之外完全模拟相同的事情

启动模式注册表和Kafka

创建一个主题

kafka-topics --create --topic sample --zookeeper localhost:2181 --partitions=1 --replication-factor=1

使用kafka avro console producer为主题生成“XML”字符串

kafka-avro-console-producer --broker-list localhost:29092 --topic sample --property value.schema='{"type":"string"}'
"<?xml version=\"1.0\"?>"

使用

kafka console consumer

（使用StringDeserializer）检查消息我们注意到它有一个“特殊字符”

kafka控制台使用者--引导服务器localhost:9092--主题示例--从头开始
*

使用

kafka avro console consumer

（使用KafkaAvroDeserializer），我们可以看到值被引号包围（因为表示是JSON字符串），但“特殊字符”不见了

kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic sample --from-beginning
"<?xml version=\"1.0\"?>"

kafka avro控制台使用者--引导服务器localhost:9092--主题示例--从头开始
""

---

因此，我没有你的Spark代码来知道你的错误发生在哪里，但对Avro数据一无所知（也就是说，它无法解析“特殊字符”，实际上是注册表中的Avro ID），也不知道如何使用模式注册表，所以我的建议是忘记Avro，因为你实际上只处理字符串

您只需将“记录类型”更新为

RAW

，而不是

JSON

。它应该像预期的那样工作

“记录类型”：“原始”

e、 g

您希望将下面的XML消息生成到卡夫卡主题，然后使用卡夫卡主题中的XML并验证XML是否相同

AFA
英镑

下面是完整的测试用例供您参考。它有“生产”和“消费”两个步骤

{
“scenarioName”：“生成并使用XML作为原始消息”，
“步骤”：[
{
“名称”：“加载卡夫卡”，
“url”：“卡夫卡主题：演示-raw1”，
“操作”：“加载”，
“请求”：{
“记录类型”：“原始”，
“记录”：[
{
“键”：“${RANDOM.NUMBER}”，
“值”：“\n\n\n AFA\n GBP\n\n\n”
}
]
},
“断言”：{
“状态”：“确定”
}
},
{
“名称”：“onload_kafka”，
“url”：“卡夫卡主题：演示-raw1”，
“操作”：“卸载”，
“请求”：{
“消费者本地配置”：{
“记录类型”：“原始”，
“commitSync”：对，
“showRecordsConsumed”：正确，
“MaxNoFreetryPollsorTimeouts”：3
}
},
“断言”：{
“大小”：1，
“记录”：[
{
“key”：“$NOT.NULL”，
“值”：“\n\n\n AFA\n GBP\n\n\n”
}
]
}
}
]
}

消费记录后，Zerocode测试运行控制台日志如下，即消费的XML与生成的XML完全相同

org.jsmart.zerocode.core.kafka.receive.KafkaReceiver - Consumer chosen recordType: RAW
2020-05-19 12:32:28,986 [main] INFO  
org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
Record Key - 6784799632078391362 , 
Record value - <?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ConversionRate xmlns="http://www.webserviceX.NET/">
      <FromCurrency>AFA</FromCurrency>
      <ToCurrency>GBP</ToCurrency>
    </ConversionRate>
  </soap:Body>
</soap:Envelope>, 

Record partition - 0, Record offset - 1

org.jsmart.zerocode.core.kafka.receive.KafkaReceiver-消费者选择的记录类型：原始
2020-05-19 12:32:28986[主要]信息
org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper-
记录键-6784799632078391362，
记录值-
AFA
英镑
, 
记录分区-0，记录偏移量-1

您可以在下面看到完整的日志：

2020-05-19 12:32:25,358 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
-------------------------- BDD: Scenario:consume as RAW message -------------------------

2020-05-19 12:32:25,461 [main] INFO  org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
---------------------------------------------------------
kafka.bootstrap.servers - localhost:9092
---------------------------------------------------------
returned default type 'RAW'.
2020-05-19 12:32:25,677 [main] INFO  org.jsmart.zerocode.core.kafka.send.KafkaSender - Sending record number: 0

2020-05-19 12:32:25,678 [main] INFO  org.jsmart.zerocode.core.kafka.send.KafkaSender - Synchronous Producer sending record - ProducerRecord(topic=demo-raw1, partition=null, headers=RecordHeaders(headers = [], isReadOnly = false), key=6784799632078391362, value=<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ConversionRate xmlns="http://www.webserviceX.NET/">
      <FromCurrency>AFA</FromCurrency>
      <ToCurrency>GBP</ToCurrency>
    </ConversionRate>
  </soap:Body>
</soap:Envelope>, timestamp=null)


2020-05-19 12:32:25,873 [main] INFO  org.jsmart.zerocode.core.engine.validators.ZeroCodeValidatorImpl - Comparing results via LENIENT matchers
2020-05-19 12:32:25,878 [main] INFO  org.jsmart.zerocode.core.runner.StepNotificationHandler - 
***Step PASSED - Scenario:consume as RAW message -> load_kafka
2020-05-19 12:32:25,882 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
--------- TEST-STEP-CORRELATION-ID: a8ebe4c6-c1fb-434c-9c00-b2cbdf2f7fe6 ---------
*requestTimeStamp:2020-05-19T12:32:25.455
step:load_kafka
url:kafka-topic:demo-raw1
method:load
request:
{
  "record Type" : "RAW",
  "records" : [ {
    "key" : "6784799632078391362",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
  } ]
} 
--------- TEST-STEP-CORRELATION-ID: a8ebe4c6-c1fb-434c-9c00-b2cbdf2f7fe6 ---------
Response:
{
  "status" : "Ok",
  "recordMetadata" : {
    "offset" : 1,
    "timestamp" : 1589887945831,
    "serializedKeySize" : 19,
    "serializedValueSize" : 407,
    "topicPartition" : {
      "hash" : 749906548,
      "partition" : 0,
      "topic" : "demo-raw1"
    }
  }
}
*responseTimeStamp:2020-05-19T12:32:25.872 
*Response delay:417.0 milli-secs 
---------> Expected Response: <----------
{
  "status" : "Ok"
} 

-done-

2020-05-19 12:32:25,884 [main] INFO  org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
---------------------------------------------------------
kafka.bootstrap.servers - localhost:9092
---------------------------------------------------------

***Step PASSED - Scenario:consume as RAW message -> onload_kafka
2020-05-19 12:32:33,025 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
--------- TEST-STEP-CORRELATION-ID: 53092b7e-ae92-4631-ad79-0602280650b9 ---------
*requestTimeStamp:2020-05-19T12:32:25.884
step:onload_kafka
url:kafka-topic:demo-raw1
method:unload
request:
{
  "consumerLocalConfigs" : {
    "recordType" : "RAW",
    "commitSync" : true,
    "showRecordsConsumed" : true,
    "maxNoOfRetryPollsOrTimeouts" : 3
  }
} 
--------- TEST-STEP-CORRELATION-ID: 53092b7e-ae92-4631-ad79-0602280650b9 ---------
Response:
{
  "records" : [ {
    "topic" : "demo-raw1",
    "partition" : 0,
    "offset" : 1,
    "timestamp" : 1589887945831,
    "timestampType" : "CREATE_TIME",
    "serializedKeySize" : 19,
    "serializedValueSize" : 407,
    "key" : "6784799632078391362",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>",
    "leaderEpoch" : {
      "value" : 0
    }
  } ],
  "size" : 1
}
*responseTimeStamp:2020-05-19T12:32:33.020 
*Response delay:7136.0 milli-secs 
---------> Expected Response: <----------
{
  "size" : 1,
  "records" : [ {
    "key" : "$NOT.NULL",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
  } ]
} 

-done-


2020-05-19 12:32:33,233 [main] INFO  org.jsmart.zerocode.core.domain.builders.ExtentReportsFactory - Where were the tests fired? Ans: OS:Mac OS X, Architecture:x86_64, Java:1.8.0_91, Vendor:Oracle Corporation

Process finished with exit code 0

2020-05-19 12:32:25358[main]INFO org.jsmart.zerocode.core.runner.zerocodemultistepscenariorunnerimpl-
--------------------------BDD：场景：作为原始消息使用-------------------------
2020-05-19 12:32:25461[main]INFO org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper-
---------------------------------------------------------
kafka.bootstrap.servers-本地主机：9092
---------------------------------------------------------
返回默认类型“RAW”。
2020-05-19 12:32:25677[main]INFO org.jsmart.zerocode.core.kafka.send.KafkaSender-发送记录编号：0
2020-05-19 12:32:25678[main]INFO org.jsmart.zerocode.core.kafka.send.KafkaSender-同步生产者发送记录-生产者记录（topic=demo-raw1，partition=null，headers=RecordHeaders（headers=[]，isReadOnly=false），key=6784799632078391362，value=
AFA
英镑
，timestamp=null）
2020-05-19 12:32:25873[main]INFO org.jsmart.zerocode.core.engine.validators.ZeroCodeValidatorImpl-通过宽松匹配器比较结果
2020-05-19 12:32:25878[main]INFO org.jsmart.zerocode.core.runner.StepNotificationHandler-
***已通过步骤-场景：作为原始消息消费->加载卡夫卡
2020-05-19 12:32:25882[main]INFO org.jsmart.zerocode.core.runner.zerocodemultistepscenariorunnerepl-
---------测试步骤相关ID:a8
{
    "scenarioName": "Produce and Consume XML as RAW message",
    "steps": [
        {
            "name": "load_kafka",
            "url": "kafka-topic:demo-raw1",
            "operation": "load",
            "request": {
                "record Type" : "RAW",
                "records": [
                    {
                        "key": "${RANDOM.NUMBER}",
                        "value": "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
                    }
                ]
            },
            "assertions": {
                "status": "Ok"
            }
        },
        {
            "name": "onload_kafka",
            "url": "kafka-topic:demo-raw1",
            "operation": "unload",
            "request": {
                "consumerLocalConfigs": {
                    "recordType": "RAW",
                    "commitSync": true,
                    "showRecordsConsumed": true,
                    "maxNoOfRetryPollsOrTimeouts": 3
                }
            },
            "assertions": {
                "size": 1,
                "records": [
                    {
                        "key" : "$NOT.NULL",
                        "value": "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
                    }
                ]
            }
        }
    ]
}


org.jsmart.zerocode.core.kafka.receive.KafkaReceiver - Consumer chosen recordType: RAW
2020-05-19 12:32:28,986 [main] INFO  
org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
Record Key - 6784799632078391362 , 
Record value - <?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ConversionRate xmlns="http://www.webserviceX.NET/">
      <FromCurrency>AFA</FromCurrency>
      <ToCurrency>GBP</ToCurrency>
    </ConversionRate>
  </soap:Body>
</soap:Envelope>, 

Record partition - 0, Record offset - 1


2020-05-19 12:32:25,358 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
-------------------------- BDD: Scenario:consume as RAW message -------------------------

2020-05-19 12:32:25,461 [main] INFO  org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
---------------------------------------------------------
kafka.bootstrap.servers - localhost:9092
---------------------------------------------------------
returned default type 'RAW'.
2020-05-19 12:32:25,677 [main] INFO  org.jsmart.zerocode.core.kafka.send.KafkaSender - Sending record number: 0

2020-05-19 12:32:25,678 [main] INFO  org.jsmart.zerocode.core.kafka.send.KafkaSender - Synchronous Producer sending record - ProducerRecord(topic=demo-raw1, partition=null, headers=RecordHeaders(headers = [], isReadOnly = false), key=6784799632078391362, value=<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ConversionRate xmlns="http://www.webserviceX.NET/">
      <FromCurrency>AFA</FromCurrency>
      <ToCurrency>GBP</ToCurrency>
    </ConversionRate>
  </soap:Body>
</soap:Envelope>, timestamp=null)


2020-05-19 12:32:25,873 [main] INFO  org.jsmart.zerocode.core.engine.validators.ZeroCodeValidatorImpl - Comparing results via LENIENT matchers
2020-05-19 12:32:25,878 [main] INFO  org.jsmart.zerocode.core.runner.StepNotificationHandler - 
***Step PASSED - Scenario:consume as RAW message -> load_kafka
2020-05-19 12:32:25,882 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
--------- TEST-STEP-CORRELATION-ID: a8ebe4c6-c1fb-434c-9c00-b2cbdf2f7fe6 ---------
*requestTimeStamp:2020-05-19T12:32:25.455
step:load_kafka
url:kafka-topic:demo-raw1
method:load
request:
{
  "record Type" : "RAW",
  "records" : [ {
    "key" : "6784799632078391362",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
  } ]
} 
--------- TEST-STEP-CORRELATION-ID: a8ebe4c6-c1fb-434c-9c00-b2cbdf2f7fe6 ---------
Response:
{
  "status" : "Ok",
  "recordMetadata" : {
    "offset" : 1,
    "timestamp" : 1589887945831,
    "serializedKeySize" : 19,
    "serializedValueSize" : 407,
    "topicPartition" : {
      "hash" : 749906548,
      "partition" : 0,
      "topic" : "demo-raw1"
    }
  }
}
*responseTimeStamp:2020-05-19T12:32:25.872 
*Response delay:417.0 milli-secs 
---------> Expected Response: <----------
{
  "status" : "Ok"
} 

-done-

2020-05-19 12:32:25,884 [main] INFO  org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
---------------------------------------------------------
kafka.bootstrap.servers - localhost:9092
---------------------------------------------------------

***Step PASSED - Scenario:consume as RAW message -> onload_kafka
2020-05-19 12:32:33,025 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
--------- TEST-STEP-CORRELATION-ID: 53092b7e-ae92-4631-ad79-0602280650b9 ---------
*requestTimeStamp:2020-05-19T12:32:25.884
step:onload_kafka
url:kafka-topic:demo-raw1
method:unload
request:
{
  "consumerLocalConfigs" : {
    "recordType" : "RAW",
    "commitSync" : true,
    "showRecordsConsumed" : true,
    "maxNoOfRetryPollsOrTimeouts" : 3
  }
} 
--------- TEST-STEP-CORRELATION-ID: 53092b7e-ae92-4631-ad79-0602280650b9 ---------
Response:
{
  "records" : [ {
    "topic" : "demo-raw1",
    "partition" : 0,
    "offset" : 1,
    "timestamp" : 1589887945831,
    "timestampType" : "CREATE_TIME",
    "serializedKeySize" : 19,
    "serializedValueSize" : 407,
    "key" : "6784799632078391362",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>",
    "leaderEpoch" : {
      "value" : 0
    }
  } ],
  "size" : 1
}
*responseTimeStamp:2020-05-19T12:32:33.020 
*Response delay:7136.0 milli-secs 
---------> Expected Response: <----------
{
  "size" : 1,
  "records" : [ {
    "key" : "$NOT.NULL",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
  } ]
} 

-done-


2020-05-19 12:32:33,233 [main] INFO  org.jsmart.zerocode.core.domain.builders.ExtentReportsFactory - Where were the tests fired? Ans: OS:Mac OS X, Architecture:x86_64, Java:1.8.0_91, Vendor:Oracle Corporation

Process finished with exit code 0