Testing SpecFlow-重试失败的测试
是否有一种方法可以实现Testing SpecFlow-重试失败的测试,testing,hook,specflow,scenarios,Testing,Hook,Specflow,Scenarios,是否有一种方法可以实现AfterScenariohook,以便在失败时重新运行当前测试 大概是这样的: [AfterScenario("retry")] public void Retry() { if (ScenarioContext.Current.TestError != null) { // ? } } 注意:我的项目中的测试组合在有序测试中,并通过MsTest执行Specflow场景的目的是断言系统按预期运行 如果某个暂时的问题导致测试失败
AfterScenario
hook,以便在失败时重新运行当前测试
大概是这样的:
[AfterScenario("retry")]
public void Retry()
{
if (ScenarioContext.Current.TestError != null)
{
// ?
}
}
注意:我的项目中的测试组合在有序测试中,并通过MsTest执行Specflow场景的目的是断言系统按预期运行 如果某个暂时的问题导致测试失败,那么让测试重新运行并“希望最好”并不能解决问题!偶尔测试失败不应该是预期的行为。测试应该在每次执行时给出一致的结果 可以找到一篇关于什么是好测试的好帖子,该答案还指出测试应该是: 可重复:每次测试应产生相同的结果。。每一个 时间测试不应依赖于不可控的参数 在这种情况下,测试失败是完全正确的。现在,您应该调查测试偶尔失败的确切原因 大多数情况下,测试失败是由于时间问题,例如页面加载期间不存在元素。在这个场景中,给定一个一致的测试环境(即相同的测试数据库、相同的测试浏览器、相同的网络设置),那么您将能够再次编写可重复的测试。查看关于使用WebDriverWait等待预定时间来测试预期DOM元素是否存在的答案 这个插件太棒了。我让它与nunit一起工作,他的例子是使用MS Test 它将允许您执行以下操作:
@retry:2
Scenario: Tag on scenario is preferred
Then scenario should be run 3 times
我希望能够重试失败的测试,但仍然在测试结果中报告为失败。这将使我能够轻松地确定代码工作的场景,但由于网络延迟等原因,这些场景也容易出现零星问题。这些故障的优先级与由于代码更改而出现的新故障的优先级不同 我使用MsTest实现了这一点,因为您可以创建一个继承自TestMethodAttribute的类 首先,我将此部分添加到csproj文件的底部,以便在生成*.feature.cs文件后但在实际生成之前调用自定义powershell脚本:
<Target Name="OverrideTestMethodAttribute" BeforeTargets="PrepareForBuild">
<Message Text="Calling OverrideTestMethodAttribute.ps1" Importance="high" />
<Exec Command="powershell -Command "$(ProjectDir)OverrideTestMethodAttribute.ps1"" />
</Target>
以及执行实际重试的IntegrationTestMethodAttribute类:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace MyCompany.MyProduct
{
public class IntegrationTestMethodAttribute : TestMethodAttribute
{
public override TestResult[] Execute(ITestMethod testMethod)
{
TestResult[] testResults = null;
var failedAttempts = new List<TestResult>();
int maxAttempts = 5;
for (int i = 0; i < maxAttempts; i++)
{
testResults = base.Execute(testMethod);
Exception ex = testResults[0].TestFailureException;
if (ex == null)
{
break;
}
failedAttempts.AddRange(testResults);
}
if (failedAttempts.Any() && failedAttempts.Count != maxAttempts)
{
TestResult testResult = testResults[0];
var messages = new StringBuilder();
for (var i = 0; i < failedAttempts.Count; i++)
{
var result = failedAttempts[i];
messages.AppendLine("");
messages.AppendLine("");
messages.AppendLine("");
messages.AppendLine($"Failure #{i + 1}:");
messages.AppendLine(result.TestFailureException.ToString());
messages.AppendLine("");
messages.AppendLine(result.TestContextMessages);
}
testResult.Outcome = UnitTestOutcome.Error;
testResult.TestFailureException = new Exception($"Test failed {failedAttempts.Count} time(s), then succeeded");
testResult.TestContextMessages = messages.ToString();
testResult.LogError = "";
testResult.DebugTrace = "";
testResult.LogOutput = "";
}
return testResults;
}
}
}
使用系统;
使用System.Collections.Generic;
使用System.Linq;
使用系统文本;
使用Microsoft.VisualStudio.TestTools.UnitTesting;
名称空间MyCompany.MyProduct
{
公共类IntegrationTestMethodAttribute:TestMethodAttribute
{
公共重写TestResult[]执行(ITestMethod testMethod)
{
TestResult[]testResults=null;
var failedAttempts=新列表();
int=5;
对于(int i=0;i
首先让我说,我同意测试应该是稳定的,不应该重试。然而,我们并不是生活在一个理想的世界中,在一些非常特定的场景中,重试测试可能是一个有效的用例
我正在运行UI测试(对angular应用程序使用selenium),有时chromedriver会因为不清楚的原因而变得无响应。这种行为完全超出了我的控制范围,不存在有效的解决方案。我无法在SpecFlow步骤中重试此操作,因为我已经“给定”了登录到应用程序的步骤。当它在“何时”步骤中失败时,我还需要重新运行“给定”步骤。在这个场景中,我想关闭驱动程序,再次启动它,然后重新运行前面的所有步骤。最后,我为SpecFlow编写了一个定制的testrunner,可以从如下错误中恢复:
[AfterScenario("retry")]
public void Retry()
{
if (ScenarioContext.Current.TestError != null)
{
// ?
}
}
免责声明:这不是预期用途,它可能会在任何版本的SpecFlow中中断。如果你是一个测试纯粹主义者,不要读更多
首先,我们创建一个类,使创建自定义ITestRunner变得容易(将所有方法提供为虚拟方法,以便可以重写它们):
接下来,我们创建一个定制的testrunner,它会记住对场景的调用,并可以重新运行前面的步骤:
public class RetryTestRunner : OverrideableTestRunner
{
/// <summary>
/// Which exceptions to handle (default: all)
/// </summary>
public Predicate<Exception> HandleExceptionFilter { private get; set; } = _ => true;
/// <summary>
/// The action that is executed to recover
/// </summary>
public Action RecoverAction { private get; set; } = () => { };
/// <summary>
/// The maximum number of retries
/// </summary>
public int MaxRetries { private get; set; } = 10;
/// <summary>
/// The executed actions for this scenario, these need to be replayed in the case of an error
/// </summary>
private readonly List<(MethodInfo method, object[] args)> _previousSteps = new List<(MethodInfo method, object[] args)>();
/// <summary>
/// The number of the current try (to make sure we don't go over the specified limit)
/// </summary>
private int _currentTryNumber = 0;
public NonSuckingTestRunner(ITestExecutionEngine engine) : base(new TestRunner(engine))
{
}
public override void OnScenarioStart()
{
base.OnScenarioStart();
_previousSteps.Clear();
_currentTryNumber = 0;
}
public override void Given(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.Given(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
public override void But(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.But(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
public override void And(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.And(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
public override void Then(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.Then(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
public override void When(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.When(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
// Use this delegate combination to make a params call possible
// It is not possible to use a params argument and the CallerMemberName
// in one method, so we curry the method to make it possible. #functionalprogramming
public delegate void ParamsFunc(params object[] args);
private ParamsFunc Checker([CallerMemberName] string method = null)
{
return args =>
{
// Record the previous step
_previousSteps.Add((GetType().GetMethod(method), args));
// Determine if we should retry
if (ScenarioContext.ScenarioExecutionStatus != ScenarioExecutionStatus.TestError || !HandleExceptionFilter(ScenarioContext.TestError) || _currentTryNumber >= MaxRetries)
{
return;
}
// HACKY: Reset the test state to a non-error state
typeof(ScenarioContext).GetProperty(nameof(ScenarioContext.ScenarioExecutionStatus)).SetValue(ScenarioContext, ScenarioExecutionStatus.OK);
typeof(ScenarioContext).GetProperty(nameof(ScenarioContext.TestError)).SetValue(ScenarioContext, null);
// Trigger the recovery action
RecoverAction.Invoke();
// Retry the steps
_currentTryNumber++;
var stepsToPlay = _previousSteps.ToList();
_previousSteps.Clear();
stepsToPlay.ForEach(s => s.method.Invoke(this, s.args));
};
}
}
要在AfterScenario步骤中使用它,可以向testrunner添加一个RetryScenario()方法并调用该方法
作为最后一句话:当你无能为力时,将此作为最后的手段。运行不可靠的测试总比不运行任何测试好。第二次运行它成功的条件是什么?好问题@rene!我想我的整个想法都是死胎。无论谁否决了我,请解释一下你为什么这么做
public class RetryTestRunner : OverrideableTestRunner
{
/// <summary>
/// Which exceptions to handle (default: all)
/// </summary>
public Predicate<Exception> HandleExceptionFilter { private get; set; } = _ => true;
/// <summary>
/// The action that is executed to recover
/// </summary>
public Action RecoverAction { private get; set; } = () => { };
/// <summary>
/// The maximum number of retries
/// </summary>
public int MaxRetries { private get; set; } = 10;
/// <summary>
/// The executed actions for this scenario, these need to be replayed in the case of an error
/// </summary>
private readonly List<(MethodInfo method, object[] args)> _previousSteps = new List<(MethodInfo method, object[] args)>();
/// <summary>
/// The number of the current try (to make sure we don't go over the specified limit)
/// </summary>
private int _currentTryNumber = 0;
public NonSuckingTestRunner(ITestExecutionEngine engine) : base(new TestRunner(engine))
{
}
public override void OnScenarioStart()
{
base.OnScenarioStart();
_previousSteps.Clear();
_currentTryNumber = 0;
}
public override void Given(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.Given(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
public override void But(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.But(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
public override void And(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.And(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
public override void Then(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.Then(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
public override void When(string text, string multilineTextArg, Table tableArg, string keyword = null)
{
base.When(text, multilineTextArg, tableArg, keyword);
Checker()(text, multilineTextArg, tableArg, keyword);
}
// Use this delegate combination to make a params call possible
// It is not possible to use a params argument and the CallerMemberName
// in one method, so we curry the method to make it possible. #functionalprogramming
public delegate void ParamsFunc(params object[] args);
private ParamsFunc Checker([CallerMemberName] string method = null)
{
return args =>
{
// Record the previous step
_previousSteps.Add((GetType().GetMethod(method), args));
// Determine if we should retry
if (ScenarioContext.ScenarioExecutionStatus != ScenarioExecutionStatus.TestError || !HandleExceptionFilter(ScenarioContext.TestError) || _currentTryNumber >= MaxRetries)
{
return;
}
// HACKY: Reset the test state to a non-error state
typeof(ScenarioContext).GetProperty(nameof(ScenarioContext.ScenarioExecutionStatus)).SetValue(ScenarioContext, ScenarioExecutionStatus.OK);
typeof(ScenarioContext).GetProperty(nameof(ScenarioContext.TestError)).SetValue(ScenarioContext, null);
// Trigger the recovery action
RecoverAction.Invoke();
// Retry the steps
_currentTryNumber++;
var stepsToPlay = _previousSteps.ToList();
_previousSteps.Clear();
stepsToPlay.ForEach(s => s.method.Invoke(this, s.args));
};
}
}
/// <summary>
/// We need this because this is the only way to configure specflow before it starts
/// </summary>
[TestClass]
public class CustomDependencyProvider : DefaultDependencyProvider
{
[AssemblyInitialize]
public static void AssemblyInitialize(TestContext testContext)
{
// Override the dependency provider of specflow
ContainerBuilder.DefaultDependencyProvider = new CustomDependencyProvider();
TestRunnerManager.OnTestRunStart(typeof(CustomDependencyProvider).Assembly);
}
[AssemblyCleanup]
public static void AssemblyCleanup()
{
TestRunnerManager.OnTestRunEnd(typeof(CustomDependencyProvider).Assembly);
}
public override void RegisterTestThreadContainerDefaults(ObjectContainer testThreadContainer)
{
base.RegisterTestThreadContainerDefaults(testThreadContainer);
// Use our own testrunner
testThreadContainer.RegisterTypeAs<NonSuckingTestRunner, ITestRunner>();
}
}
<PropertyGroup>
<GenerateSpecFlowAssemblyHooksFile>false</GenerateSpecFlowAssemblyHooksFile>
</PropertyGroup>
[Binding]
public class TestInitialize
{
private readonly RetryTestRunner _testRunner;
public TestInitialize(ITestRunner testRunner)
{
_testRunner = testRunner as RetryTestRunner;
}
[BeforeScenario()]
public void TestInit()
{
_testRunner.RecoverAction = () =>
{
StopDriver();
StartDriver();
};
_testRunner.HandleExceptionFilter = ex => ex is WebDriverException;
}
}