C# 并行for循环和httpclient死锁并引发异常
我不明白为什么我的parallels.for循环在一个httpclient调用中不断爆炸。 该代码适用于大约10-15个请求,然后将挂起很长时间,并在System.AggregateException上出错 我尝试了多种不同的变体,包括webclient。 请考虑以下事项:C# 并行for循环和httpclient死锁并引发异常,c#,asynchronous,task-parallel-library,dotnet-httpclient,parallel.foreach,C#,Asynchronous,Task Parallel Library,Dotnet Httpclient,Parallel.foreach,我不明白为什么我的parallels.for循环在一个httpclient调用中不断爆炸。 该代码适用于大约10-15个请求,然后将挂起很长时间,并在System.AggregateException上出错 我尝试了多种不同的变体,包括webclient。 请考虑以下事项: class Program { static void Main(string[] args) { Parallel.For(0, 250, i => {
class Program
{
static void Main(string[] args)
{
Parallel.For(0, 250, i =>
{
var task = GetPages.CallHttp();
task.Wait();
var content = task.Result;
TestObject obj = content.ToTestObject(); //take the string and find some values in it.
Console.WriteLine(obj.H1Tag);
});
}
}
public static class GetPages
{
private static readonly HttpClient client = new HttpClient() {Timeout = new TimeSpan(0,5,0)};
public static async Task<string> CallHttp()
{
client.DefaultRequestHeaders.UserAgent.ParseAdd("customAgent/1.0");
string astr = await client.GetStringAsync("the url I am testing").ConfigureAwait(false);
return astr;
}
}
public static class StringExtensions
{
private static readonly object objLock = new object();
public static TestObject ToTestObject(this string content)
{
lock (objLock)
{
var obj = new TestObject();
// creates a bunch of properties inspecting the html string
var result = new HtmlExtractions(content);
obj.PageTitle = result.PageTitle;
obj.H1Tag = result.H1Tag;
...
return obj;
}
}
}
public class HtmlExtractions
{
internal HtmlDocument doc;
public HtmlExtractions(string contentToRead)
{
doc = new HtmlDocument();
doc.LoadHtml(contentToRead);
}
public string PageTitle => doc.DocumentNode.Descendants("title").FirstOrDefault()?.InnerHtml.Replace("&", "&").Trim();
...
}
异步任务(释放线程)和并行任务。For
(强制使用多个线程)往往不能很好地混合使用.Wait()
和.Result
在用于异步任务时都是阻塞调用和邀请死锁。尝试使用Task重新编写Main
方法。当所有并避免阻塞调用时:
var tasks = Enumerable.Range(0, 250).Select(i => GetPages.CallHttp());
var contents = await Task.WhenAll(tasks);
foreach (var content in contents)
{
TestObject obj = content.ToTestObject(); //take the string and find some values in it.
Console.WriteLine(obj.H1Tag);
}
查看System.aggregateeexception
的InnerExceptions
属性。此外,不需要使用lock
语句。据我所知,没有并发读写操作。内部异常在文章的底部。我应该说得更清楚些。一项任务被取消。InnerException:Id=50,Status=cancelled,Method=“{null}”,Result=“{Not Not computed}”关于锁,我同意,但此时正在黑暗中射箭。@PeterBons知道可能是什么问题吗?我在没有锁的情况下尝试了相同的代码,但遇到了相同的问题代码>执行250次,在我的测试中,这会生成一个异常。您应该重构它,使它只执行一次,而不是CallHttp
的一部分。我想这个集合不是线程安全的。没有任何类型的真正运气。在底部更新了有问题的代码。谢谢你的建议。首先创建集合,然后进行迭代是有意义的。
static async void RunPagesAsync()
{
Console.WriteLine("getting contents");
var tasks = Enumerable.Range(0, 5).Select(i => GetPages.CallHttp());
var contents = await Task.WhenAll(tasks);
Console.WriteLine("Got Contents..continuing");
foreach (var content in contents)
{
TestObject obj = content.ToTestObject(); //take the string and find some values in it.
Console.WriteLine(obj.H1Tag);
}
Console.WriteLine("completed");
}
static void Main(string[] args)
{
//Task.Run(() => RunPagesAsync()); //doesn't work.
// RunPagesAsync(); //just hangs after what looks like 2 itterations
// var tasks = Enumerable.Range(0, 5).Select(i => GetPages.CallHttp());
// var contents = await Task.WhenAll(tasks); //won't compile under synch Main dues to await
// foreach (var content in contents)
// {
// TestObject obj = content.ToTestObject(); //take the string and find some values in it.
// Console.WriteLine(obj.H1Tag);
// }
var tasks1 = Enumerable.Range(0, 5).Select(i => GetPages.CallHttp());
var contents1 = Task.WhenAll(tasks1);
contents1.Wait();
foreach (var content in contents1.Result)
{
TestObject obj = content.ToTestObject(); //take the string and find some values in it.
Console.WriteLine(obj.H1Tag);
}
var tasks = Enumerable.Range(0, 250).Select(i => GetPages.CallHttp());
var contents = await Task.WhenAll(tasks);
foreach (var content in contents)
{
TestObject obj = content.ToTestObject(); //take the string and find some values in it.
Console.WriteLine(obj.H1Tag);
}