C# 通过字符串数组的多个webrequest
我有2000多个url调用要做,下面的代码几乎需要2分钟才能完成。有人能帮我加快这个过程吗C# 通过字符串数组的多个webrequest,c#,arrays,webrequest,webresponse,C#,Arrays,Webrequest,Webresponse,我有2000多个url调用要做,下面的代码几乎需要2分钟才能完成。有人能帮我加快这个过程吗 private void button4_Click(object sender, EventArgs e) { WebRequest req; WebResponse res; string[] lines = File.ReadAllLines(@"c:\data\temp.txt"); for (int i = 0; i <
private void button4_Click(object sender, EventArgs e)
{
WebRequest req;
WebResponse res;
string[] lines = File.ReadAllLines(@"c:\data\temp.txt");
for (int i = 0; i < lines.Count(); i++)
{
req = WebRequest.Create(lines[i]);
res = req.GetResponse();
StreamReader rd = new StreamReader(res.GetResponseStream(), Encoding.ASCII);
rd.Close();
res.Close();
textBox1.Text += ".";
}
}
非常感谢您无法加快速度,因为瓶颈是您的互联网连接。但是,您可以做一些事情: 1不要LINQ计数行,它是一个数组,它的大小是已知的微优化,你永远不会注意到这个变化 2使用释放一次性对象与速度无关,更好的错误处理:如果代码出现问题,您将使用GC释放资源 3使它们平行。这将稍微加快速度: 请注意: MaxDegreeOfParallelism设置最大并发请求数。多个活动并发连接不会无限期地加快速度,甚至可能会减慢速度。一些试验将帮助您将此值设置为合理值 没有任何错误检查,但网络可能暂时出错,但在短暂延迟后,它们可能会按预期工作。我建议您也阅读和了解I/O操作 为了使其成为一个更完整的示例,您的点击平衡处理程序将是:
private void button4_Click(object sender, EventArgs e)
{
var options = new ParallelOptions { MaxDegreeOfParallelism = 4 };
Parallel.ForEach(ReadUrlList(@"c:\data\temp.txt"), options, ProcessUrl);
}
处理每个URL和读取URL列表的实际代码:
private static string[] ReadUrlList(string path)
{
return File.ReadAllLines(@"c:\data\temp.txt");
}
private void ProcessUrl(string url)
{
ProcessResponse(response =>
{
using (var reader = new StreamReader(response.GetResponseStream(), Encoding.ASCII))
{
// Do your stuff
// We're working on separate threads, to access UI we
// have to dispatch the call to UI thread. Note that
// code will be executed asynchronously then local
// objects may have been disposed!
BeginInvoke(new MethodInvoker(delegate
{
textBox1.Text += ".";
}));
}
});
}
使用此帮助器方法隐藏网络操作的尝试/等待模式:
private static void ProcessResponse(string url, Action<WebResponse> action)
{
for (int i=1; i <= NumberOfRetries; ++i)
{
try
{
var request = WebRequest.Create(line);
using (var response = request.GetResponse())
{
action(response);
}
break;
}
catch (Exception e)
{
if (i == NumberOfRetries)
throw;
Thread.Sleep(DelayOnRetry);
}
}
}
private const int NumberOfRetries = 3;
private const int DelayOnRetry = 1000;
因为您没有指定框架版本,所以我假设您至少使用了4.5 您可以使用轻松地同时执行多个调用。ActionBlock在单个线程中执行其action方法,可以同时执行多个执行 您可以使用以下内容:
var options=new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 10
}
var block=new ActionBlock<string>(url=>
{
using(var req = WebRequest.Create(url))
using(var res = req.GetResponse())
{
//Process the response here
}
});
string[] lines = File.ReadAllLines(@"c:\data\temp.txt");
foreach(var line in lines)
{
block.Post(line);
}
block.Complete();
await block.Completion;
处理请求和响应非常重要。除非处置响应,否则与服务器的连接将保持活动状态。NET强制每个域2个并发请求ie URL限制,因此孤立响应可能会导致延迟,直到垃圾收集器运行并收集它们。虽然您可以覆盖该限制,但最好始终处理响应。我建议您使用Microsoft的反应式框架来实现此目的。NuGet Rx Main、Rx WinForms/Rx WPF 下面是代码的样子:
private void button4_Click(object sender, EventArgs e)
{
var query =
from line in File.ReadAllLines(@"c:\data\temp.txt").ToObservable()
from result in Observable.Defer(() =>
{
var req = WebRequest.Create(line);
return
Observable.Using(
() => req.GetResponse(),
res => Observable.Using(
() => new StreamReader(res.GetResponseStream(), Encoding.ASCII),
st => Observable.Start(() => st.ReadToEnd())));
})
select new { line, result };
query
.ObserveOn(textBox1)
.Subscribe(x => textBox1.Text += ".");
}
我假设您正在尝试从流中读取字符串
这段代码很好地处理了所有中间对象。它还正确地对请求执行多线程处理,并将结果封送到UI线程并更新文本框文本
此代码的一个稍微干净的版本是:
private void button4_Click(object sender, EventArgs e)
{
var query =
from line in File.ReadAllLines(@"c:\data\temp.txt").ToObservable()
from result in Observable.Using(
() => new WebClient(),
wc => Observable.Start(() => wc.DownloadString(new Uri(line))))
select new { line, result };
query
.ObserveOn(textBox1)
.Subscribe(x => textBox1.Text += ".");
}
它使用WebClient进行下载。它仍然可以根据需要使用多线程。购买更快的internet。使用线程。您可以同时调用更多的电话。我也会考虑禁用代理查找,除非需要关闭NAGLE。
private void button4_Click(object sender, EventArgs e)
{
var query =
from line in File.ReadAllLines(@"c:\data\temp.txt").ToObservable()
from result in Observable.Defer(() =>
{
var req = WebRequest.Create(line);
return
Observable.Using(
() => req.GetResponse(),
res => Observable.Using(
() => new StreamReader(res.GetResponseStream(), Encoding.ASCII),
st => Observable.Start(() => st.ReadToEnd())));
})
select new { line, result };
query
.ObserveOn(textBox1)
.Subscribe(x => textBox1.Text += ".");
}
private void button4_Click(object sender, EventArgs e)
{
var query =
from line in File.ReadAllLines(@"c:\data\temp.txt").ToObservable()
from result in Observable.Using(
() => new WebClient(),
wc => Observable.Start(() => wc.DownloadString(new Uri(line))))
select new { line, result };
query
.ObserveOn(textBox1)
.Subscribe(x => textBox1.Text += ".");
}