C# 如何检测文本文件的字符编码？
我尝试检测文件中使用的字符编码 我尝试使用此代码来获得标准编码C# 如何检测文本文件的字符编码？,c#,encoding,character-encoding,byte-order-mark,C#,Encoding,Character Encoding,Byte Order Mark,我尝试检测文件中使用的字符编码 我尝试使用此代码来获得标准编码 public static Encoding GetFileEncoding(string srcFile) { // *** Use Default of Encoding.Default (Ansi CodePage) Encoding enc = Encoding.Default; // *** Detect byte order mark if any - otherwise a
public static Encoding GetFileEncoding(string srcFile)
{
// *** Use Default of Encoding.Default (Ansi CodePage)
Encoding enc = Encoding.Default;
// *** Detect byte order mark if any - otherwise assume default
byte[] buffer = new byte[5];
FileStream file = new FileStream(srcFile, FileMode.Open);
file.Read(buffer, 0, 5);
file.Close();
if (buffer[0] == 0xef && buffer[1] == 0xbb && buffer[2] == 0xbf)
enc = Encoding.UTF8;
else if (buffer[0] == 0xfe && buffer[1] == 0xff)
enc = Encoding.Unicode;
else if (buffer[0] == 0 && buffer[1] == 0 && buffer[2] == 0xfe && buffer[3] == 0xff)
enc = Encoding.UTF32;
else if (buffer[0] == 0x2b && buffer[1] == 0x2f && buffer[2] == 0x76)
enc = Encoding.UTF7;
else if (buffer[0] == 0xFE && buffer[1] == 0xFF)
// 1201 unicodeFFFE Unicode (Big-Endian)
enc = Encoding.GetEncoding(1201);
else if (buffer[0] == 0xFF && buffer[1] == 0xFE)
// 1200 utf-16 Unicode
enc = Encoding.GetEncoding(1200);
return enc;
}
是否有图表显示哪种编码与前五个字节相匹配？是的，这里有一个：。您应该阅读以下内容：使用
StreamReaderusing (var reader = new System.IO.StreamReader(path, true))
{
var currentEncoding = reader.CurrentEncoding;
}
以根据它切换逻辑。如果文件以字节60、118、56、46和49开头，则大小写不明确。它可以是UTF-8（无BOM）或任何单字节编码，如ASCII、ANSI、ISO-8859-1等。您不能依赖具有BOM的文件。UTF-8不需要它。非Unicode编码甚至没有BOM表。但是，还有其他方法可以检测编码 UTF-32 BOM为00 00 FE FF（对于BE）或FF FE 00 00（对于LE） 但UTF-32即使没有BOM也很容易检测。这是因为Unicode代码点范围被限制为U+10FFFF，因此UTF-32单元始终具有模式00{00-10}xx xx xx xx（对于BE）或xx xx{00-10}00（对于LE）。如果数据的长度是4的倍数，并且遵循其中一种模式，则可以安全地假定它是UTF-32。由于面向字节的编码中很少有00字节，误报几乎是不可能的 US-ASCII 没有BOM，但您不需要BOM。ASCII可以通过80-FF范围内缺少字节来轻松识别 UTF-8 BOM是EF BB BF。但你不能相信这一点。很多UTF-8文件都没有BOM表，特别是当它们源于非Windows系统时 但您可以安全地假设，如果文件验证为UTF-8，那么它就是UTF-8。假阳性很少见 具体来说，如果数据不是ASCII码，则2字节序列的误报率仅为3.9%（1920/49152）。对于一个7字节的序列，它小于1%。对于一个12字节的序列，它小于0.1%。对于一个24字节的序列，它少于百万分之一 UTF-16 BOM是FE FF（用于BE）或FF FE（用于LE）。请注意，UTF-16LE BOM位于UTF-32LE BOM的开头，因此请先检查UTF-32 如果您碰巧有一个主要由ISO-8859-1字符组成的文件，那么将该文件的一半字节设置为00也将是UTF-16的一个有力指标 否则，在没有BOM的情况下识别UTF-16的唯一可靠方法是寻找替代项对（D[8-B]xx D[C-F]xx），但非BMP字符很少使用，因此这种方法不实用 XML
如果您的文件以字节3C 3F 78 6D 6C开头（即ASCII字符“如果您想寻求一个“简单”的解决方案，您可能会发现我放在一起的这个类很有用： 它首先自动进行BOM检测，然后尝试区分没有BOM的Unicode编码和其他一些默认编码（通常是Windows-1252，在.Net中错误地标记为encoding.ASCII） 如上所述，涉及Ncharde或MLang的“更重”解决方案可能更合适，正如我在本课程的概述页面上所指出的，最好是尽可能提供与用户的某种形式的交互，因为根本不可能有100%的检测率 站点脱机时的代码段：
using System;
using System.Text;
using System.Text.RegularExpressions;
using System.IO;
namespace KlerksSoft
{
public static class TextFileEncodingDetector
{
/*
* Simple class to handle text file encoding woes (in a primarily English-speaking tech
* world).
*
* - This code is fully managed, no shady calls to MLang (the unmanaged codepage
* detection library originally developed for Internet Explorer).
*
* - This class does NOT try to detect arbitrary codepages/charsets, it really only
* aims to differentiate between some of the most common variants of Unicode
* encoding, and a "default" (western / ascii-based) encoding alternative provided
* by the caller.
*
* - As there is no "Reliable" way to distinguish between UTF-8 (without BOM) and
* Windows-1252 (in .Net, also incorrectly called "ASCII") encodings, we use a
* heuristic - so the more of the file we can sample the better the guess. If you
* are going to read the whole file into memory at some point, then best to pass
* in the whole byte byte array directly. Otherwise, decide how to trade off
* reliability against performance / memory usage.
*
* - The UTF-8 detection heuristic only works for western text, as it relies on
* the presence of UTF-8 encoded accented and other characters found in the upper
* ranges of the Latin-1 and (particularly) Windows-1252 codepages.
*
* - For more general detection routines, see existing projects / resources:
* - MLang - Microsoft library originally for IE6, available in Windows XP and later APIs now (I think?)
* - MLang .Net bindings: http://www.codeproject.com/KB/recipes/DetectEncoding.aspx
* - CharDet - Mozilla browser's detection routines
* - Ported to Java then .Net: http://www.conceptdevelopment.net/Localization/NCharDet/
* - Ported straight to .Net: http://code.google.com/p/chardetsharp/source/browse
*
* Copyright Tao Klerks, 2010-2012, tao@klerks.biz
* Licensed under the modified BSD license:
*
Redistribution and use in source and binary forms, with or without modification, are
permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of
conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list
of conditions and the following disclaimer in the documentation and/or other materials
provided with the distribution.
- The name of the author may not be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
OF SUCH DAMAGE.
*
* CHANGELOG:
* - 2012-02-03:
* - Simpler methods, removing the silly "DefaultEncoding" parameter (with "??" operator, saves no typing)
* - More complete methods
* - Optionally return indication of whether BOM was found in "Detect" methods
* - Provide straight-to-string method for byte arrays (GetStringFromByteArray)
*/
const long _defaultHeuristicSampleSize = 0x10000; //completely arbitrary - inappropriate for high numbers of files / high speed requirements
public static Encoding DetectTextFileEncoding(string InputFilename)
{
using (FileStream textfileStream = File.OpenRead(InputFilename))
{
return DetectTextFileEncoding(textfileStream, _defaultHeuristicSampleSize);
}
}
public static Encoding DetectTextFileEncoding(FileStream InputFileStream, long HeuristicSampleSize)
{
bool uselessBool = false;
return DetectTextFileEncoding(InputFileStream, _defaultHeuristicSampleSize, out uselessBool);
}
public static Encoding DetectTextFileEncoding(FileStream InputFileStream, long HeuristicSampleSize, out bool HasBOM)
{
if (InputFileStream == null)
throw new ArgumentNullException("Must provide a valid Filestream!", "InputFileStream");
if (!InputFileStream.CanRead)
throw new ArgumentException("Provided file stream is not readable!", "InputFileStream");
if (!InputFileStream.CanSeek)
throw new ArgumentException("Provided file stream cannot seek!", "InputFileStream");
Encoding encodingFound = null;
long originalPos = InputFileStream.Position;
InputFileStream.Position = 0;
//First read only what we need for BOM detection
byte[] bomBytes = new byte[InputFileStream.Length > 4 ? 4 : InputFileStream.Length];
InputFileStream.Read(bomBytes, 0, bomBytes.Length);
encodingFound = DetectBOMBytes(bomBytes);
if (encodingFound != null)
{
InputFileStream.Position = originalPos;
HasBOM = true;
return encodingFound;
}
//BOM Detection failed, going for heuristics now.
// create sample byte array and populate it
byte[] sampleBytes = new byte[HeuristicSampleSize > InputFileStream.Length ? InputFileStream.Length : HeuristicSampleSize];
Array.Copy(bomBytes, sampleBytes, bomBytes.Length);
if (InputFileStream.Length > bomBytes.Length)
InputFileStream.Read(sampleBytes, bomBytes.Length, sampleBytes.Length - bomBytes.Length);
InputFileStream.Position = originalPos;
//test byte array content
encodingFound = DetectUnicodeInByteSampleByHeuristics(sampleBytes);
HasBOM = false;
return encodingFound;
}
public static Encoding DetectTextByteArrayEncoding(byte[] TextData)
{
bool uselessBool = false;
return DetectTextByteArrayEncoding(TextData, out uselessBool);
}
public static Encoding DetectTextByteArrayEncoding(byte[] TextData, out bool HasBOM)
{
if (TextData == null)
throw new ArgumentNullException("Must provide a valid text data byte array!", "TextData");
Encoding encodingFound = null;
encodingFound = DetectBOMBytes(TextData);
if (encodingFound != null)
{
HasBOM = true;
return encodingFound;
}
else
{
//test byte array content
encodingFound = DetectUnicodeInByteSampleByHeuristics(TextData);
HasBOM = false;
return encodingFound;
}
}
public static string GetStringFromByteArray(byte[] TextData, Encoding DefaultEncoding)
{
return GetStringFromByteArray(TextData, DefaultEncoding, _defaultHeuristicSampleSize);
}
public static string GetStringFromByteArray(byte[] TextData, Encoding DefaultEncoding, long MaxHeuristicSampleSize)
{
if (TextData == null)
throw new ArgumentNullException("Must provide a valid text data byte array!", "TextData");
Encoding encodingFound = null;
encodingFound = DetectBOMBytes(TextData);
if (encodingFound != null)
{
//For some reason, the default encodings don't detect/swallow their own preambles!!
return encodingFound.GetString(TextData, encodingFound.GetPreamble().Length, TextData.Length - encodingFound.GetPreamble().Length);
}
else
{
byte[] heuristicSample = null;
if (TextData.Length > MaxHeuristicSampleSize)
{
heuristicSample = new byte[MaxHeuristicSampleSize];
Array.Copy(TextData, heuristicSample, MaxHeuristicSampleSize);
}
else
{
heuristicSample = TextData;
}
encodingFound = DetectUnicodeInByteSampleByHeuristics(TextData) ?? DefaultEncoding;
return encodingFound.GetString(TextData);
}
}
public static Encoding DetectBOMBytes(byte[] BOMBytes)
{
if (BOMBytes == null)
throw new ArgumentNullException("Must provide a valid BOM byte array!", "BOMBytes");
if (BOMBytes.Length < 2)
return null;
if (BOMBytes[0] == 0xff
&& BOMBytes[1] == 0xfe
&& (BOMBytes.Length < 4
|| BOMBytes[2] != 0
|| BOMBytes[3] != 0
)
)
return Encoding.Unicode;
if (BOMBytes[0] == 0xfe
&& BOMBytes[1] == 0xff
)
return Encoding.BigEndianUnicode;
if (BOMBytes.Length < 3)
return null;
if (BOMBytes[0] == 0xef && BOMBytes[1] == 0xbb && BOMBytes[2] == 0xbf)
return Encoding.UTF8;
if (BOMBytes[0] == 0x2b && BOMBytes[1] == 0x2f && BOMBytes[2] == 0x76)
return Encoding.UTF7;
if (BOMBytes.Length < 4)
return null;
if (BOMBytes[0] == 0xff && BOMBytes[1] == 0xfe && BOMBytes[2] == 0 && BOMBytes[3] == 0)
return Encoding.UTF32;
if (BOMBytes[0] == 0 && BOMBytes[1] == 0 && BOMBytes[2] == 0xfe && BOMBytes[3] == 0xff)
return Encoding.GetEncoding(12001);
return null;
}
public static Encoding DetectUnicodeInByteSampleByHeuristics(byte[] SampleBytes)
{
long oddBinaryNullsInSample = 0;
long evenBinaryNullsInSample = 0;
long suspiciousUTF8SequenceCount = 0;
long suspiciousUTF8BytesTotal = 0;
long likelyUSASCIIBytesInSample = 0;
//Cycle through, keeping count of binary null positions, possible UTF-8
// sequences from upper ranges of Windows-1252, and probable US-ASCII
// character counts.
long currentPos = 0;
int skipUTF8Bytes = 0;
while (currentPos < SampleBytes.Length)
{
//binary null distribution
if (SampleBytes[currentPos] == 0)
{
if (currentPos % 2 == 0)
evenBinaryNullsInSample++;
else
oddBinaryNullsInSample++;
}
//likely US-ASCII characters
if (IsCommonUSASCIIByte(SampleBytes[currentPos]))
likelyUSASCIIBytesInSample++;
//suspicious sequences (look like UTF-8)
if (skipUTF8Bytes == 0)
{
int lengthFound = DetectSuspiciousUTF8SequenceLength(SampleBytes, currentPos);
if (lengthFound > 0)
{
suspiciousUTF8SequenceCount++;
suspiciousUTF8BytesTotal += lengthFound;
skipUTF8Bytes = lengthFound - 1;
}
}
else
{
skipUTF8Bytes--;
}
currentPos++;
}
//1: UTF-16 LE - in english / european environments, this is usually characterized by a
// high proportion of odd binary nulls (starting at 0), with (as this is text) a low
// proportion of even binary nulls.
// The thresholds here used (less than 20% nulls where you expect non-nulls, and more than
// 60% nulls where you do expect nulls) are completely arbitrary.
if (((evenBinaryNullsInSample * 2.0) / SampleBytes.Length) < 0.2
&& ((oddBinaryNullsInSample * 2.0) / SampleBytes.Length) > 0.6
)
return Encoding.Unicode;
//2: UTF-16 BE - in english / european environments, this is usually characterized by a
// high proportion of even binary nulls (starting at 0), with (as this is text) a low
// proportion of odd binary nulls.
// The thresholds here used (less than 20% nulls where you expect non-nulls, and more than
// 60% nulls where you do expect nulls) are completely arbitrary.
if (((oddBinaryNullsInSample * 2.0) / SampleBytes.Length) < 0.2
&& ((evenBinaryNullsInSample * 2.0) / SampleBytes.Length) > 0.6
)
return Encoding.BigEndianUnicode;
//3: UTF-8 - Martin Dürst outlines a method for detecting whether something CAN be UTF-8 content
// using regexp, in his w3c.org unicode FAQ entry:
// http://www.w3.org/International/questions/qa-forms-utf-8
// adapted here for C#.
string potentiallyMangledString = Encoding.ASCII.GetString(SampleBytes);
Regex UTF8Validator = new Regex(@"\A("
+ @"[\x09\x0A\x0D\x20-\x7E]"
+ @"|[\xC2-\xDF][\x80-\xBF]"
+ @"|\xE0[\xA0-\xBF][\x80-\xBF]"
+ @"|[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}"
+ @"|\xED[\x80-\x9F][\x80-\xBF]"
+ @"|\xF0[\x90-\xBF][\x80-\xBF]{2}"
+ @"|[\xF1-\xF3][\x80-\xBF]{3}"
+ @"|\xF4[\x80-\x8F][\x80-\xBF]{2}"
+ @")*\z");
if (UTF8Validator.IsMatch(potentiallyMangledString))
{
//Unfortunately, just the fact that it CAN be UTF-8 doesn't tell you much about probabilities.
//If all the characters are in the 0-127 range, no harm done, most western charsets are same as UTF-8 in these ranges.
//If some of the characters were in the upper range (western accented characters), however, they would likely be mangled to 2-byte by the UTF-8 encoding process.
// So, we need to play stats.
// The "Random" likelihood of any pair of randomly generated characters being one
// of these "suspicious" character sequences is:
// 128 / (256 * 256) = 0.2%.
//
// In western text data, that is SIGNIFICANTLY reduced - most text data stays in the <127
// character range, so we assume that more than 1 in 500,000 of these character
// sequences indicates UTF-8. The number 500,000 is completely arbitrary - so sue me.
//
// We can only assume these character sequences will be rare if we ALSO assume that this
// IS in fact western text - in which case the bulk of the UTF-8 encoded data (that is
// not already suspicious sequences) should be plain US-ASCII bytes. This, I
// arbitrarily decided, should be 80% (a random distribution, eg binary data, would yield
// approx 40%, so the chances of hitting this threshold by accident in random data are
// VERY low).
if ((suspiciousUTF8SequenceCount * 500000.0 / SampleBytes.Length >= 1) //suspicious sequences
&& (
//all suspicious, so cannot evaluate proportion of US-Ascii
SampleBytes.Length - suspiciousUTF8BytesTotal == 0
||
likelyUSASCIIBytesInSample * 1.0 / (SampleBytes.Length - suspiciousUTF8BytesTotal) >= 0.8
)
)
return Encoding.UTF8;
}
return null;
}
private static bool IsCommonUSASCIIByte(byte testByte)
{
if (testByte == 0x0A //lf
|| testByte == 0x0D //cr
|| testByte == 0x09 //tab
|| (testByte >= 0x20 && testByte <= 0x2F) //common punctuation
|| (testByte >= 0x30 && testByte <= 0x39) //digits
|| (testByte >= 0x3A && testByte <= 0x40) //common punctuation
|| (testByte >= 0x41 && testByte <= 0x5A) //capital letters
|| (testByte >= 0x5B && testByte <= 0x60) //common punctuation
|| (testByte >= 0x61 && testByte <= 0x7A) //lowercase letters
|| (testByte >= 0x7B && testByte <= 0x7E) //common punctuation
)
return true;
else
return false;
}
private static int DetectSuspiciousUTF8SequenceLength(byte[] SampleBytes, long currentPos)
{
int lengthFound = 0;
if (SampleBytes.Length >= currentPos + 1
&& SampleBytes[currentPos] == 0xC2
)
{
if (SampleBytes[currentPos + 1] == 0x81
|| SampleBytes[currentPos + 1] == 0x8D
|| SampleBytes[currentPos + 1] == 0x8F
)
lengthFound = 2;
else if (SampleBytes[currentPos + 1] == 0x90
|| SampleBytes[currentPos + 1] == 0x9D
)
lengthFound = 2;
else if (SampleBytes[currentPos + 1] >= 0xA0
&& SampleBytes[currentPos + 1] <= 0xBF
)
lengthFound = 2;
}
else if (SampleBytes.Length >= currentPos + 1
&& SampleBytes[currentPos] == 0xC3
)
{
if (SampleBytes[currentPos + 1] >= 0x80
&& SampleBytes[currentPos + 1] <= 0xBF
)
lengthFound = 2;
}
else if (SampleBytes.Length >= currentPos + 1
&& SampleBytes[currentPos] == 0xC5
)
{
if (SampleBytes[currentPos + 1] == 0x92
|| SampleBytes[currentPos + 1] == 0x93
)
lengthFound = 2;
else if (SampleBytes[currentPos + 1] == 0xA0
|| SampleBytes[currentPos + 1] == 0xA1
)
lengthFound = 2;
else if (SampleBytes[currentPos + 1] == 0xB8
|| SampleBytes[currentPos + 1] == 0xBD
|| SampleBytes[currentPos + 1] == 0xBE
)
lengthFound = 2;
}
else if (SampleBytes.Length >= currentPos + 1
&& SampleBytes[currentPos] == 0xC6
)
{
if (SampleBytes[currentPos + 1] == 0x92)
lengthFound = 2;
}
else if (SampleBytes.Length >= currentPos + 1
&& SampleBytes[currentPos] == 0xCB
)
{
if (SampleBytes[currentPos + 1] == 0x86
|| SampleBytes[currentPos + 1] == 0x9C
)
lengthFound = 2;
}
else if (SampleBytes.Length >= currentPos + 2
&& SampleBytes[currentPos] == 0xE2
)
{
if (SampleBytes[currentPos + 1] == 0x80)
{
if (SampleBytes[currentPos + 2] == 0x93
|| SampleBytes[currentPos + 2] == 0x94
)
lengthFound = 3;
if (SampleBytes[currentPos + 2] == 0x98
|| SampleBytes[currentPos + 2] == 0x99
|| SampleBytes[currentPos + 2] == 0x9A
)
lengthFound = 3;
if (SampleBytes[currentPos + 2] == 0x9C
|| SampleBytes[currentPos + 2] == 0x9D
|| SampleBytes[currentPos + 2] == 0x9E
)
lengthFound = 3;
if (SampleBytes[currentPos + 2] == 0xA0
|| SampleBytes[currentPos + 2] == 0xA1
|| SampleBytes[currentPos + 2] == 0xA2
)
lengthFound = 3;
if (SampleBytes[currentPos + 2] == 0xA6)
lengthFound = 3;
if (SampleBytes[currentPos + 2] == 0xB0)
lengthFound = 3;
if (SampleBytes[currentPos + 2] == 0xB9
|| SampleBytes[currentPos + 2] == 0xBA
)
lengthFound = 3;
}
else if (SampleBytes[currentPos + 1] == 0x82
&& SampleBytes[currentPos + 2] == 0xAC
)
lengthFound = 3;
else if (SampleBytes[currentPos + 1] == 0x84
&& SampleBytes[currentPos + 2] == 0xA2
)
lengthFound = 3;
}
return lengthFound;
}
}
}
使用系统；
使用系统文本；
使用System.Text.RegularExpressions；
使用System.IO；
名称空间Klerksoft
{
公共静态类TextFileEncodingDetector
{
/*
*处理文本文件编码问题的简单类（以英语为主）
*世界）。
*
*-此代码是完全托管的，没有对MLang（非托管代码页）的隐藏调用
*最初为Internet Explorer开发的检测库）。
*
*-该类不尝试检测任意代码页/字符集，它实际上只是
*旨在区分一些最常见的Unicode变体
*编码，并提供“默认”（基于西方/ascii）编码替代方案
*打电话的人。
*
*-因为没有“可靠”的方法来区分UTF-8（无BOM）和
*Windows-1252（在.Net中，也称为“ASCII”）编码，我们使用
*启发式-因此我们可以对文件进行越多的采样，猜测就越好
*将在某个时候将整个文件读入内存，然后最好通过
*直接在整个字节数组中。否则，决定如何权衡
*针对性能/内存使用的可靠性。
*
*-UTF-8检测启发式仅适用于西方文本，因为它依赖于
*上表中存在UTF-8编码的重音符号和其他字符
*拉丁语-1和（特别是）Windows-1252代码页的范围。
*
*-有关更一般的检测例程，请参阅现有项目/资源：
*-MLang-最初用于IE6的Microsoft库，现在可以在Windows XP和更高版本的API中使用（我想是吧？）
*-MLang.Net绑定：http://www.codeproject.com/KB/recipes/DetectEncoding.aspx
*-CharDet-Mozilla浏览器的检测例程
*-移植到Java然后.Net：http://www.conceptdevelopment.net/Localization/NCharDet/
*-直接移植到.Net：http://code.google.com/p/chardetsharp/source/browse
*
*Tao Klerks版权所有，2010-2012，tao@klerks.biz
*根据修改后的BSD许可证进行许可：
*
在源代码和二进制格式中重新分发和使用，无论是否修改，都是必要的
如果满足以下条件，则允许：
-重新分发源代码必须保留上述版权声明，此列表为
条件和以下免责声明。
-
/// <summary>
/// UTF8 : EF BB BF
/// UTF16 BE: FE FF
/// UTF16 LE: FF FE
/// UTF32 BE: 00 00 FE FF
/// UTF32 LE: FF FE 00 00
/// </summary>
public static Encoding DetectEncoding(Stream i_Stream)
{
if (!i_Stream.CanSeek || !i_Stream.CanRead)
throw new Exception("DetectEncoding() requires a seekable and readable Stream");
// Try to read 4 bytes. If the stream is shorter, less bytes will be read.
Byte[] u8_Buf = new Byte[4];
int s32_Count = i_Stream.Read(u8_Buf, 0, 4);
if (s32_Count >= 2)
{
if (u8_Buf[0] == 0xFE && u8_Buf[1] == 0xFF)
{
i_Stream.Position = 2;
return new UnicodeEncoding(true, true);
}
if (u8_Buf[0] == 0xFF && u8_Buf[1] == 0xFE)
{
if (s32_Count >= 4 && u8_Buf[2] == 0 && u8_Buf[3] == 0)
{
i_Stream.Position = 4;
return new UTF32Encoding(false, true);
}
else
{
i_Stream.Position = 2;
return new UnicodeEncoding(false, true);
}
}
if (s32_Count >= 3 && u8_Buf[0] == 0xEF && u8_Buf[1] == 0xBB && u8_Buf[2] == 0xBF)
{
i_Stream.Position = 3;
return Encoding.UTF8;
}
if (s32_Count >= 4 && u8_Buf[0] == 0 && u8_Buf[1] == 0 && u8_Buf[2] == 0xFE && u8_Buf[3] == 0xFF)
{
i_Stream.Position = 4;
return new UTF32Encoding(true, true);
}
}
i_Stream.Position = 0;
return Encoding.Default;
}