处理来自Java输入流的下一个字符(完整unicode代码点)流
我需要逐个字符解析UTF-8输入流(UTF-8代码点,而不是Java的字符)。最好的方法是什么 更新问题以使其更清楚(感谢@skomisa): 因此,以下非流媒体:处理来自Java输入流的下一个字符(完整unicode代码点)流,java,unicode,stream,Java,Unicode,Stream,我需要逐个字符解析UTF-8输入流(UTF-8代码点,而不是Java的字符)。最好的方法是什么 更新问题以使其更清楚(感谢@skomisa): 因此,以下非流媒体: private static String replaceNonBPMWithUnknownCharacter(final String input) { StringBuilder result = new StringBuilder(input.length()); input.codePoints().forE
private static String replaceNonBPMWithUnknownCharacter(final String input) {
StringBuilder result = new StringBuilder(input.length());
input.codePoints().forEach((codePoint) -> {
if (isBmpCodePoint(codePoint)) {
result.append('\ufffd');
} else {
result.append(isBmpCodePoint(codePoint) ? toChars(codePoint) : REPLACEMENT_CHAR);
}
});
return result.toString();
}
String result = replaceNonBPMWithUnknownCharacter("\uD83D\uDE0E? X")
我想要流媒体版本,例如:
InputStream stream = replaceNonBPMWithUnknownCharacter(new ByteArrayInputStream("\uD83D\uDE0E? Y".getBytes(UTF_8)))
使用尽可能少的cpu和内存。以下问题类似,但不是流式问题:
最重要的是:如何从流中读取代码点?(因此,我如何将我知道它们是UTF-8编码的字节流转换为代码点流)。首先请注意:
- UTF-8字符可以由1、2、3或4字节序列组成
- 字符中的字节数由第一个(或唯一)字节中的某些位设置确定。详见中的表3.6 UTF-8位分布
- 读取第一个字节并检查其位模式,以确定第一个字符中有多少字节
- 读取第一个字符所需的任何后续字节
- 基于组成该字符的字节创建一个
,然后调用字符串
获取其代码点String.codePointAt()
- 将该代码点添加到
列表中
- 对后续字节重复前面的所有步骤,直到EOF
- 将
作为代码点流返回列表
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;
import static java.nio.charset.StandardCharsets.UTF_8;
public class Main {
public static void main(String[] args) {
String text = "¢\uD841\uDF31\u30918औWあش";
Stream<Integer> codePoints = Main.processByteStream(new ByteArrayInputStream(text.getBytes(UTF_8)));
codePoints.forEach(System.out::println);
}
/**
* Processes a stream of bytes, and returns a Stream of Unicode codepoints
* associated with the characters derived from that byte stream.
*
* @param bais ByteArrayInputStream to be processed.
* @return A stream of Unicode codepoints derived from UTF-8 characters in the supplied stream.
*/
private static Stream<Integer> processByteStream(ByteArrayInputStream bais) {
int nextByte = 0;
byte b = 0;
byte[] utf8Bytes = null;
int byteCount = 0;
List<Integer> codePoints = new ArrayList<>();
while ((nextByte = bais.read()) != -1) {
b = (byte) nextByte;
byteCount = Main.getByteCount(b);
utf8Bytes = new byte[byteCount];
utf8Bytes[0] = (byte) nextByte;
for (int i = 1; i < byteCount; i++) { // Get any subsequent bytes for this UTF-8 character.
nextByte = bais.read();
utf8Bytes[i] = (byte) nextByte;
}
int codePoint = new String(utf8Bytes, StandardCharsets.UTF_8).codePointAt(0);
codePoints.add(codePoint);
}
return codePoints.stream();
}
/**
* Returns the number of bytes in a UTF-8 character based on the bit pattern
* of the supplied byte. The only valid values are 1, 2 3 or 4. If the
* byte has an invalid bit pattern an IllegalArgumentException is thrown.
*
* @param b The first byte of a UTF-8 character.
* @return The number of bytes for this UTF-* character.
* @throws IllegalArgumentException if the bit pattern is invalid.
*/
private static int getByteCount(byte b) throws IllegalArgumentException {
if ((b >= 0)) return 1; // Pattern is 0xxxxxxx.
if ((b >= (byte) 0b11000000) && (b <= (byte) 0b11011111)) return 2; // Pattern is 110xxxxx.
if ((b >= (byte) 0b11100000) && (b <= (byte) 0b11101111)) return 3; // Pattern is 1110xxxx.
if ((b >= (byte) 0b11110000) && (b <= (byte) 0b11110111)) return 4; // Pattern is 11110xxx.
throw new IllegalArgumentException(); // Invalid first byte for UTF-8 character.
}
}
注:
- 可以直接从字符的字节派生代码点,但是。调用
是一种效率较低但更干净的替代方法String.codePointAt()
- 我无法生成无效数据。似乎任何无效字节都会以某种方式转换为U+FFFD(十进制65533),因此抛出
可能是没有必要的IllegalArgumentException
返回新字符串(bais.readAllBytes(),StandardCharsets.UTF_8).codePoints();
),但对于非常大的输入流,我看不到任何避免字节级处理的方法。您当然可以通过一次读取多个字节来提高性能,但这会使我的答案中的代码更加复杂。
C:\Java\openJDK\jdk-11.0.2\bin\java.exe -javaagent:C:\Users\johndoe\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\191.4738.6\lib\idea_rt.jar=60544:C:\Users\johndoe\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\191.4738.6\bin -Dfile.encoding=UTF-8 -classpath C:\Users\johndoe\IdeaProjects\Codepoint\out\production\Codepoint Main
162
132913
12433
56
2324
87
12354
1588
Process finished with exit code 0