用于复杂/大型XML的Java Stax

我有一个4.2GB的XML文件！显然，解析整个DOM是不实际的。我一直在研究SAX和STAX来完成对这个巨大XML文件的解析。然而，我看到的所有例子都很简单。我正在处理的XML文件已嵌套在嵌套上。在某些领域，它可以达到10+个级别

我找到了这个教程，但不确定它是否是一个可行的解决方案

（使用STAX的botton示例）

我不确定如何处理嵌套对象

我创建了Java对象来模拟XML的结构。这里有一些，太多了，无法展示

Record.java

public class Record implements Serializable {

    String uid;
    StaticData staticData;
    DynamicData dynamicData;
}

Summary.java

public class Summary {

    EWUID ewuid;
    PubInfo pubInfo;
    Titles titles;
    Names names;
    DocTypes docTypes;
    Publishers publishers;
}

EWUID.java

public class EWUID {

    String collId;
    String edition;
}

PubInfo.java

public class PubInfo {

    String coverDate;
    String hasAbstract;
    String issue;
    String pubMonth;
    String pubType;
    String pubYear;
    String sortDate;
    String volume;
}

这是我到目前为止想出的代码

public class TRWOSParser {

    XMLEventReader eventReader;
    XMLInputFactory inputFactory;
    InputStream inputStream;

    public TRWOSParser(String file) throws FileNotFoundException, XMLStreamException {
        inputFactory = XMLInputFactory.newInstance();
        inputStream = new FileInputStream(file);
        eventReader = inputFactory.createXMLEventReader(inputStream);
    }

    public void parse() throws XMLStreamException{

        while (eventReader.hasNext()) {
            XMLEvent event = eventReader.nextEvent();

            if (event.isStartElement()) {
                StartElement startElement = event.asStartElement();
                if (startElement.getName().getLocalPart().equals("record")) {
                    Record record = new Record();
                    Iterator<Attribute> attributes = startElement.getAttributes();
                    while (attributes.hasNext()) {
                        Attribute attribute = attributes.next();
                        if (attribute.getName().toString().equals("UID")) {
                            System.out.println("UID: " + attribute.getValue());
                        }
                    }
                }
            }
        }
    }
}

公共类TRWOSParser{
XMLEventReader事件阅读器；
XMLInputFactory输入工厂；
输入流输入流；
公共TRWOSParser（字符串文件）抛出FileNotFoundException、XMLStreamException{
inputFactory=XMLInputFactory.newInstance（）；
inputStream=新文件inputStream（文件）；
eventReader=inputFactory.createXMLEventReader（inputStream）；
}
public void parse（）引发XMLStreamException{
while（eventReader.hasNext（））{
XMLEvent=eventReader.nextEvent（）；
if（event.isStartElement（））{
StartElement StartElement=event.asStartElement（）；
if（startElement.getName（）.getLocalPart（）.equals（“记录”））{
记录=新记录（）；
迭代器属性=startElement.getAttributes（）；
while（attributes.hasNext（））{
Attribute=attributes.next（）；
if（attribute.getName（）.toString（）.equals（“UID”））{
System.out.println（“UID:+attribute.getValue（））；
}
}
}
}
}
}
}

更新：

XML中的数据已获得许可，因此无法显示完整文件。这是一个非常非常小的片段，我在其中对数据进行了置乱

<?xml version="1.0" encoding="UTF-8"?>
<records>
    <REC>
        <UID>WOS:000310438600004</UID>
        <static_data>
            <summary>
                <EWUID>
                    <WUID coll_id="WOS" />
                    <edition value="WOS.SCI" />
                </EWUID>
                <pub_info coverdate="NOV 2012" has_abstract="N" issue="5" pubmonth="NOV" pubtype="Journal" pubyear="2012" sortdate="2012-11-01" vol="188">
                    <page begin="1662" end="1663" page_count="2">1662-1663</page>
                </pub_info>
                <titles count="6">
                    <title type="source">JOURNAL OF UROLOGY</title>
                    <title type="source_abbrev">J UROLOGY</title>
                    <title type="abbrev_iso">J. Urol.</title>
                    <title type="abbrev_11">J UROL</title>
                    <title type="abbrev_29">J UROL</title>
                    <title type="item">Something something</title>
                </titles>
                <names count="1">
                    <name addr_no="1 2 3" reprint="Y" role="author" seq_no="1">
                        <display_name>John Doe</display_name>
                        <full_name>John Doe</full_name>
                        <wos_standard>Doe, John</wos_standard>
                        <first_name>John</first_name>
                        <last_name>Doe</last_name>
                    </name>
                </names>
                <doctypes count="1">
                    <doctype>Editorial Material</doctype>
                </doctypes>
                <publishers>
                    <publisher>
                        <address_spec addr_no="1">
                            <full_address>360 PARK AVE SOUTH, NEW YORK, NY 10010-1710 USA</full_address>
                            <city>NEW YORK</city>
                        </address_spec>
                        <names count="1">
                            <name addr_no="1" role="publisher" seq_no="1">
                                <display_name>ELSEVIER SCIENCE INC</display_name>
                                <full_name>ELSEVIER SCIENCE INC</full_name>
                            </name>
                        </names>
                    </publisher>
                </publishers>
            </summary>
        </static_data>
    </REC>
</records>

WOS:000310438600004
1662-1663
泌尿外科杂志
泌尿学杂志
尤罗。
尤罗
尤罗
什么东西
无名氏
无名氏
多伊，约翰
约翰
雌鹿
编辑材料
美国纽约州纽约市公园大道南360号，邮编：10010-1710
纽约
爱思唯尔科学公司
爱思唯尔科学公司

我做了两个假设：1）有早期的重复，2）你可以用部分文档做一些有意义的事情

让我们假设您可以移动某种级别的嵌套，然后多次处理文档，每次“处理”文档时都会删除工作级别的节点。这意味着在任何给定的时间，只有一个工作子树在内存中

下面是一段有效的代码片段：

package bigparse;

import static javax.xml.stream.XMLStreamConstants.CHARACTERS;
import static javax.xml.stream.XMLStreamConstants.END_DOCUMENT;
import static javax.xml.stream.XMLStreamConstants.END_ELEMENT;
import static javax.xml.stream.XMLStreamConstants.START_DOCUMENT;
import static javax.xml.stream.XMLStreamConstants.START_ELEMENT;

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.StringWriter;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamReader;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

public class BigParse {

    public static void main(String... args) {

        XMLInputFactory factory = XMLInputFactory.newInstance();
        DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();

        try {

            XMLStreamReader streamReader = factory.createXMLStreamReader(new FileReader("src/main/resources/test.xml"));
            DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();

            Document document = documentBuilder.newDocument();
            Element rootElement = null;
            Element currentElement = null;

            int branchLevel = 0;
            int maxBranchLevel = 1;

            while (streamReader.hasNext()) {
                int event = streamReader.next();
                switch (event) {
                case START_DOCUMENT:
                    continue;
                case START_ELEMENT:

                    if (branchLevel < maxBranchLevel) {
                        Element workingElement = readElementOnly(streamReader, document);

                        if (rootElement == null) {
                            document.appendChild(workingElement);
                            rootElement = document.getDocumentElement();
                            currentElement = rootElement;
                        } else {
                            currentElement.appendChild(workingElement);
                            currentElement = workingElement;
                        }

                        branchLevel++;
                    } else {

                        workingLoop(streamReader, document, currentElement);

                    }

                    continue;
                case CHARACTERS:
                    currentElement.setTextContent(streamReader.getText());
                    continue;

                case END_ELEMENT:
                    if (currentElement != rootElement) {
                        currentElement = (Element) currentElement.getParentNode();
                        branchLevel--;
                    }

                    continue;

                case END_DOCUMENT:
                    break;
                }

            }

        } catch (ParserConfigurationException
                | FileNotFoundException
                | XMLStreamException e) {
            throw new RuntimeException(e);
        }

    }

    private static Element readElementOnly(XMLStreamReader streamReader, Document document) {

        Element workingElement = document.createElement(streamReader.getLocalName());

        for (int attributeIndex = 0; attributeIndex < streamReader.getAttributeCount(); attributeIndex++) {
            workingElement.setAttribute(
                    streamReader.getAttributeLocalName(attributeIndex),
                    streamReader.getAttributeValue(attributeIndex));

        }
        return workingElement;
    }

    private static void workingLoop(final XMLStreamReader streamReader, final Document document, final Element fragmentRoot)
            throws XMLStreamException {

        Element startElement = readElementOnly(streamReader, document);
        fragmentRoot.appendChild(startElement);

        Element currentElement = startElement;

        while (streamReader.hasNext()) {
            int event = streamReader.next();
            switch (event) {
            case START_DOCUMENT:
                continue;
            case START_ELEMENT:

                Element workingElement = readElementOnly(streamReader, document);
                currentElement.appendChild(workingElement);
                currentElement = workingElement;

                continue;
            case CHARACTERS:
                currentElement.setTextContent(streamReader.getText());
                continue;

            case END_ELEMENT:
                if (currentElement != startElement) {
                    currentElement = (Element) currentElement.getParentNode();
                    continue;

                } else {

                    handleDocument(document, startElement);

                    fragmentRoot.removeChild(startElement);
                    startElement = null;
                    return;
                }

            }

        }

    }

    // THIS FUNCTION DOES SOMETHING MEANINFUL
    private static void handleDocument(Document document, Element startElement) {

        System.out.println(stringify(document));
    }

    private static String stringify(Document document) {

        try {
            Transformer transformer = TransformerFactory.newInstance().newTransformer();
            transformer.setOutputProperty(OutputKeys.INDENT, "yes");

            StreamResult result = new StreamResult(new StringWriter());
            DOMSource source = new DOMSource(document);

            transformer.transform(source, result);

            String xmlString = result.getWriter().toString();
            return xmlString;

        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

}

packagebigparse；
导入静态javax.xml.stream.XMLStreamConstants.CHARACTERS；
导入静态javax.xml.stream.XMLStreamConstants.END_文档；
导入静态javax.xml.stream.XMLStreamConstants.END_元素；
导入静态javax.xml.stream.XMLStreamConstants.START_文档；
导入静态javax.xml.stream.XMLStreamConstants.START_元素；
导入java.io.FileNotFoundException；
导入java.io.FileReader；
导入java.io.StringWriter；
导入javax.xml.parsers.DocumentBuilder；
导入javax.xml.parsers.DocumentBuilderFactory；
导入javax.xml.parsers.parserConfiguration异常；
导入javax.xml.stream.XMLInputFactory；
导入javax.xml.stream.XMLStreamException；
导入javax.xml.stream.XMLStreamReader；
导入javax.xml.transform.OutputKeys；
导入javax.xml.transform.Transformer；
导入javax.xml.transform.TransformerFactory；
导入javax.xml.transform.dom.DOMSource；
导入javax.xml.transform.stream.StreamResult；
导入org.w3c.dom.Document；
导入org.w3c.dom.Element；
公共类BigParse{
公共静态void main（字符串…参数）{
XMLInputFactory=XMLInputFactory.newInstance（）；
DocumentBuilderFactory DocumentBuilderFactory=DocumentBuilderFactory.newInstance（）；
试一试{
XMLStreamReader=factory.createXMLStreamReader（新文件阅读器（“src/main/resources/test.xml”）；
DocumentBuilder DocumentBuilder=documentBuilderFactory.newDocumentBuilder（）；
Document Document=documentBuilder.newDocument（）；
元素rootElement=null；
元素currentElement=null；
int branchLevel=0；
int maxBranchLevel=1；
while（streamReader.hasNext（））{
int event=streamReader.next（）；
开关（事件）{
案例启动文件：
继续；
案例开始元素：
if（branchLevel<xsl:template name="main">
  <xsl:stream href="bigInput.xml">
    <xsl:for-each select="copy-of(/records/REC)">
      <!-- process one record -->
    </xsl:for-each>
  </xsl:stream>
</xsl:template>