在SAX（Java）中解析大型XML文件时在DOM中加载本地块

我有一个xml文件，可以避免在内存中加载所有文件。 众所周知，对于这样的文件，我最好使用SAX解析器（如果找到相关的东西，它将沿着文件调用事件）

我当前的问题是我想“按块”处理文件，这意味着：

解析文件并找到相关标记（节点）

将此标记完全加载到内存中（就像我们在DOM中那样）

执行此实体（该本地块）的进程

当我处理完区块后，释放它并继续1。（直到“文件结束”）

在一个完美的世界里，我在寻找这样的东西：

// 1. Create a parser and set the file to load
      IdealParser p = new IdealParser("BigFile.xml");
// 2. Set an XPath to define the interesting nodes
      p.setRelevantNodesPath("/path/to/relevant/nodes");
// 3. Add a handler to callback the right method once a node is found
      p.setHandler(new Handler(){
// 4. The method callback by the parser when a relevant node is found
      void aNodeIsFound(saxNode aNode)
   {
   // 5. Inflate the current node i.e. load it (and all its content) in memory
         DomNode d = aNode.expand();
   // 6. Do something with the inflated node (method to be defined somewhere)
         doThingWithNode(d);
    }
   });
// 7. Start the parser
      p.start();

我目前一直在研究如何有效地使用“sax节点”（理解我…）

---

有没有与这类任务相关的Java框架或库？

我做过SAX之后有一段时间，但您要做的是处理每个标记，直到找到要处理的组的结束标记，然后运行流程，清除它并查找下一个开始标记。

可以用SAX完成。。。但我认为较新的StAX（用于XML的流式API）将更好地满足您的需求。您可以使用它来解析文件，检测哪些节点符合您的条件之一。对于简单的基于路径的选择（不是真正的XPath，而是一些简单的

/

分隔路径），您需要通过在新元素的字符串中添加条目或在结束标记上剪切条目来维护当前节点的路径。布尔标志足以维持您当前是否处于“相关模式”

当您从您的读者那里获得信息时，您可以将相关的信息复制到某个合适的占位符上，如StringWriter或ByteArrayOutputStream。一旦您完成了一些XML提取的复制，这些XML提取形成了您希望为其构建DOM的“子文档”，只需以适当的形式将占位符提供给

这里的限制是您没有充分利用XPath语言的所有功能。如果您希望考虑节点位置之类的因素，那么您必须在自己的路径中预见到这一点。也许有人知道一种将真正的XPath实现集成到其中的好方法

StAX非常好，它让您可以控制解析，而不是通过SAX这样的处理程序使用回调接口

还有另一种选择：使用XSLT。XSLT样式表是只过滤相关内容的理想方法。您可以对输入进行一次转换，以获得所需的片段并对其进行处理。或者在同一输入上运行多个样式表，以每次获得所需的提取。然而，一个更好（更有效）的解决方案是使用

扩展函数的实现方式可以独立于所使用的XSLT处理器。它们在Java中的使用非常简单，我知道可以使用它们将完整的XML提取传递给方法，因为我已经这样做了。可能需要一些实验，但这是一个强大的机制。DOM提取（或节点）可能是此类方法可接受的参数类型之一。这将使文档由XSLT处理器生成，这将更加容易

扩展元素也非常有用，但我认为它们需要以特定于实现的方式使用。如果您愿意将自己绑定到特定的JAXP设置，比如Xerces+Xalan，那么他们可能就是答案

在使用XSLT时，您将拥有完整XPath1.0实现的所有优点，以及知道XSLT在Java中处于非常好的状态所带来的安心。它将输入树的构建限制在任何时候都需要的节点上，而且速度非常快，因为处理器倾向于将样式表编译成Java字节码，而不是解释它们。但是，使用编译而不是解释可能会失去使用扩展元素的可能性。对此不确定。扩展功能仍然是可能的

无论您选择哪种方式，Java中的XML处理有太多内容，如果您没有找到现成的解决方案，您将在实现这一点上找到大量帮助。那当然是最明显的事情了。。。当有人做了艰苦的工作时，没有必要重新发明轮子

祝你好运

编辑：因为我实际上一次也没有感到沮丧，这里有一个使用我开发的StAX解决方案的演示。这当然不是最干净的代码，但它会给你一个基本的想法：

package staxdom;

import java.io.IOException;
import java.io.InputStream;
import java.io.StringReader;
import java.io.StringWriter;
import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
import java.util.Stack;
import javax.xml.namespace.QName;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.stream.XMLEventReader;
import javax.xml.stream.XMLEventWriter;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.events.StartElement;
import javax.xml.stream.events.XMLEvent;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;

public class DOMExtractor {

    private final Set<String> paths;
    private final XMLInputFactory inputFactory;
    private final XMLOutputFactory outputFactory;
    private final DocumentBuilderFactory docBuilderFactory;
    private final Stack<QName> activeStack = new Stack<QName>();

    private boolean active = false;
    private String currentPath = "";

    public DOMExtractor(final Set<String> paths) {

        this.paths = Collections.unmodifiableSet(new HashSet<String>(paths));
        inputFactory = XMLInputFactory.newFactory();
        outputFactory = XMLOutputFactory.newFactory();
        docBuilderFactory = DocumentBuilderFactory.newInstance();

    }

    public void parse(final InputStream input) throws XMLStreamException, ParserConfigurationException, SAXException, IOException {

        final XMLEventReader reader = inputFactory.createXMLEventReader(input);
        XMLEventWriter writer = null;
        StringWriter buffer = null;
        final DocumentBuilder builder = docBuilderFactory.newDocumentBuilder();

        XMLEvent currentEvent = reader.nextEvent();

        do {

            if(active)
                writer.add(currentEvent);

            if(currentEvent.isEndElement()) {

                if(active) {

                    activeStack.pop();

                    if(activeStack.isEmpty()) {
                        writer.flush();
                        writer.close();
                        final Document doc;
                        final StringReader docReader = new StringReader(buffer.toString());
                        try {
                            doc = builder.parse(new InputSource(docReader));
                        } finally {
                            docReader.close();
                        }
                        //TODO: use doc
                        //Next bit is only for demo...
                        outputDoc(doc);
                        active = false;
                        writer = null;
                        buffer = null;
                    }

                }

                int index;
                if((index = currentPath.lastIndexOf('/')) >= 0)
                    currentPath = currentPath.substring(0, index);

            } else if(currentEvent.isStartElement()) {

                final StartElement start = (StartElement)currentEvent;
                final QName qName = start.getName();
                final String local = qName.getLocalPart();

                currentPath += "/" + local;

                if(!active && paths.contains(currentPath)) {

                    active = true;

                    buffer = new StringWriter();
                    writer = outputFactory.createXMLEventWriter(buffer);

                    writer.add(currentEvent);

                }

                if(active)
                    activeStack.push(qName);

            }

            currentEvent = reader.nextEvent();

        } while(!currentEvent.isEndDocument());

    }

    private void outputDoc(final Document doc) {


        try {
            final Transformer t = TransformerFactory.newInstance().newTransformer();
            t.transform(new DOMSource(doc), new StreamResult(System.out));
            System.out.println("");
            System.out.println("");
        } catch(TransformerException ex) {
            ex.printStackTrace();
        }

    }

    public static void main(String[] args) {

        final Set<String> paths = new HashSet<String>();
        paths.add("/root/one");
        paths.add("/root/three/embedded");

        final DOMExtractor me = new DOMExtractor(paths);

        InputStream stream = null;
        try {
            stream = DOMExtractor.class.getResourceAsStream("sample.xml");
            me.parse(stream);
        } catch(final Exception e) {
            e.printStackTrace();
        } finally {
            if(stream != null)
                try {
                    stream.close();
                } catch(IOException ex) {
                    ex.printStackTrace();
                }
        }

    }

}

包statxdom；
导入java.io.IOException；
导入java.io.InputStream；
导入java.io.StringReader；
导入java.io.StringWriter；
导入java.util.Collections；
导入java.util.HashSet；
导入java.util.Set；
导入java.util.Stack；
导入javax.xml.namespace.QName；
导入javax.xml.parsers.DocumentBuilder；
导入javax.xml.parsers.DocumentBuilderFactory；
导入javax.xml.parsers.parserConfiguration异常；
导入javax.xml.stream.XMLEventReader；
导入javax.xml.stream.XMLEventWriter；
导入javax.xml.stream.XMLInputFactory；
导入javax.xml.stream.XMLOutputFactory；
导入javax.xml.stream.XMLStreamException；
导入javax.xml.stream.events.StartElement；
导入javax.xml.stream.events.XMLEvent；
导入javax.xml.transform.Transformer；
导入javax.xml.transform.TransformerConfiguration异常；
导入javax.xml.transform.TransformerException；
导入javax.xml.transform.TransformerFactory；
导入javax.xml.transform.dom.DOMSource；
导入javax.xml.transform.stream.StreamResult；
导入org.w3c.dom.Document；
导入org.xml.sax.InputSource；
导入org.xml.sax.SAXException；
公共类吸顶器{
私有最终集路径；
私有最终XMLInputFactory inputFactory；
私有最终XMLOutputFactory输出工厂；
私人最终文件建筑商工厂文件建筑商工厂；
私有最终堆栈activeStack=新堆栈（）；
<?xml version="1.0" encoding="UTF-8"?>
<root>
    <one>
        <two>this is text</two>
        look, I can even handle mixed!
    </one>
    ... not sure what to do with this, though
    <two>
        <willbeignored/>
    </two>
    <three>
        <embedded>
            <and><here><we><go>
                Creative Commons Legal Code

                Attribution 3.0 Unported

                    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
                    LEGAL SERVICES. DISTRIBUTION OF THIS LICENSE DOES NOT CREATE AN
                    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
                    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
                    REGARDING THE INFORMATION PROVIDED, AND DISCLAIMS LIABILITY FOR
                    DAMAGES RESULTING FROM ITS USE.

                License

                THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE
                COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY
                COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS
                AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.

                BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE
                TO BE BOUND BY THE TERMS OF THIS LICENSE. TO THE EXTENT THIS LICENSE MAY
                BE CONSIDERED TO BE A CONTRACT, THE LICENSOR GRANTS YOU THE RIGHTS
                CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND
                CONDITIONS.
            </go></we></here></and>
        </embedded>
    </three>
</root>

package forum7998733;

import java.io.FileReader;
import javax.xml.xpath.*;
import org.w3c.dom.Node;
import org.xml.sax.InputSource;

public class XPathDemo {

    public static void main(String[] args) throws Exception {
        XPathFactory xpf = XPathFactory.newInstance();
        XPath xpath = xpf.newXPath();
        InputSource xml = new InputSource(new FileReader("BigFile.xml"));
        Node result = (Node) xpath.evaluate("/path/to/relevant/nodes", xml, XPathConstants.NODE);
        System.out.println(result);
    }

}

<statements>
   <statement account="123">
      ...stuff...
   </statement>
   <statement account="456">
      ...stuff...
   </statement>
</statements>

package forum7998733;

import java.io.FileReader;
import javax.xml.stream.*;
import javax.xml.transform.*;
import javax.xml.transform.stax.StAXSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.transform.dom.*;

public class Demo {

    public static void main(String[] args) throws Exception  {
        XMLInputFactory xif = XMLInputFactory.newInstance();
        XMLStreamReader xsr = xif.createXMLStreamReader(new FileReader("src/forum7998733/input.xml"));
        xsr.nextTag(); // Advance to statements element

        TransformerFactory tf = TransformerFactory.newInstance();
        Transformer t = tf.newTransformer();
        while(xsr.nextTag() == XMLStreamConstants.START_ELEMENT) {
            DOMResult domResult = new DOMResult();
            t.transform(new StAXSource(xsr), domResult);

            DOMSource domSource = new DOMSource(domResult.getNode());
            StreamResult streamResult = new StreamResult(System.out);
            t.transform(domSource, streamResult);
        }
    }

}

<?xml version="1.0" encoding="UTF-8" standalone="no"?><statement account="123">
      ...stuff...
   </statement><?xml version="1.0" encoding="UTF-8" standalone="no"?><statement account="456">
      ...stuff...
   </statement>

try 
        {
            /* CREATE THE PARSER  */
            XMLParser parser      = new XMLParser();
            /* CREATE THE FILTER (THIS IS A REGEX (X)PATH FILTER) */
            XMLRegexFilter filter = new XMLRegexFilter("statements/statement");
            /* CREATE THE HANDLER WHICH WILL BE CALLED WHEN A NODE IS FOUND */
            XMLHandler handler    = new XMLHandler()
            {
                public void nodeFound(StringBuilder node, XMLStackFilter withFilter)
                {
                    // DO SOMETHING WITH THE FOUND XML NODE
                    System.out.println("Node found");
                    System.out.println(node.toString());
                }
            };
            /* ATTACH THE FILTER WITH THE HANDLER */
            parser.addFilterWithHandler(filter, handler);
            /* SET THE FILE TO PARSE */
            parser.setFilePath("/path/to/bigfile.xml");
            /* RUN THE PARSER */
            parser.parse();
        } 
        catch (Exception ex) 
        {
            ex.printStackTrace();
        }

import java.io.BufferedReader;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Stack;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import javax.xml.stream.*;

/* IMPLEMENT THIS TO YOUR CLASS IN ORDER TO TO BE NOTIFIED WHEN A NODE IS FOUND*/
interface XMLNodeFoundNotifier {

    abstract void nodeFound(StringBuilder node, XMLStackFilter withFilter);
}

/* A SMALL HANDER USEFULL FOR EXPLICIT CLASS DECLARATION */
abstract class XMLHandler implements XMLNodeFoundNotifier {
}

/* INTERFACE TO WRITE YOUR OWN FILTER BASED ON THE CURRENT NODES STACK (PATH)*/
interface XMLStackFilter {

    abstract boolean isRelevant(Stack fullPath);
}

/* A VERY USEFULL FILTER USING REGEX AS THE PATH FILTER */
class XMLRegexFilter implements XMLStackFilter {

    Pattern relevantExpression;

    XMLRegexFilter(String filterRules) {
        relevantExpression = Pattern.compile(filterRules);
    }

    /* HERE WE ARE ARE ASK TO TELL IF THE CURRENT STACK (LIST OF NODES) IS RELEVANT
     * OR NOT ACCORDING TO WHAT WE WANT. RETURN TRUE IF THIS IS THE CASE */
    @Override
    public boolean isRelevant(Stack fullPath) {
        /* A POSSIBLE CLEVER WAY COULD BE TO SERIALIZE THE WHOLE PATH (INCLUDING
         * ATTRIBUTES) TO A STRING AND TO MATCH IT WITH A REGEX BEING THE FILTER
         * FOR NOW StackToString DOES NOT SERIALIZE ATTRIBUTES */
        String stackPath = XMLParser.StackToString(fullPath);
        Matcher m = relevantExpression.matcher(stackPath);
        return  m.matches();
    }
}

/* THE MAIN PARSER'S CLASS */
public class XMLParser {

    HashMap<XMLStackFilter, XMLNodeFoundNotifier> filterHandler;
    HashMap<Integer, Integer> feedingStreams;
    Stack<HashMap> currentStack;
    String filePath;

    XMLParser() {
        currentStack   = new <HashMap>Stack();
        filterHandler  = new <XMLStackFilter, XMLNodeFoundNotifier> HashMap();
        feedingStreams = new <Integer, Integer>HashMap();
    }

    public void addFilterWithHandler(XMLStackFilter f, XMLNodeFoundNotifier h) {
        filterHandler.put(f, h);
    }

    public void setFilePath(String filePath) {
        this.filePath = filePath;
    }

    /* CONVERT A STACK OF NODES TO A REGULAR PATH STRING. NOTE THAT PER DEFAULT 
     * I DID NOT ADDED THE ATTRIBUTES INTO THE PATH. UNCOMENT THE LINKS ABOVE TO
     * DO SO
     */
    public static String StackToString(Stack<HashMap> s) {
        int k = s.size();
        if (k == 0) {
            return null;
        }
        StringBuilder out = new StringBuilder();
        out.append(s.get(0).get("tag"));
        for (int x = 1; x < k; ++x) {
            HashMap node = s.get(x);
            out.append('/').append(node.get("tag"));
            /* 
            // UNCOMMENT THIS TO ADD THE ATTRIBUTES SUPPORT TO THE PATH

            ArrayList <String[]>attributes = (ArrayList)node.get("attr");
            if (attributes.size()>0)
            {
            out.append("[");
            for (int i = 0 ; i<attributes.size(); i++)
            {
            String[]keyValuePair = attributes.get(i);
            if (i>0) out.append(",");
            out.append(keyValuePair[0]);
            out.append("=\"");
            out.append(keyValuePair[1]);
            out.append("\"");
            }
            out.append("]");
            }*/
        }
        return out.toString();
    }

    /*
     * ONCE A NODE HAS BEEN SUCCESSFULLY FOUND, WE GET THE DELIMITERS OF THE FILE
     * WE THEN RETRIEVE THE DATA FROM IT.
     */
    private StringBuilder getChunk(int from, int to) throws Exception {
        int length = to - from;
        FileReader f = new FileReader(filePath);
        BufferedReader br = new BufferedReader(f);
        br.skip(from);
        char[] readb = new char[length];
        br.read(readb, 0, length);
        StringBuilder b = new StringBuilder();
        b.append(readb);
        return b;
    }
    /* TRANSFORMS AN XSR NODE TO A HASHMAP NODE'S REPRESENTATION */
    public HashMap XSRNode2HashMap(XMLStreamReader xsr) {
        HashMap h = new HashMap();
        ArrayList attributes = new ArrayList();

        for (int i = 0; i < xsr.getAttributeCount(); i++) {
            String[] s = new String[2];
            s[0] = xsr.getAttributeName(i).toString();
            s[1] = xsr.getAttributeValue(i);
            attributes.add(s);
        }

        h.put("tag", xsr.getName());
        h.put("attr", attributes);

        return h;
    }

    public void parse() throws Exception {
        FileReader f         = new FileReader(filePath);
        XMLInputFactory xif  = XMLInputFactory.newInstance();
        XMLStreamReader xsr  = xif.createXMLStreamReader(f);
        Location previousLoc = xsr.getLocation();

        while (xsr.hasNext()) {
            switch (xsr.next()) {
                case XMLStreamConstants.START_ELEMENT:
                    currentStack.add(XSRNode2HashMap(xsr));
                    for (XMLStackFilter filter : filterHandler.keySet()) {
                        if (filter.isRelevant(currentStack)) {
                            feedingStreams.put(currentStack.hashCode(), new Integer(previousLoc.getCharacterOffset()));
                        }
                    }
                    previousLoc = xsr.getLocation();
                    break;

                case XMLStreamConstants.END_ELEMENT:
                    Integer stream = null;
                    if ((stream = feedingStreams.get(currentStack.hashCode())) != null) {
                        // FIND ALL THE FILTERS RELATED TO THIS FeedingStreem AND CALL THEIR HANDLER.
                        for (XMLStackFilter filter : filterHandler.keySet()) {
                            if (filter.isRelevant(currentStack)) {
                                XMLNodeFoundNotifier h = filterHandler.get(filter);

                                StringBuilder aChunk = getChunk(stream.intValue(), xsr.getLocation().getCharacterOffset());
                                h.nodeFound(aChunk, filter);
                            }
                        }
                        feedingStreams.remove(currentStack.hashCode());
                    }
                    previousLoc = xsr.getLocation();
                    currentStack.pop();
                    break;
                default:
                    break;
            }
        }
    }
}