2
\$\begingroup\$

I am trying to read and parse an XML file inside of a ZIP archive as fast as possible. The XML file is roughly 85GB in size, so I know for a fact that I/O also plays a role here, as I need to essentially read 85GB total. However, reading 85GB on a 500 MB/s SSD, means I can do this in less than 3 minutes, if we ignore CPU, decompression, and so on.

As I cannot store the ZIP file nor XML file in memory, I've streamed the content, and it works really well. Reading and parsing 100.000 XML nodes/elements (which is roughly 450.000 lines of XML) takes about 10 seconds. Don't get me wrong: Reading 400.000+ lines of XML and parsing it in ~10 seconds is fast. But it's not as fast as I want.

Multi threading this is probably not an option, but I am interested in hearing, if it's possible to speed this up.

_models is just a List<Model> that I use to keep my objects in, which gets cleared once they're saved to the database.

public static async Task Main(string[] args)
{
    XmlReaderSettings settings = new XmlReaderSettings();
    settings.Async = true;

    using (var stream = new ZipInputStream(File.OpenRead($@"C:\file.zip")))
    {
        // this is fine, as we know there is only 1 file in the ZIP
        stream.GetNextEntry();

        using (XmlReader reader = XmlReader.Create(stream, settings))
        {
            reader.MoveToContent();

            while (await reader.ReadAsync())
            {
                if (reader.NodeType == XmlNodeType.Element)
                {
                    if (reader.Name == "ns:ElementName")
                    {
                        await ExtractModel(reader);
                    }
                }
            }
        }
    }

    // await save to database
}

private static async Task ExtractModel(XmlReader reader)
{
    XElement el = XNode.ReadFrom(reader) as XElement;

    var serializer = new XmlSerializer(typeof(Model));

    var model = (Model)serializer.Deserialize(el.CreateReader());

    _models.Add(model);

    // save 100.000 models to database
    if (_models.Count == 100000)
    {
        //await save to database
        _models.Clear();
    }
}
```
\$\endgroup\$
13
  • \$\begingroup\$ You do a new XmlSerializer(typeof(Model)) each time you call ExtractModel: why not do this once at class level and then re-use the same serializer? Also, you can do var settings = new XmlReaderSettings { Async = true };. \$\endgroup\$ – BCdotWEB Feb 8 at 14:05
  • \$\begingroup\$ @BCdotWEB The performance gained from not newing up a new XmlSerializer was next to nothing. Even if I do that, all of the checks and calls inside of Main (even the one to ExtractModel) says < 1 milliseconds. So nothing is "taking too long", it's simply slow because of the sheer amount of data. \$\endgroup\$ – MortenMoulder Feb 8 at 14:33
  • \$\begingroup\$ @MortenMoulder Have you used some sort of profiling tool like CodeTrack to know where does your code spend most of its time? If so, could you please share with us the results? Please also share with us the expected performance requirement. But it's not as fast as I want. << It is not a requirement. \$\endgroup\$ – Peter Csala Feb 8 at 15:10
  • 1
    \$\begingroup\$ @aepot Thanks for the reply. I've moved to array instead of List, and it seems like it has improved the speed a bit. However, instead of doing the XNode.ReadFrom(reader) as XElement and so on, I can use reader.ReadSubtree() which gave a huge speed improvement. It's about 40% faster now \$\endgroup\$ – MortenMoulder Feb 9 at 0:04
  • 1
    \$\begingroup\$ @MortenMoulder Why don't use proper dedicated tools to profile and measure. Stopwatch is not dedicated for performance measurement. Have you tried Benchmark.net? \$\endgroup\$ – Peter Csala Feb 9 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.