I am trying to read and parse an XML file inside of a ZIP archive as fast as possible. The XML file is roughly 85GB in size, so I know for a fact that I/O also plays a role here, as I need to essentially read 85GB total. However, reading 85GB on a 500 MB/s SSD, means I can do this in less than 3 minutes, if we ignore CPU, decompression, and so on.
As I cannot store the ZIP file nor XML file in memory, I've streamed the content, and it works really well. Reading and parsing 100.000 XML nodes/elements (which is roughly 450.000 lines of XML) takes about 10 seconds. Don't get me wrong: Reading 400.000+ lines of XML and parsing it in ~10 seconds is fast. But it's not as fast as I want.
Multi threading this is probably not an option, but I am interested in hearing, if it's possible to speed this up.
_models
is just a List<Model>
that I use to keep my objects in, which gets cleared once they're saved to the database.
public static async Task Main(string[] args)
{
XmlReaderSettings settings = new XmlReaderSettings();
settings.Async = true;
using (var stream = new ZipInputStream(File.OpenRead($@"C:\file.zip")))
{
// this is fine, as we know there is only 1 file in the ZIP
stream.GetNextEntry();
using (XmlReader reader = XmlReader.Create(stream, settings))
{
reader.MoveToContent();
while (await reader.ReadAsync())
{
if (reader.NodeType == XmlNodeType.Element)
{
if (reader.Name == "ns:ElementName")
{
await ExtractModel(reader);
}
}
}
}
}
// await save to database
}
private static async Task ExtractModel(XmlReader reader)
{
XElement el = XNode.ReadFrom(reader) as XElement;
var serializer = new XmlSerializer(typeof(Model));
var model = (Model)serializer.Deserialize(el.CreateReader());
_models.Add(model);
// save 100.000 models to database
if (_models.Count == 100000)
{
//await save to database
_models.Clear();
}
}
```
new XmlSerializer(typeof(Model))
each time you callExtractModel
: why not do this once at class level and then re-use the same serializer? Also, you can dovar settings = new XmlReaderSettings { Async = true };
. \$\endgroup\$ – BCdotWEB Feb 8 at 14:05XNode.ReadFrom(reader) as XElement
and so on, I can usereader.ReadSubtree()
which gave a huge speed improvement. It's about 40% faster now \$\endgroup\$ – MortenMoulder Feb 9 at 0:04Stopwatch
is not dedicated for performance measurement. Have you tried Benchmark.net? \$\endgroup\$ – Peter Csala Feb 9 at 8:08