9
\$\begingroup\$

In a project I'm working on, I need a component that is able to make a best guess about file's type based on its contents (a Stream). It doesn't need to do a particularly great job - basically just need to guess a file extension (which will later be mapped to a mime type). I can't trust any metadata - I can only act based on the content.

I opened up Bing and Googled around a bit and found this gem on magic numbers. I figured it would probably do the job well enough so decided to roll a limited implementation in C#.

Most of the time it's a straightforward "starts with" check. However, some formats (I'm looking at you JPEG and PDF) make life slightly more difficult.

E.g. JPEG starts with 0xFF, 0xD, 0xFF, 0xE0, xx, xx, 0x4A, 0x46, 0x49, 0x46, 0x00 where xx can be anything (I think).

EDIT The second byte in a jpeg is actually 0xD8 - I obviously had a copy paste error here

PDFs can have their header anywhere within the first 1024 bytes which is why it's not always a "starts with" operation.

FileType.cs

public class FileType
{
    private static readonly FileType unknown = new FileType("unknown", string.Empty, new byte?[0]);

    private readonly string name;
    private readonly string extension;
    private readonly byte?[] magicSequence;
    private readonly int maximumStartIndex;

    public string Name { get { return name; } }

    public string Extension { get { return extension; } }

    public byte?[] MagicSequence { get { return magicSequence; } }

    public int MaximumStartLocation { get { return maximumStartIndex; } }

    public static FileType Unknown { get { return unknown; } }

    public FileType(string name, string extension, byte?[] magicSequence, int maximumStartIndex = 0)
    {
        this.name = name;
        this.extension = extension;
        this.magicSequence = magicSequence;
        this.maximumStartIndex = maximumStartIndex;
    }
}

FileTypeChecker.cs

public class FileTypeChecker
{
    private static readonly IList<FileType> knownFileTypes = new List<FileType>
    {
        new FileType("Bitmap", ".bmp", new byte?[] { 0x42, 0x4d }),
        new FileType("Portable Network Graphic", ".png", new byte?[] { 0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A }),
        new FileType("JPEG", ".jpg", new byte?[] { 0xFF, /*0xD*/ 0xD8, 0xFF, 0xE0, null, null, 0x4A, 0x46, 0x49, 0x46, 0x00 }),
        new FileType("Graphics Interchange Format 87a", ".gif", new byte?[] { 0x47, 0x49, 0x46, 0x38, 0x37, 0x61 }),
        new FileType("Graphics Interchange Format 89a", ".gif", new byte?[] { 0x47, 0x49, 0x46, 0x38, 0x39, 0x61 }),
        new FileType("Portable Document Format", ".pdf", new byte?[] { 0x25, 0x50, 0x44, 0x46 }, 1019)
        // ... Potentially more in future
    }
        .OrderBy(f => f.MaximumStartLocation)
        .ThenBy(f => f.MagicSequence.Length).ToList();

    private static ByteWithWildcardComparer _wildcardComparer = new ByteWithWildcardComparer();

    public FileType GetFileType(Stream fileContent)
    {
        if (fileContent == null)
        {
            throw new ArgumentNullException("fileContent");
        }
        if (!fileContent.CanRead || (fileContent.Position != 0 && !fileContent.CanSeek))
        {
            throw new ArgumentException("File contents must be a readable stream", "fileContent");
        }
        if (fileContent.Position != 0)
        {
            fileContent.Seek(0, SeekOrigin.Begin);
        }
        var startOfFile = new byte[1024];
        fileContent.Read(startOfFile, 0, startOfFile.Length);
        var result = knownFileTypes.FirstOrDefault(f => StartOfFileContainsFileType(f, startOfFile));

        return result ?? FileType.Unknown;
    }

    public IEnumerable<FileType> GetFileTypes(Stream fileContent)
    {
        if (fileContent == null)
        {
            throw new ArgumentNullException("fileContent");
        }
        if (!fileContent.CanRead || (fileContent.Position != 0 && !fileContent.CanSeek))
        {
            throw new ArgumentException("File contents must be a readable stream", "fileContent");
        }
        if (fileContent.Position != 0)
        {
            fileContent.Seek(0, SeekOrigin.Begin);
        }
        var startOfFile = new byte[1024];
        fileContent.Read(startOfFile, 0, startOfFile.Length);
        return knownFileTypes.Where(f => StartOfFileContainsFileType(f, startOfFile));
    }

    private static bool StartOfFileContainsFileType(FileType fileType, byte[] startOfFile)
    {
        var counter = 0;
        do
        {
            if (startOfFile.Skip(counter)
                    .Take(fileType.MagicSequence.Length)
                    .Cast<byte?>()
                    .SequenceEqual(fileType.MagicSequence, _wildcardComparer))
            {
                return true;
            }
        }
        while (++counter <= fileType.MaximumStartLocation);
        return false;
    }

    private class ByteWithWildcardComparer : IEqualityComparer<byte?>
    {
        public bool Equals(byte? x, byte? y)
        {
            // null is a wildcard - it matches the other regardless of value.
            return x == null || y == null || x.GetValueOrDefault() == y.GetValueOrDefault();
        }

        public int GetHashCode(byte? obj)
        {
            return obj.GetValueOrDefault().GetHashCode();
        }
    }
}

Usage (Console app)

var checker = new FileTypeChecker();
using (var fileStream = File.OpenRead(@"C:\someFile"))
{
    var result = checker.GetFileType(fileStream);
    Console.WriteLine("FileType: {0}", result.Name);
}

As you can see, I've modelled the MagicSequence as an array of nullable bytes so that I can use a custom IEqualityComparer<byte?> to make null a wildcard (i.e. match anything).

I've tested the above out on a few different files and it seems to work well (although the PDF check slows things down massively).

I don't think this is quite right because I can't ask whether a file is a PDF without asking what type a file is and then comparing the extension/name to what I'm expecting.

Looking for tips on structure (and all general points) but particularly so I can easily do stuff like:

  • What's the best guess type for this file?
  • What types could this file be?
  • Is this a PDF?
  • Is this file one of these allowed types?
\$\endgroup\$
  • \$\begingroup\$ Why not use one of the many tools created specifically for this? E.g. on a Unix based system file x in a terminal is an advanced, powerful and fast tool that will do a much better job than your code to work out the file type of x \$\endgroup\$ – texasflood Mar 27 '15 at 22:02
  • \$\begingroup\$ @texasflood - 1. I thought it would be a fun thing to write. 2. it's only going to be used as a quick sanity check in an application. \$\endgroup\$ – RobH Mar 28 '15 at 13:08
7
\$\begingroup\$

The more I think about it, StartOfFileContainsFileType is doing too many things: exact matching, fuzzy matching, and matching within a range.

Here's one possible solution. Create an abstract class (or interface) FileTypeMatcher. I went with an abstract class just to keep all the validation in one place.

public abstract class FileTypeMatcher
{
    public bool Matches(Stream stream, bool resetPosition = true)
    {
        if (stream == null)
        {
            throw new ArgumentNullException("stream");
        }
        if (!stream.CanRead || (stream.Position != 0 && !stream.CanSeek))
        {
            throw new ArgumentException("File contents must be a readable stream", "stream");
        }
        if (stream.Position != 0 && resetPosition)
        {
            stream.Position = 0;
        }

        return MatchesPrivate(stream);
    }

    protected abstract bool MatchesPrivate(Stream stream);
}

Then we need a few classes that derive from this.

public class ExactFileTypeMatcher : FileTypeMatcher
{
    private readonly byte[] bytes;

    public ExactFileTypeMatcher(IEnumerable<byte> bytes)
    {
        this.bytes = bytes.ToArray();
    }

    protected override bool MatchesPrivate(Stream stream)
    {
        foreach (var b in this.bytes)
        {
            if (stream.ReadByte() != b)
            {
                return false;
            }
        }

        return true;
    }
}

public class FuzzyFileTypeMatcher : FileTypeMatcher
{
    private readonly byte?[] bytes;

    public FuzzyFileTypeMatcher(IEnumerable<byte?> bytes)
    {
        this.bytes = bytes.ToArray();
    }

    protected override bool MatchesPrivate(Stream stream)
    {
        foreach (var b in this.bytes)
        {
            var c = stream.ReadByte();
            if (c == -1 || (b.HasValue && c != b.Value))
            {
                return false;
            }
        }

        return true;
    }
}

public class RangeFileTypeMatcher : FileTypeMatcher
{
    private readonly FileTypeMatcher matcher;

    private readonly int maximumStartLocation;

    public RangeFileTypeMatcher(FileTypeMatcher matcher, int maximumStartLocation)
    {
        this.matcher = matcher;
        this.maximumStartLocation = maximumStartLocation;
    }

    protected override bool MatchesPrivate(Stream stream)
    {
        for (var i = 0; i < this.maximumStartLocation; i++)
        {
            // Might want to check if i >= stream.Length.
            stream.Position = i;
            if (matcher.Matches(stream, resetPosition: false))
            {
                return true;
            }
        }

        return false;
    }
}

(I've been a bit fast and loose with this, there might be bugs present.)

With all that in place, we can rewrite FileType to delegate to FileTypeMatcher

public class FileType
{
    private static readonly FileType unknown = new FileType("unknown", string.Empty, null);

    private readonly string name;

    private readonly string extension;

    private readonly FileTypeMatcher fileTypeMatcher;

    public string Name { get { return name; } }

    public string Extension { get { return extension; } }

    public static FileType Unknown { get { return unknown; } }

    public FileType(string name, string extension, FileTypeMatcher matcher)
    {
        this.name = name;
        this.extension = extension;
        this.fileTypeMatcher = matcher;
    }

    public bool Matches(Stream stream)
    {
        return this.fileTypeMatcher == null || this.fileTypeMatcher.Matches(stream);
    }
}

And finally FileTypeChecker to delegate to FileType

public class FileTypeChecker
{
    private static readonly IList<FileType> knownFileTypes = new List<FileType>
    {
        new FileType("Bitmap", ".bmp", new ExactFileTypeMatcher(new byte[] {0x42, 0x4d})),
        new FileType("Portable Network Graphic", ".png",
            new ExactFileTypeMatcher(new byte[] {0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A})),
        new FileType("JPEG", ".jpg",
            new FuzzyFileTypeMatcher(new byte?[] {0xFF, 0xD, 0xFF, 0xE0, null, null, 0x4A, 0x46, 0x49, 0x46, 0x00})),
        new FileType("Graphics Interchange Format 87a", ".gif",
            new ExactFileTypeMatcher(new byte[] {0x47, 0x49, 0x46, 0x38, 0x37, 0x61})),
        new FileType("Graphics Interchange Format 89a", ".gif",
            new ExactFileTypeMatcher(new byte[] {0x47, 0x49, 0x46, 0x38, 0x39, 0x61})),
        new FileType("Portable Document Format", ".pdf", new RangeFileTypeMatcher(new ExactFileTypeMatcher(new byte[] { 0x25, 0x50, 0x44, 0x46 }), 1019))
        // ... Potentially more in future
    };

    public FileType GetFileType(Stream fileContent)
    {
        return GetFileTypes(fileContent).FirstOrDefault() ?? FileType.Unknown;
    }

    public IEnumerable<FileType> GetFileTypes(Stream stream)
    {
        return knownFileTypes.Where(fileType => fileType.Matches(stream));
    }
}

So what has all this done for us? For one, I think the code is more composable and amenable to change. But in practical terms, I set up some unit tests to compare the two solutions on a PDF and a BMP. Each test was run 1000 times, and the cumulative times were

        Original      This
PDF       0.02s       0.01s
BMP       3.21s       0.11s

Of course, take this with a grain of salt and be sure to do your own measurements. I've put the code up (including the tests and test files) on GitHub if you want to play around with it.

\$\endgroup\$
  • \$\begingroup\$ While implementing these ideas I found an error with my JPEG magic number - I had the second byte wrong. I've done my own measurements and can confirm (even with small changes and more LINQ) that your approach is significantly faster. \$\endgroup\$ – RobH Mar 27 '15 at 9:38
  • \$\begingroup\$ I see you take an IEnumerable<>, convert it ToArray(), but then do a foreach on it. Is this just to force a deep copy? \$\endgroup\$ – Snowbody Mar 27 '15 at 16:41
  • \$\begingroup\$ This MatchesPrivate is $O(n^2)$. By building a state machine you could get an $O(m+n)$ algorithm. \$\endgroup\$ – Snowbody Mar 27 '15 at 16:42
  • \$\begingroup\$ Ha, nice. this is a nice modular solution. I'd personally do a _bytes = bytes as byte[] ?? bytes.ToArray(); \$\endgroup\$ – apieceoffruit Mar 27 '15 at 16:49
6
\$\begingroup\$

The implementations of GetFileType(Stream) and GetFileTypes(Stream) have so much in common that you could write the former in terms of the latter.

public FileType GetFileType(Stream fileContent)
{
    return GetFileTypes(fileContent).FirstOrDefault() ?? FileType.Unknown;
}

Also note that with Stream.Read()

An implementation is free to return fewer bytes than requested even if the end of the stream has not been reached.

So you should keep reading until you reach the end of the stream, or 1024 bytes have been read.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.