Ruby code to identify 2-byte characters

Question

I just wrote this fuzzy snippet to identify any 2-byte character in my data file which assumes only 1-byte characters (ANSI). Please review and suggest me any better solution!

File.open('data.txt', 'r') do |f|
    line_number = 0
    while line = f.gets
        line_number = line_number + 1
        line.each_byte do |c|
            if c & 0xff80 != 0
                puts 'A non-ascii character is found at line ' + line_number.to_s
                break
            end
        end
    end
end

Answer 1

File.open('data.txt', 'r') do |f| ... end

Good job. This is indeed the preferred way to read a file in Ruby. You get to read the file as a stream, and the file gets closed automatically. One thing to note is that the r flag is the default and can be left out. It is fine to leave it there for clarity, however.

Don't forget to set the correct encoding. If the real encoding is compatible with low-ASCII, you can assume ASCII-8BIT but it will treat characters in multibyte encodings as multiple characters. See later, however. You can set a default one, or you can do it while opening the file with :external_encoding => ....

If you strive for simplicity rather than speed (or if the file is small enough to be read all at once), you can also load the file into memory at once, then iterate over it:

f = IO.read('data.txt')

---

line_number = 0
while line = f.gets
    line_number = line_number + 1
    ...
end

Apart from the fact that assignment within conditional is frowned upon in many languages because it can be confused with accidental assignment instead of comparison,

File inherits several iterators from IO (don't forget to check for inherited methods - from superclasses AND modules) when looking up documentation). One of them IO#each_line, formerly IO#lines. For the line number, you can use IO#lineno. You can also use f.each_line.with_index{|line, index|...}, but that is zero-based.

---

line.each_byte do |c|
    if c & 0xff80 != 0
        puts 'A non-ascii character is found at line ' + line_number.to_s
        break
    end
end

I think you're looking for String#each_codepoint instead of String#each_byte. A UTF-16 non-ASCII character could well be composed of two ASCII bytes. UTF-8 non-ASCII characters are readily composed of multiple bytes, so you'd be getting multiple reports per character. If you assume a single-byte encoding, both will do (and each_byte is more explicit), but ...

If your intention is to test single-byte encodings, then the mask 0x80 will suffice. If your intention is to support any encoding of the Unicode character set, then the mask 0xff80 will not suffice as Unicode code points are 20 bits long. Some rare characters actually lie in the U+10000 .. U+1007F region. Perhaps the surest test is to use > 0x7F

Better yet, realise you are working with strings, and use string methods to manipulate them.

Even if you use each_byte, you still have to specify the external encoding. If the external encoding is set to UTF-8 but the source file isn't, you'll get an exception. That being said, you could set a 7-bit encoding (US-ASCII) and rely on the exception being thrown, but that won't get you the line number the character occurs on.

---

While string concatenation is the way to go in languages that lack string interpolation, string interpolation is the preferred way in Ruby. The nice thing is that it performs automatic string conversion where concatenation requires you to convert manually. Just don't use huge blocks or nested interpolation inside unless you care for character count rather than readability. Compute the part first, then interpolate it in.

One last thing to note is that you are not very consistent in the use of one-letter / descriptive variable names. Descriptive names never hurt, and single-letter variables should only be used if their scope is small enough (or if the single letter is descriptive per se).

This is how I would modify your code:

File.open('data.txt') do |file|
    file.each_line.with_index do |line, index|
        if /[^\x00-\x7F]/ =~ line
            puts "A non-ascii character is found at line #{index + 1}"
            break
        end
    end
end

If you are not afraid of perlisms,

File.open('data.txt') do |f|
    while f.gets
        if /[^\x00-\x7F]/
            puts "A non-ascii character is found at line #{$.}"
            break
        end
    end
end