3
\$\begingroup\$

I use a lot <cctype> functions, but they're not very type safe and and calling them with a negative char is (theoretically) undefined behavior, so my code is cluttered with static_casts like:

[](const char ch){ return static_cast<char>(std::tolower(static_cast<unsigned char>(ch))); });

So I started to write a simple wrapper, then I wanted to have a little more control and one thing led to another I finished to replace the thing entirely. I know it's not good, but the damage is already done and I'm actually quite happy using it.

I'm posting here the code to get some feedback and advices from more experience developers in order to improve and consolidate a little bit.

godbolt github

#pragma once
//  ---------------------------------------------
//  ASCII encoding predicates and facilities
//  ---------------------------------------------
//  #include "ascii_predicates.hpp" // ascii::is_*
//  ---------------------------------------------
#include <concepts> // std::same_as<>
#include <cstdint> // std::uint16_t


//:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
namespace ascii //:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
{

namespace details
{   // The standard <cctype> behaviour:
    // |         |                           |is   |is   |is   |is   |is   |is   |is   |is   |is   |is   |is   |is    |
    // | ASCII   | characters                |cntrl|print|graph|space|blank|punct|alnum|alpha|upper|lower|digit|xdigit|
    // |---------|---------------------------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|------|
    // | 0–8     | control codes (NUL, etc.) |  X  |     |     |     |     |     |     |     |     |     |     |      |
    // | 9       | tab (\t)                  |  X  |     |     |  X  |  X  |     |     |     |     |     |     |      |
    // | 10–13   | whitespaces (\n,\v,\f,\r) |  X  |     |     |  X  |     |     |     |     |     |     |     |      |
    // | 14–31   | control codes             |  X  |     |     |     |     |     |     |     |     |     |     |      |
    // | 32      | space                     |     |  X  |     |  X  |  X  |     |     |     |     |     |     |      |
    // | 33–47   | !"#$%&amp;'()*+,-./       |     |  X  |  X  |     |     |  X  |     |     |     |     |     |      |
    // | 48–57   | 0123456789                |     |  X  |  X  |     |     |     |  X  |     |     |     |  X  |  X   |
    // | 58–64   | :;&lt;=&gt;?@             |     |  X  |  X  |     |     |  X  |     |     |     |     |     |      |
    // | 65–70   | ABCDEF                    |     |  X  |  X  |     |     |     |  X  |  X  |  X  |     |     |  X   |
    // | 71–90   | GHIJKLMNOPQRSTUVWXYZ      |     |  X  |  X  |     |     |     |  X  |  X  |  X  |     |     |      |
    // | 91–96   | [\]^_`                    |     |  X  |  X  |     |     |  X  |     |     |     |     |     |      |
    // | 97–102  | abcdef                    |     |  X  |  X  |     |     |     |  X  |  X  |     |  X  |     |  X   |
    // | 103–122 | ghijklmnopqrstuvwxyz      |     |  X  |  X  |     |     |     |  X  |  X  |     |  X  |     |      |
    // | 123–126 | {|}~                      |     |  X  |  X  |     |     |  X  |     |     |     |     |     |      |
    // | 127     | backspace character (DEL) |  X  |     |     |     |     |     |     |     |     |     |     |      |

    using table_entry_t = std::uint16_t;
    enum mask_t : table_entry_t
       {
        ISLOWER = 0x0001u, // std::islower
        ISUPPER = 0x0002u, // std::isupper
        ISSPACE = 0x0004u, // std::isspace
        ISBLANK = 0x0008u, // std::isspace except '\n' (not equiv to std::isblank)

        ISALPHA = 0x0010u, // std::isalpha
        ISALNUM = 0x0020u, // std::isalnum
        ISDIGIT = 0x0040u, // std::isdigit
        ISXDIGI = 0x0080u, // std::isxdigit

        ISPUNCT = 0x0100u, // std::ispunct
        ISIDENT = 0x0200u, // std::isalnum plus '_'

        ISCNTRL = 0x1000u, // std::iscntrl
        ISGRAPH = 0x2000u, // std::isgraph
        ISPRINT = 0x4000u  // std::isprint
       };

    static_assert( sizeof(unsigned char)==1 ); // Should I check CHAR_BIT?
    static constexpr table_entry_t ascii_lookup_table[256] = // Lookup table generation: https://gcc.godbolt.org/z/4G6nxh8vh
      { 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x100C, 0x1004, 0x100C, 0x100C, 0x100C, 0x1000, 0x1000,
        0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000, 0x1000,
        0x400C, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100,
        0x62E0, 0x62E0, 0x62E0, 0x62E0, 0x62E0, 0x62E0, 0x62E0, 0x62E0, 0x62E0, 0x62E0, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100, 0x6100,
        0x6100, 0x62B2, 0x62B2, 0x62B2, 0x62B2, 0x62B2, 0x62B2, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232,
        0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6232, 0x6100, 0x6100, 0x6100, 0x6100, 0x6300,
        0x6100, 0x62B1, 0x62B1, 0x62B1, 0x62B1, 0x62B1, 0x62B1, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231,
        0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6231, 0x6100, 0x6100, 0x6100, 0x6100, 0x1000,
        0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0,
        0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0,
        0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0,
        0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0,
        0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0,
        0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0,
        0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0,
        0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0   , 0x0 };


    [[nodiscard]] inline constexpr bool check(const unsigned char ch, const mask_t mask){ return ascii_lookup_table[ch] & mask; }
    [[nodiscard]] inline constexpr bool check(const char ch, const mask_t mask){ return check(static_cast<unsigned char>(ch), mask); }
    [[nodiscard]] inline constexpr bool check(const char32_t cp, const mask_t mask)
       {
        // Good enough for parsing code where non ASCII codepoints are just in strings or comments
        if( cp<U'\x80' ) [[likely]]
           {
            return check(static_cast<unsigned char>(cp), mask);
           }
        return false;
       }

    // Facility for case conversion
    template<bool SET> [[nodiscard]] inline constexpr char set_case_bit(char c) noexcept
       {
        static constexpr char case_bit = '\x20';
        if constexpr (SET) c |= case_bit;
        else               c &= ~case_bit;
        return c;
       }
}

// Supporting the following character types:
template<typename T> concept CharLike = std::same_as<T, char> or std::same_as<T, unsigned char> or std::same_as<T, char32_t>;

// Basic predicates
template<CharLike Chr> [[nodiscard]] constexpr bool is_lower(const Chr c) noexcept { return details::check(c, details::ISLOWER); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_upper(const Chr c) noexcept { return details::check(c, details::ISUPPER); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_space(const Chr c) noexcept { return details::check(c, details::ISSPACE); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_blank(const Chr c) noexcept { return details::check(c, details::ISBLANK); } // not equiv to std::isblank
template<CharLike Chr> [[nodiscard]] constexpr bool is_alpha(const Chr c) noexcept { return details::check(c, details::ISALPHA); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_alnum(const Chr c) noexcept { return details::check(c, details::ISALNUM); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_digit(const Chr c) noexcept { return details::check(c, details::ISDIGIT); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_xdigi(const Chr c) noexcept { return details::check(c, details::ISXDIGI); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_punct(const Chr c) noexcept { return details::check(c, details::ISPUNCT); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_ident(const Chr c) noexcept { return details::check(c, details::ISIDENT); } // std::isalnum or '_'
template<CharLike Chr> [[nodiscard]] constexpr bool is_cntrl(const Chr c) noexcept { return details::check(c, details::ISCNTRL); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_graph(const Chr c) noexcept { return details::check(c, details::ISGRAPH); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_print(const Chr c) noexcept { return details::check(c, details::ISPRINT); }
// Other predicates
template<CharLike Chr> [[nodiscard]] constexpr bool is_endline(const Chr c) noexcept { return c==static_cast<Chr>('\n'); }
template<CharLike Chr> [[nodiscard]] constexpr bool is_always_false(const Chr) noexcept { return false; }
// Composite predicates
template<CharLike Chr> [[nodiscard]] constexpr bool is_space_or_punct(const Chr c) noexcept { return details::check(c, details::ISSPACE | details::ISPUNCT); }


//---------------------------------------------------------------------------
// Non-type parameterized template predicates

template<CharLike auto CH>
[[nodiscard]] constexpr bool is(const decltype(CH) ch) noexcept
   {
    return ch==CH;
   }

template<CharLike auto CH1, decltype(CH1)... CHS>
[[nodiscard]] constexpr bool is_any_of(const decltype(CH1) ch)
   {
    return ch==CH1 or ((ch==CHS) or ...);
   }

template<CharLike auto CH1, decltype(CH1)... CHS>
[[nodiscard]] constexpr bool is_none_of(const decltype(CH1) ch)
   {
    return ch!=CH1 and ((ch!=CHS) and ...);
   }


// Some examples of composite predicates

template<CharLike auto CH1, decltype(CH1)... CHS>
[[nodiscard]] constexpr bool is_space_or_any_of(const decltype(CH1) ch)
   {
    return is_space(ch) or is_any_of<CH1, CHS ...>(ch);
   }

template<CharLike auto CH1, decltype(CH1)... CHS>
[[nodiscard]] constexpr bool is_alnum_or_any_of(const decltype(CH1) ch)
   {
    return is_alnum(ch) or is_any_of<CH1, CHS ...>(ch);
   }

template<CharLike auto CH1, decltype(CH1)... CHS>
[[nodiscard]] constexpr bool is_punct_and_none_of(const decltype(CH1) ch)
   {
    return is_punct(ch) and is_none_of<CH1, CHS ...>(ch);
   }



//---------------------------------------------------------------------------
// Case conversion
[[nodiscard]] constexpr char to_lower(char c) noexcept { if(is_upper(c)) c = details::set_case_bit<true>(c); return c; }
[[nodiscard]] constexpr char to_upper(char c) noexcept { if(is_lower(c)) c = details::set_case_bit<false>(c); return c; }

}//::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

\$\endgroup\$
4
  • 1
    \$\begingroup\$ It seems a step backward to support only ASCII. You might be better off moving to the the <locale> functions, which are templated to accept different character types. \$\endgroup\$ Jan 30 at 12:20
  • \$\begingroup\$ @TobySpeight Well the aim of this little header is very low and remains confined in the ASCII realm. In my application domain I find myself dealing with unicode files where the parts I need to parse is ASCII (markup, keywords and numbers) and the bigger codepoints found in names, text blobs or comments don't need manipulation. \$\endgroup\$
    – MatG
    Jan 30 at 13:06
  • \$\begingroup\$ I’d seriously consider parsing the character property tables from the Unicode consortium and generating your source file from that. Then, supporting any other code page is just a matter of looking up which Unicode character each codepoint maps to. You could memo-ize this. \$\endgroup\$
    – Davislor
    Feb 2 at 21:24
  • \$\begingroup\$ @Davislor I think codepages and related single bites encodings should finally be dropped, while supporting non ascii codepoints is a big task concerning cultures and locales: I hope that (c++29?) standard library will have unicode facilities. However my own personal needs is to parse source files whose important parts (markup/keywords) have codepoints<0x80, so these ascii predicates are all I need. \$\endgroup\$
    – MatG
    Feb 3 at 8:43

1 Answer 1

6
\$\begingroup\$

Consider using the versions from <locale>

As Toby Speight already mentioned, all these C functions have a much nicer C++ counterpart in the <locale> header. These are templated on a character type, just like your code.

A drawback of these functions is that you always need to provide a locale to every call, and that this locale has to be checked every call, which means performance is going to be much worse, which might be a problem if you are calling them a lot.

What is the point of CharLike if you are only handling US-ASCII?

Your code only handles US-ASCII characters. It doesn't make sense to handle other types than char then; if you feed it wide character strings or Unicode, it's just going to return unexpected results. I think it would be better to get a compile error in that case.

You are also inconsistent; why are the is_something() functions templated on CharLike, but your to_upper() and to_lower() are not?

Why doesn't CharLike accept wchar_t and other standard character types?

You limit CharLike to char, unsigned char and char32_t, but there is also wchar_t, char8_t and char16_t.

About your parameter packs

I see why you made your "non-type parameterized template predicates" take a CH1 and CHS; you want to ensure all the parameters have the same type. However, some conversions are still allowed. For example, the following will compile just fine without any warning:

auto result = is_any_of<U'a', L'b', u8'c', 'd'>(42);

To fix this, don't restrict any of the (template) parameter types where they are declared, and instead use a requires clause to ensure all types are the same, like so:

template<CharLike auto... CHS, typename CH>
requires (std::same_as<CH, decltype(CHS)> and ...)
[[nodiscard]] constexpr bool is_any_of(const CH ch)
{
    return ((ch == CHS) or ...);
}

Unnecessary use of non-type template parameters

There is no reason to pass SET as a template parameter. You can just pass it as a regular parameter. The same goes for your other predicates. All these functions are constexpr, so everything can be evaluated at compile time anyway.

\$\endgroup\$
4
  • \$\begingroup\$ Thanks for pointing out the implicit conversions of non type template parameters! I used them over normal parameters to ensure a compile time evaluation, a constexpr does not guarantee that, although I'm not sure if this enhances more optimizations or not. \$\endgroup\$
    – MatG
    Jan 30 at 15:52
  • \$\begingroup\$ Regarding locale I'm not interested to handle non ASCII codepoints (I'm waiting a standard C++ unicode library for that), the namespace helps to remind this. I've added support for char32_t because that's the type I use when parsing unicode text (and when parsing say an xml I use a lot those ascii:: predicates for the markup); the other char types are at best useless (biased opinion, won't discuss it here). \$\endgroup\$
    – MatG
    Jan 30 at 15:53
  • \$\begingroup\$ I have added to_lower()/to_upper() for convenience but I shouldn't have because could deceive the user despite the ascii:: namespace. To mitigate this I defined those function just with a char type. Case conversion should have been outside the scope of this little header. \$\endgroup\$
    – MatG
    Jan 30 at 15:53
  • 1
    \$\begingroup\$ You are right about constexpr, but actually nothing in the language prevents the compiler from passing and evaluating non-type template parameters at runtime either. \$\endgroup\$
    – G. Sliepen
    Jan 30 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.