Questions tagged [utf-8]

UTF-8 (Unicode Transformation Format, 8 bits) is a character encoding that describes each Unicode code point using a byte sequence of one to six bytes. It is backwards-compatible with ASCII while still supporting representation of all Unicode code points.

Filter by
Sorted by
Tagged with
2 votes
2 answers
84 views

Wielding .NET masterfully to encode non-alphanumeric characters into utf-8 hex representation

I have these two methods that work, but I also hate because they almost certainly can be improved. I'm hoping to gain some guidance from others who are more knowledgable of .NET's offering for ...
user avatar
  • 23
-1 votes
1 answer
227 views

I wrote a header file to write German umlauts in a textfile properly

This function is about the fact that a std::wstring was used in another cpp file in order to be able to read strings with German umlauts from the console. Since it is difficult to get wstrings into a ...
user avatar
  • 177
2 votes
1 answer
451 views

C++ UTF-8 decoder

While writing simple text rendering I found a lack of utf-8 decoders. Most decoders I found required allocating enough space for decoded string. In worse case that would mean that the decoded string ...
user avatar
  • 145
2 votes
1 answer
35 views

Validator and Sanitizer for HTML 5 attribute regex according to current HTML living standard

According to https://html.spec.whatwg.org/multipage/syntax.html#attributes-2 an HTML 5 attribute name is defined like this: Attribute names must consist of one or more characters other than controls, ...
user avatar
  • 137
2 votes
0 answers
88 views

Find the UTF-8 Length of a given codepoint

A codepoint in UTF-8 can be represented by one to four bytes. Given a codepoint, I need to determine the length (in bytes) of the codepoint if it were represented in UTF-8. For this, I've written the ...
user avatar
2 votes
1 answer
130 views

A C++ function to read Code Points from an UTF-8 Stream

I've written a function that reads and returns one UTF-8 code point from an istream. I am wondering if the code is efficient or if there are some obvious problems with the implementation. ...
user avatar
1 vote
1 answer
373 views

The conversion from UTF-16 to UTF-8

I have created a function that converts from UTF-16 to UTF-8. This function converts from UTF-16 to codepoint firstly, then from codepoint to UTF-8. ...
user avatar
  • 129
20 votes
6 answers
4k views

Can I slim this UTF8-encoding program?

Below is my entire program. You can read what it does thanks to the comments and specifications in particular. My question is: can it be improved? Would it be possible, for example, to avoid writing ...
user avatar
3 votes
1 answer
73 views

Save on typing while using UTF8 encoding

Typing in something like Encoding.UTF8.GetString(...) and Encoding.UTF8.GetBytes(...) everywhere in your code could be ...
user avatar
  • 6,123
8 votes
1 answer
98 views

Checking whether a string fragment could be part of a longer UTF-8 string

Although UTF-8 validation is a common task, I'm trying to solve a slightly different task; given a string of bytes, work out whether it could potentially be a fragment of a valid UTF-8 string. That's ...
user avatar
  • 181
8 votes
1 answer
163 views

myUTF-8 small lib (validate UTF-8, guess language, count chars)

I'm new to C language and never got my self into the details of UTF-8, and after reading some articles about it, I wanted to try and play with UTF-8 with C language for both fun and practicing ...
user avatar
4 votes
1 answer
9k views

Convert to UTF-8 all files in a directory

1. Summary I can't find, how I can to refactor multiple with open for one file. 2. Expected behavior of program Program detect encoding for each file in the ...
user avatar
9 votes
5 answers
4k views

Convert UTF8 string to UTF32 string in C

I'm doing some recreational programming in C (after spending some time in C++, but professionally using only PHP/JavaScript). I wrote a UTF8 to UTF32 converter and just wanted to know if I made some ...
user avatar
  • 193
3 votes
1 answer
2k views

Decode string with hex character codes to UTF-8 characters

From a system we receive messages that contain codes that represent UTF-8 characters. For example : var str="Test =64 =C2=AE =E1=A6=92 test"; To decode these ...
user avatar
  • 133
5 votes
1 answer
235 views

Convert a mix of latin 1 and UTF-8 to proper UTF-8

The following program takes an arbitrary byte sequence as input and outputs well-formed UTF-8. All UTF-8 sequences from the input are copied unmodified. All other bytes are assumed to be encoded in ...
user avatar
7 votes
1 answer
281 views

Wrapper to enhance a program to translate from LTL formulae to Büchi automata

The following is a script that is supposed to act as a wrapper for ltl2ba, a program which transforms LTL formulas into Buchi Automatons encoded in the Promela language. The wrapper adds a limited set ...
user avatar
5 votes
1 answer
3k views

Mapping a CSV with UTF-8 encoding to JSON in Python

I'm not a programmer, just a guy who needs to solve particular things using code. I needed to create a JSON from a CSV with predefined headers. I'm downloading an attachment from JIRA (REST API), ...
user avatar
10 votes
3 answers
4k views

UTF-8 Validation

I've recently solved the "UTF-8 Validation" LeetCode problem: A character in UTF8 can be from 1 to 4 bytes long, subjected to the following rules: For 1-byte character, the first bit is a 0, ...
user avatar
  • 17k
3 votes
0 answers
771 views

Convert Unicode text to font glyphs for display on Android devices

Prior to Android 6.0, Mongolian text is not supported on Android devices. And as far as I know, OpenType smart font rendering is still not supported. Therefore, in all my Mongolian apps I use a ...
user avatar
  • 405
4 votes
2 answers
652 views

Encode unicode codepoints to UTF-8 manually

I want to encode unicode codepoints to UTF-8 manually. I wrote the following C# code. I tested it with some cases I know, but I would like to know if it's correct for all inputs. I know that Unicode ...
user avatar
  • 235
2 votes
2 answers
310 views

Parsing and saving JSON result from GoEuro API

I am new to python so I decided to write a script that parses a json array and writes data in a file. I wrote the code that does that but I think it can be done way better, So I was hoping to get ...
user avatar
5 votes
1 answer
122 views

UTF8 Codepoint decode and length v.2

This is the complete rewrite of a function to decode an UTF-8 codepoints and return the encoding length (or 0 if the codepoint is ...
user avatar
  • 335
7 votes
1 answer
566 views

UTF8 Codepoint decode and length

I needed a function that would: Decode, and return, the first character in an UTF8 encoded strings Return the length of encoding with the special case that lenght of ...
user avatar
  • 335
7 votes
1 answer
704 views

UTF-8 encoding/decoding

I have two functions, a collection of possible error codes, and a unit-testing framework. The parsing of a character into its unary prefix and payload is handled by a few named functions and macros <...
user avatar
  • 1,906
4 votes
1 answer
2k views

Truncating Incomplete UTF-8 character

I created a function that truncates an incomplete UTF-8 character at the end of std::string in C++. C++'s Standard Library does not yet support character based <...
user avatar
  • 143
11 votes
1 answer
237 views

Metalsmith plugin for stripping UTF8 BOM from files

I've been developing a metalsmith static site and I came across an issue where Visual Studio was automatically adding a BOM to the pages. I wrote the following plugin for metalsmith (it needs to be ...
user avatar
  • 3,065
1 vote
1 answer
3k views

Robustly dealing with malformed Unicode files

I'm writing a script that deals reads UTF-8-encoded XML files and writes parts of those files into a tempfile for further processing. Sometimes, the input files will have a few malformed characters. ...
user avatar
  • 465
13 votes
2 answers
19k views

Validating UTF-8 byte array

I'm writing a validator function that receives a byte[] and checks whether it represents a valid UTF-8 byte sequence, according to this table. Is my approach ...
user avatar
7 votes
3 answers
1k views

Functions to escape CSS rules in PHP

Some context I've been tasked with supplying an escaping function to arbitrary CSS values that are entered through a form. The goals and caveats are: I know it's bad practice to let users input CSS. ...
user avatar
7 votes
1 answer
5k views

Method to return a string of max length (in bytes vs. characters)

In my (c#) code, I need to generate a string (from a longer string) which when UTF-8 encoded, is no longer than a given max length (in bytes). ...
user avatar
  • 205
3 votes
1 answer
2k views

Replacing Perisan and Arabic digits

I'm using this function to replace UTF-8 characters representing numbers in text with 'normal' digits. I'm wondering if this is optimized code since this is using two ...
user avatar
1 vote
1 answer
75 views

Avoiding use of .encode() in rss2html

My concern with this code is the excessive use of .encode('utf-8'). Any advice on refining these functions would be very helpful. rss2html GitHub repo ...
user avatar
  • 1,685
10 votes
2 answers
1k views

Customised Java UTF-8

I have implemented a customized UTF-8 encoding mechanism. The code works fine, but I have a lot of concerns regarding the code. ...
user avatar
  • 461
10 votes
4 answers
12k views

Function to convert ISO-8859-1 to UTF-8

I wrote this function last year to convert between the two encodings and just found it. It takes a text buffer and its size, then converts to UTF-8 if there's enough space. What should be changed to ...
user avatar
  • 1,959
16 votes
3 answers
28k views

Count byte length of string

I am looking for some guidance and optimization pointers for my custom JavaScript function which counts the bytes in a string rather than just chars. The website uses UTF-8 and I am looking to ...
user avatar
7 votes
1 answer
20k views

Splitting a CSV file with headers

I have been looking for an algorithm for splitting a file into smaller pieces, which satisfy the following requirements: The first line in the original file is a header, this header must be carried ...
user avatar
  • 640
4 votes
2 answers
287 views

Macros to detect UTF-8

I'm working on a program that handles UTF-8 characters. I've made the following macros to detect UTF-8. I've tested them with a few thousand words and they seem to work. I'll add another one to do ...
user avatar
  • 1,959
5 votes
1 answer
7k views

Better code for converting a char to its UTF-8 percent encoding representation?

This is working code for a URI template (RFC 6570) implementation; when the character to render is not within a specific character set, it is needed to grab the UTF-8 representation of that character ...
user avatar
  • 1,733
7 votes
1 answer
639 views

UTF-8 character reader function

You may see full code here (note that the link points to the specific commit). Language is "clean C" (that is, a subset of C89, C99 and C++98 — it is intended to compile under all of these standards)....
user avatar