Questions tagged [utf-8]

UTF-8 (Unicode Transformation Format, 8 bits) is a character encoding that describes each Unicode code point using a byte sequence of one to six bytes. It is backwards-compatible with ASCII while still supporting representation of all Unicode code points.

Filter by
Sorted by
Tagged with
8
votes
0answers
50 views

Checking whether a string fragment could be part of a longer UTF-8 string

Although UTF-8 validation is a common task, I'm trying to solve a slightly different task; given a string of bytes, work out whether it could potentially be a fragment of a valid UTF-8 string. That's ...
8
votes
1answer
154 views

myUTF-8 small lib (validate UTF-8, guess language, count chars)

I'm new to C language and never got my self into the details of UTF-8, and after reading some articles about it, I wanted to try and play with UTF-8 with C language for both fun and practicing ...
4
votes
1answer
3k views

Convert to UTF-8 all files in a directory

1. Summary I can't find, how I can to refactor multiple with open for one file. 2. Expected behavior of program Program detect encoding for each file in the ...
6
votes
5answers
1k views

Convert UTF8 string to UTF32 string in C

I'm doing some recreational programming in C (after spending some time in C++, but professionally using only PHP/JavaScript). I wrote a UTF8 to UTF32 converter and just wanted to know if I made some ...
3
votes
1answer
2k views

Decode string with hex character codes to UTF-8 characters

From a system we receive messages that contain codes that represent UTF-8 characters. For example : var str="Test =64 =C2=AE =E1=A6=92 test"; To decode these ...
5
votes
1answer
99 views

Convert a mix of latin 1 and UTF-8 to proper UTF-8

The following program takes an arbitrary byte sequence as input and outputs well-formed UTF-8. All UTF-8 sequences from the input are copied unmodified. All other bytes are assumed to be encoded in ...
7
votes
1answer
198 views

Wrapper to enhance a program to translate from LTL formulae to Büchi automata

The following is a script that is supposed to act as a wrapper for ltl2ba, a program which transforms LTL formulas into Buchi Automatons encoded in the Promela language. The wrapper adds a limited set ...
4
votes
1answer
3k views

Mapping a CSV with UTF-8 encoding to JSON in Python

I'm not a programmer, just a guy who needs to solve particular things using code. I needed to create a JSON from a CSV with predefined headers. I'm downloading an attachment from JIRA (REST API), ...
10
votes
3answers
3k views

UTF-8 Validation

I've recently solved the "UTF-8 Validation" LeetCode problem: A character in UTF8 can be from 1 to 4 bytes long, subjected to the following rules: For 1-byte character, the first bit is a 0, ...
3
votes
0answers
653 views

Convert Unicode text to font glyphs for display on Android devices

Prior to Android 6.0, Mongolian text is not supported on Android devices. And as far as I know, OpenType smart font rendering is still not supported. Therefore, in all my Mongolian apps I use a ...
4
votes
2answers
365 views

Encode unicode codepoints to UTF-8 manually

I want to encode unicode codepoints to UTF-8 manually. I wrote the following C# code. I tested it with some cases I know, but I would like to know if it's correct for all inputs. I know that Unicode ...
2
votes
2answers
289 views

Parsing and saving JSON result from GoEuro API

I am new to python so I decided to write a script that parses a json array and writes data in a file. I wrote the code that does that but I think it can be done way better, So I was hoping to get ...
5
votes
1answer
86 views

UTF8 Codepoint decode and length v.2

This is the complete rewrite of a function to decode an UTF-8 codepoints and return the encoding length (or 0 if the codepoint is ...
7
votes
1answer
300 views

UTF8 Codepoint decode and length

I needed a function that would: Decode, and return, the first character in an UTF8 encoded strings Return the length of encoding with the special case that lenght of ...
5
votes
1answer
493 views

UTF-8 encoding/decoding

I have two functions, a collection of possible error codes, and a unit-testing framework. The parsing of a character into its unary prefix and payload is handled by a few named functions and macros <...
4
votes
1answer
1k views

Truncating Incomplete UTF-8 character

I created a function that truncates an incomplete UTF-8 character at the end of std::string in C++. C++'s Standard Library does not yet support character based <...
11
votes
1answer
221 views

Metalsmith plugin for stripping UTF8 BOM from files

I've been developing a metalsmith static site and I came across an issue where Visual Studio was automatically adding a BOM to the pages. I wrote the following plugin for metalsmith (it needs to be ...
1
vote
1answer
2k views

Robustly dealing with malformed Unicode files

I'm writing a script that deals reads UTF-8-encoded XML files and writes parts of those files into a tempfile for further processing. Sometimes, the input files will have a few malformed characters. ...
13
votes
2answers
17k views

Validating UTF-8 byte array

I'm writing a validator function that receives a byte[] and checks whether it represents a valid UTF-8 byte sequence, according to this table. Is my approach ...
7
votes
3answers
975 views

Functions to escape CSS rules in PHP

Some context I've been tasked with supplying an escaping function to arbitrary CSS values that are entered through a form. The goals and caveats are: I know it's bad practice to let users input CSS. ...
7
votes
1answer
5k views

Method to return a string of max length (in bytes vs. characters)

In my (c#) code, I need to generate a string (from a longer string) which when UTF-8 encoded, is no longer than a given max length (in bytes). ...
3
votes
1answer
2k views

Replacing Perisan and Arabic digits

I'm using this function to replace UTF-8 characters representing numbers in text with 'normal' digits. I'm wondering if this is optimized code since this is using two ...
1
vote
1answer
73 views

Avoiding use of .encode() in rss2html

My concern with this code is the excessive use of .encode('utf-8'). Any advice on refining these functions would be very helpful. rss2html GitHub repo ...
10
votes
2answers
1k views

Customised Java UTF-8

I have implemented a customized UTF-8 encoding mechanism. The code works fine, but I have a lot of concerns regarding the code. ...
10
votes
4answers
9k views

Function to convert ISO-8859-1 to UTF-8

I wrote this function last year to convert between the two encodings and just found it. It takes a text buffer and its size, then converts to UTF-8 if there's enough space. What should be changed to ...
13
votes
3answers
24k views

Count byte length of string

I am looking for some guidance and optimization pointers for my custom JavaScript function which counts the bytes in a string rather than just chars. The website uses UTF-8 and I am looking to ...
7
votes
1answer
17k views

Splitting a CSV file with headers

I have been looking for an algorithm for splitting a file into smaller pieces, which satisfy the following requirements: The first line in the original file is a header, this header must be carried ...
4
votes
2answers
256 views

Macros to detect UTF-8

I'm working on a program that handles UTF-8 characters. I've made the following macros to detect UTF-8. I've tested them with a few thousand words and they seem to work. I'll add another one to do ...
5
votes
1answer
6k views

Better code for converting a char to its UTF-8 percent encoding representation?

This is working code for a URI template (RFC 6570) implementation; when the character to render is not within a specific character set, it is needed to grab the UTF-8 representation of that character ...
6
votes
1answer
595 views

UTF-8 character reader function

You may see full code here (note that the link points to the specific commit). Language is "clean C" (that is, a subset of C89, C99 and C++98 — it is intended to compile under all of these standards)....