Questions tagged [utf-8]
UTF-8 (Unicode Transformation Format, 8 bits) is a character encoding that describes each Unicode code point using a byte sequence of one to six bytes. It is backwards-compatible with ASCII while still supporting representation of all Unicode code points.
40
questions
3
votes
1
answer
94
views
Determining if a file is UTF-8 text by looking at its first n bytes
I'm trying to find out whether a particular file is UTF-8 encoded readable text, by which I mean printable symbols, whitespaces, \n, ...
- 33
2
votes
2
answers
129
views
Wielding .NET masterfully to encode non-alphanumeric characters into utf-8 hex representation
I have these two methods that work, but I also hate because they almost certainly can be improved. I'm hoping to gain some guidance from others who are more knowledgable of .NET's offering for ...
- 23
-1
votes
1
answer
426
views
I wrote a header file to write German umlauts in a textfile properly
This function is about the fact that a std::wstring was used in another cpp file in order to be able to read strings with German umlauts from the console. Since it is difficult to get wstrings into a ...
- 219
3
votes
1
answer
1k
views
C++ UTF-8 decoder
While writing simple text rendering I found a lack of utf-8 decoders. Most decoders I found required allocating enough space for decoded string. In worse case that would mean that the decoded string ...
- 155
2
votes
1
answer
52
views
Validator and Sanitizer for HTML 5 attribute regex according to current HTML living standard
According to https://html.spec.whatwg.org/multipage/syntax.html#attributes-2 an HTML 5 attribute name is defined like this:
Attribute names must consist of one or more characters other than
controls, ...
- 137
2
votes
1
answer
175
views
Find the UTF-8 Length of a given codepoint
A codepoint in UTF-8 can be represented by one to four bytes. Given a codepoint, I need to determine the length (in bytes) of the codepoint if it were represented in UTF-8. For this, I've written the ...
- 195
2
votes
1
answer
224
views
A C++ function to read Code Points from an UTF-8 Stream
I've written a function that reads and returns one UTF-8 code point from an istream. I am wondering if the code is efficient or if there are some obvious problems with the implementation.
...
- 195
1
vote
1
answer
776
views
The conversion from UTF-16 to UTF-8
I have created a function that converts from UTF-16 to UTF-8.
This function converts from UTF-16 to codepoint firstly, then from codepoint to UTF-8.
...
- 129
20
votes
6
answers
4k
views
Can I slim this UTF8-encoding program?
Below is my entire program. You can read what it does thanks to the comments and specifications in particular.
My question is: can it be improved? Would it be possible, for example, to avoid writing ...
- 387
3
votes
1
answer
91
views
Save on typing while using UTF8 encoding
Typing in something like Encoding.UTF8.GetString(...) and Encoding.UTF8.GetBytes(...) everywhere in your code could be ...
- 6,123
8
votes
1
answer
106
views
Checking whether a string fragment could be part of a longer UTF-8 string
Although UTF-8 validation is a common task, I'm trying to solve a slightly different task; given a string of bytes, work out whether it could potentially be a fragment of a valid UTF-8 string. That's ...
- 181
8
votes
1
answer
169
views
myUTF-8 small lib (validate UTF-8, guess language, count chars)
I'm new to C language and never got my self into the details of UTF-8, and after reading some articles about it, I wanted to try and play with UTF-8 with C language for both fun and practicing ...
- 457
5
votes
1
answer
11k
views
Convert to UTF-8 all files in a directory
1. Summary
I can't find, how I can to refactor multiple with open for one file.
2. Expected behavior of program
Program detect encoding for each file in the ...
- 153
9
votes
5
answers
6k
views
Convert UTF8 string to UTF32 string in C
I'm doing some recreational programming in C (after spending some time in C++, but professionally using only PHP/JavaScript).
I wrote a UTF8 to UTF32 converter and just wanted to know if I made some ...
- 193
3
votes
1
answer
2k
views
Decode string with hex character codes to UTF-8 characters
From a system we receive messages that contain codes that represent UTF-8 characters.
For example :
var str="Test =64 =C2=AE =E1=A6=92 test";
To decode these ...
- 133
5
votes
1
answer
311
views
Convert a mix of latin 1 and UTF-8 to proper UTF-8
The following program takes an arbitrary byte sequence as input and outputs well-formed UTF-8. All UTF-8 sequences from the input are copied unmodified. All other bytes are assumed to be encoded in ...
- 21.1k
7
votes
1
answer
296
views
Wrapper to enhance a program to translate from LTL formulae to Büchi automata
The following is a script that is supposed to act as a wrapper for ltl2ba, a program which transforms LTL formulas into Buchi Automatons encoded in the Promela language. The wrapper adds a limited set ...
- 195
5
votes
1
answer
3k
views
Mapping a CSV with UTF-8 encoding to JSON in Python
I'm not a programmer, just a guy who needs to solve particular things using code.
I needed to create a JSON from a CSV with predefined headers. I'm downloading an attachment from JIRA (REST API), ...
- 153
10
votes
3
answers
4k
views
UTF-8 Validation
I've recently solved the "UTF-8 Validation" LeetCode problem:
A character in UTF8 can be from 1 to 4 bytes long, subjected to the
following rules:
For 1-byte character, the first bit is a 0, ...
- 17.2k
4
votes
0
answers
793
views
Convert Unicode text to font glyphs for display on Android devices
Prior to Android 6.0, Mongolian text is not supported on Android devices. And as far as I know, OpenType smart font rendering is still not supported. Therefore, in all my Mongolian apps I use a ...
- 425
4
votes
2
answers
806
views
Encode unicode codepoints to UTF-8 manually
I want to encode unicode codepoints to UTF-8 manually. I wrote the following C# code. I tested it with some cases I know, but I would like to know if it's correct for all inputs. I know that Unicode ...
- 235
2
votes
2
answers
315
views
Parsing and saving JSON result from GoEuro API
I am new to python so I decided to write a script that parses a json array and writes data in a file.
I wrote the code that does that but I think it can be done way better, So I was hoping to get ...
- 21
5
votes
1
answer
131
views
UTF8 Codepoint decode and length v.2
This is the complete rewrite of a function to decode an UTF-8 codepoints and return the encoding length (or 0 if the codepoint is ...
- 335
7
votes
1
answer
623
views
UTF8 Codepoint decode and length
I needed a function that would:
Decode, and return, the first character in an UTF8 encoded strings
Return the length of encoding with the special case that lenght of ...
- 335
7
votes
1
answer
762
views
UTF-8 encoding/decoding
I have two functions, a collection of possible error codes, and a unit-testing framework.
The parsing of a character into its unary prefix and payload is handled by a few named functions and macros <...
- 2,008
4
votes
1
answer
2k
views
Truncating Incomplete UTF-8 character
I created a function that truncates an incomplete UTF-8 character at the end of std::string in C++.
C++'s Standard Library does not yet support character based <...
- 143
11
votes
1
answer
240
views
Metalsmith plugin for stripping UTF8 BOM from files
I've been developing a metalsmith static site and I came across an issue where Visual Studio was automatically adding a BOM to the pages.
I wrote the following plugin for metalsmith (it needs to be ...
- 3,095
1
vote
1
answer
3k
views
Robustly dealing with malformed Unicode files
I'm writing a script that deals reads UTF-8-encoded XML files and writes parts of those files into a tempfile for further processing.
Sometimes, the input files will have a few malformed characters. ...
- 465
14
votes
2
answers
20k
views
Validating UTF-8 byte array
I'm writing a validator function that receives a byte[] and checks whether it represents a valid UTF-8 byte sequence, according to this table.
Is my approach ...
- 559
7
votes
3
answers
1k
views
Functions to escape CSS rules in PHP
Some context
I've been tasked with supplying an escaping function to arbitrary CSS values that are entered through a form. The goals and caveats are:
I know it's bad practice to let users input CSS. ...
- 4,747
7
votes
1
answer
6k
views
Method to return a string of max length (in bytes vs. characters)
In my (c#) code, I need to generate a string (from a longer string) which when UTF-8 encoded, is no longer than a given max length (in bytes).
...
- 205
3
votes
1
answer
2k
views
Replacing Perisan and Arabic digits
I'm using this function to replace UTF-8 characters representing numbers in text with 'normal' digits.
I'm wondering if this is optimized code since this is using two ...
- 133
1
vote
1
answer
75
views
Avoiding use of .encode() in rss2html
My concern with this code is the excessive use of .encode('utf-8'). Any advice on refining these functions would be very helpful.
rss2html GitHub repo
...
- 1,695
11
votes
2
answers
1k
views
Customised Java UTF-8
I have implemented a customized UTF-8 encoding mechanism. The code works fine, but I have a lot of concerns regarding the code.
...
- 491
10
votes
4
answers
12k
views
Function to convert ISO-8859-1 to UTF-8
I wrote this function last year to convert between the two encodings and just found it. It takes a text buffer and its size, then converts to UTF-8 if there's enough space.
What should be changed to ...
- 1,979
17
votes
3
answers
28k
views
Count byte length of string
I am looking for some guidance and optimization pointers for my custom JavaScript function which counts the bytes in a string rather than just chars. The website uses UTF-8 and I am looking to ...
- 483
7
votes
1
answer
21k
views
Splitting a CSV file with headers
I have been looking for an algorithm for splitting a file into smaller pieces, which satisfy the following requirements:
The first line in the original file is a header, this header must be carried ...
- 650
4
votes
2
answers
292
views
Macros to detect UTF-8
I'm working on a program that handles UTF-8 characters. I've made the following macros to detect UTF-8. I've tested them with a few thousand words and they seem to work.
I'll add another one to do ...
- 1,979
5
votes
1
answer
7k
views
Better code for converting a char to its UTF-8 percent encoding representation?
This is working code for a URI template (RFC 6570) implementation; when the character to render is not within a specific character set, it is needed to grab the UTF-8 representation of that character ...
- 1,743
7
votes
1
answer
652
views
UTF-8 character reader function
You may see full code here (note that the link points to the specific commit).
Language is "clean C" (that is, a subset of C89, C99 and C++98 — it is intended to compile under all of these standards)....
