File: sed.info,  Node: Locale Considerations,  Prev: Escapes,  Up: sed regular expressions

5.9 Multibyte characters and Locale Considerations
==================================================

GNU ‘sed’ processes valid multibyte characters in multibyte locales
(e.g.  ‘UTF-8’).  (1)

The following example uses the Greek letter Capital Sigma (Σ, Unicode
code point ‘0x03A3’).  In a ‘UTF-8’ locale, ‘sed’ correctly processes
the Sigma as one character despite it being 2 octets (bytes):

     $ locale | grep LANG
     LANG=en_US.UTF-8

     $ printf 'a\u03A3b'
     aΣb

     $ printf 'a\u03A3b' | sed 's/./X/g'
     XXX

     $ printf 'a\u03A3b' | od -tx1 -An
      61 ce a3 62

To force ‘sed’ to process octets separately, use the ‘C’ locale (also
known as the ‘POSIX’ locale):

     $ printf 'a\u03A3b' | LC_ALL=C sed 's/./X/g'
     XXXX

5.9.1 Invalid multibyte characters
----------------------------------

‘sed’'s regular expressions _do not_ match invalid multibyte sequences
in a multibyte locale.

In the following examples, the ascii value ‘0xCE’ is an incomplete
multibyte character (shown here as �).  The regular expression ‘.’ does
not match it:

     $ printf 'a\xCEb\n'
     a�e

     $ printf 'a\xCEb\n' | sed 's/./X/g'
     X�X

     $ printf 'a\xCEc\n' | sed 's/./X/g' | od -tx1c -An
       58  ce  58  0a
        X      X   \n

Similarly, the 'catch-all' regular expression ‘.*’ does not match the
entire line:

     $ printf 'a\xCEc\n' | sed 's/.*//' | od -tx1c -An
       ce  63  0a
            c  \n

GNU ‘sed’ offers the special ‘z’ command to clear the current pattern
space regardless of invalid multibyte characters (i.e.  it works like
‘s/.*//’ but also removes invalid multibyte characters):

     $ printf 'a\xCEc\n' | sed 'z' | od -tx1c -An
        0a
        \n

Alternatively, force the ‘C’ locale to process each octet separately
(every octet is a valid character in the ‘C’ locale):

     $ printf 'a\xCEc\n' | LC_ALL=C sed 's/.*//' | od -tx1c -An
       0a
       \n

   ‘sed’'s inability to process invalid multibyte characters can be used
to detect such invalid sequences in a file.  In the following examples,
the ‘\xCE\xCE’ is an invalid multibyte sequence, while ‘\xCE\A3’ is a
valid multibyte sequence (of the Greek Sigma character).

The following ‘sed’ program removes all valid characters using ‘s/.//g’.
Any content left in the pattern space (the invalid characters) are added
to the hold space using the ‘H’ command.  On the last line (‘$’), the
hold space is retrieved (‘x’), newlines are removed (‘s/\n//g’), and any
remaining octets are printed unambiguously (‘l’).  Thus, any invalid
multibyte sequences are printed as octal values:

     $ printf 'ab\nc\n\xCE\xCEde\n\xCE\xA3f\n' > invalid.txt

     $ cat invalid.txt
     ab
     c
     ��de
     Σf

     $ sed -n 's/.//g ; H ; ${x;s/\n//g;l}' invalid.txt
     \316\316$

With a few more commands, ‘sed’ can print the exact line number
corresponding to each invalid characters (line 3).  These characters can
then be removed by forcing the ‘C’ locale and using octal escape
sequences:

     $ sed -n 's/.//g;=;l' invalid.txt | paste - -  | awk '$2!="$"'
     3       \316\316$

     $ LC_ALL=C sed '3s/\o316\o316//' invalid.txt > fixed.txt

5.9.2 Upper/Lower case conversion
---------------------------------

GNU ‘sed’'s substitute command (‘s’) supports upper/lower case
conversions using ‘\U’,‘\L’ codes.  These conversions support multibyte
characters:

     $ printf 'ABC\u03a3\n'
     ABCΣ

     $ printf 'ABC\u03a3\n' | sed 's/.*/\L&/'
     abcσ

*Note The "s" Command::.

5.9.3 Multibyte regexp character classes
----------------------------------------

In other locales, the sorting sequence is not specified, and ‘[a-d]’
might be equivalent to ‘[abcd]’ or to ‘[aBbCcDd]’, or it might fail to
match any character, or the set of characters that it matches might even
be erratic.  To obtain the traditional interpretation of bracket
expressions, you can use the ‘C’ locale by setting the ‘LC_ALL’
environment variable to the value ‘C’.

     # TODO: is there any real-world system/locale where 'A'
     #       is replaced by '-' ?
     $ echo A | sed 's/[a-z]/-/'
     A

   Their interpretation depends on the ‘LC_CTYPE’ locale; for example,
‘[[:alnum:]]’ means the character class of numbers and letters in the
current locale.

   TODO: show example of collation

     # TODO: this works on glibc systems, not on musl-libc/freebsd/macosx.
     $ printf 'cliché\n' | LC_ALL=fr_FR.utf8 sed 's/[[=e=]]/X/g'
     clichX

   ---------- Footnotes ----------

   (1) Some regexp edge-cases depends on the operating system and libc
implementation.  The examples shown are known to work as-expected on
GNU/Linux systems using glibc.