groovy.util
Class CharsetToolkit

Field Summary
 def EMPTY_BYTE_ARRAY
           
 def buffer
           
 Charset charset
           
 Charset defaultCharset
           
 boolean enforce8Bit
           
 File file
           
 
Constructor Summary
CharsetToolkit(File file)
            Constructor of the CharsetToolkit utility class.
 
Method Summary
static def getAvailableCharsets()
           Retrieves all the available Charsets on the platform, among which the default charset.
Charset getCharset()
          
Charset getDefaultCharset()
           Retrieves the default Charset
static Charset getDefaultSystemCharset()
           Retrieve the default charset of the system.
boolean getEnforce8Bit()
           Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.
BufferedReader getReader()
           Gets a BufferedReader (indeed a LineNumberReader) from the File specified in the constructor of CharsetToolkit using the charset discovered by the method guessEncoding().
Charset guessEncoding()
          

Guess the encoding of the provided buffer.

boolean hasUTF16BEBom()
           Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2).
boolean hasUTF16LEBom()
           Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le).
boolean hasUTF8Bom()
           Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors).
static boolean isContinuationChar(byte b)
           If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character;
static boolean isFiveBytesSequence(byte b)
           If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character.
static boolean isFourBytesSequence(byte b)
           If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character.
static boolean isSixBytesSequence(byte b)
           If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character.
static boolean isThreeBytesSequence(byte b)
           If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character.
static boolean isTwoBytesSequence(byte b)
           If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character.
void setDefaultCharset(Charset defaultCharset)
           Defines the default Charset used in case the buffer represents an 8-bit Charset.
void setEnforce8Bit(boolean enforce)
           If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII.
 

Constructor Detail

CharsetToolkit

public CharsetToolkit(File file)
Constructor of the CharsetToolkit utility class.
param:
file of which we want to know the encoding.


Method Detail

getAvailableCharsets

public static def getAvailableCharsets()
Retrieves all the available Charsets on the platform, among which the default charset.
return:
an array of Charsets.


getCharset

public Charset getCharset()


getDefaultCharset

public Charset getDefaultCharset()
Retrieves the default Charset


getDefaultSystemCharset

public static Charset getDefaultSystemCharset()
Retrieve the default charset of the system.
return:
the default Charset.


getEnforce8Bit

public boolean getEnforce8Bit()
Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.
return:
a boolean representing the flag of use of US-ASCII.


getReader

public BufferedReader getReader()
Gets a BufferedReader (indeed a LineNumberReader) from the File specified in the constructor of CharsetToolkit using the charset discovered by the method guessEncoding().
return:
a BufferedReader
throws:
FileNotFoundException if the file is not found.


guessEncoding

Charset guessEncoding()

Guess the encoding of the provided buffer.

If Byte Order Markers are encountered at the beginning of the buffer, we immidiately return the charset implied by this BOM. Otherwise, the file would not be a human readable text file.

If there is no BOM, this method tries to discern whether the file is UTF-8 or not. If it is not UTF-8, we assume the encoding is the default system encoding (of course, it might be any 8-bit charset, but usually, an 8-bit charset is the default one).

It is possible to discern UTF-8 thanks to the pattern of characters with a multi-byte sequence.

 UCS-4 range (hex.)        UTF-8 octet sequence (binary)
 0000 0000-0000 007F       0xxxxxxx
 0000 0080-0000 07FF       110xxxxx 10xxxxxx
 0000 0800-0000 FFFF       1110xxxx 10xxxxxx 10xxxxxx
 0001 0000-001F FFFF       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
 0020 0000-03FF FFFF       111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
 0400 0000-7FFF FFFF       1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
 

With UTF-8, 0xFE and 0xFF never appear.

return:
the Charset recognized.


hasUTF16BEBom

public boolean hasUTF16BEBom()
Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2).
return:
true if the buffer has a BOM for UTF-16 Big Endian.


hasUTF16LEBom

public boolean hasUTF16LEBom()
Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le).
return:
true if the buffer has a BOM for UTF-16 Low Endian.


hasUTF8Bom

public boolean hasUTF8Bom()
Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors).
return:
true if the buffer has a BOM for UTF8.


isContinuationChar

static boolean isContinuationChar(byte b)
If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character;
param:
b a byte.
return:
true if it's a continuation char.


isFiveBytesSequence

static boolean isFiveBytesSequence(byte b)
If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character.
param:
b a byte.
return:
true if it's the first byte of a five-bytes sequence.


isFourBytesSequence

static boolean isFourBytesSequence(byte b)
If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character.
param:
b a byte.
return:
true if it's the first byte of a four-bytes sequence.


isSixBytesSequence

static boolean isSixBytesSequence(byte b)
If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character.
param:
b a byte.
return:
true if it's the first byte of a six-bytes sequence.


isThreeBytesSequence

static boolean isThreeBytesSequence(byte b)
If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character.
param:
b a byte.
return:
true if it's the first byte of a three-bytes sequence.


isTwoBytesSequence

static boolean isTwoBytesSequence(byte b)
If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character.
param:
b a byte.
return:
true if it's the first byte of a two-bytes sequence.


setDefaultCharset

public void setDefaultCharset(Charset defaultCharset)
Defines the default Charset used in case the buffer represents an 8-bit Charset.
param:
defaultCharset the default Charset to be returned by guessEncoding()
if an 8-bit Charset is encountered.


setEnforce8Bit

public void setEnforce8Bit(boolean enforce)
If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII. It might be a file without any special character in the range 128-255, but that may be or become a file encoded with the default charset rather than US-ASCII.
param:
enforce a boolean specifying the use or not of US-ASCII.