characters

Characters are strings viewed as sequences of user-perceived characters, also known as Unicode (extended) grapheme clusters.

The Characters class allows access to the individual characters of a string, and a way to navigate back and forth between them using a CharacterRange.

Based on Unicode version 16.0.0.

Unicode characters and representations

There is no such thing as plain text.

Computers only know numbers, so any "text" on a computer is represented by numbers, which are again stored as bytes in memory.

The meaning of those bytes are provided by layers of interpretation, building up to the glyphs that the computer displays on the screen.

Abstraction	Dart Type	Usage	Example
Bytes	ByteBuffer, Uint8List	Physical layout: Memory or network communication.	file.readAsBytesSync()
Code units	Uint8List (UTF‑8) Uint16List, String (UTF‑16)	Standard formats for encoding code points in memory. Stored in memory using one (UTF‑8) or more (UTF‑16) bytes. One or more code units encode a code point.	string.codeUnits string.codeUnitAt(index) utf8.encode(string)
Code points	Runes	The Unicode unit of meaning.	string.runes
Grapheme Clusters	Characters	Human perceived character. One or more code points.	string.characters
Glyphs		Visual rendering of grapheme clusters.	print(string)

A Dart String is a sequence of UTF-16 code units, just like strings in JavaScript and Java. The runtime system decides on the underlying physical representation.

That makes plain strings inadequate when needing to manipulate the text that a user is viewing, or entering, because string operations are not working at the grapheme cluster level.

For example, to abbreviate a text to, say, the 15 first characters or glyphs, a string like "A 🇬🇧 text in English" should abbreviate to "A 🇬🇧 text in Eng… when counting characters, but will become "A 🇬🇧 text in …" if counting code units using String operations.

Whenever you need to manipulate strings at the character level, you should be using the Characters type, not the methods of the String class.

The Characters class

The Characters class exposes a string as a sequence of grapheme clusters. All operations on Characters operate on entire grapheme clusters, so it removes the risk of splitting combined characters or emojis that are inherent in the code-unit based String operations.

You can get a Characters object for a string using either the constructor Characters(string) or the extension getter string.characters.

At its core, the class is an Iterable<String> where the element strings are single grapheme clusters. This allows sequential access to the individual grapheme clusters of the original string.

On top of that, there are operations mirroring the operations of String that are not index, code-unit or code-point based, like startsWith or replaceAll. There are some differences between these and the String operations. For example the replace methods only accept characters as pattern. Regular expressions are not grapheme cluster aware, so they cannot be used safely on a sequence of characters.

Grapheme clusters have varying length in the underlying representation, so operations on a Characters sequence cannot be index based. Instead, the CharacterRange iterator provided by Characters.iterator has been greatly enhanced. It can move both forwards and backwards, and it can span a range of grapheme cluster. Most operations that can be performed on a full Characters can also be performed on the grapheme clusters in the range of a CharacterRange. The range can be contracted, expanded or moved in various ways, not restricted to using moveNext, to move to the next grapheme cluster.

Example:

// Using String indices.
                String? firstTagString(String source) {
                  var start = source.indexOf('<') + 1;
                  if (start > 0) {
                    var end = source.indexOf('>', start);
                    if (end >= 0) {
                      return source.substring(start, end);
                    }
                  }
                  return null;
                }
                
                // Using CharacterRange operations.
                Characters? firstTagCharacters(Characters source) {
                  var range = source.findFirst('<'.characters);
                  if (range != null && range.moveUntil('>'.characters)) {
                    return range.currentCharacters;
                  }
                  return null;
                }
                

1.4.0

• Updated to use Unicode 16.0.0.

1.3.1

• Fixed README rendering on pub.dev and API docs.
• Require Dart ^3.4.0.
• Move to dart-lang/core monorepo.

1.3.0

• Updated to use Unicode 15.0.0.

1.2.1

• Update the value of the pubspec repository field.

1.2.0

• Fix Characters.where which unnecessarily did the iteration and test twice.
• Adds Characters.empty constant and makes Characters("") return it.
• Changes the argument type of Characters.contains to (covariant) String. The implementation still accepts Object?, so it can be cast to Iterable<Object?>, but you get warned if you try to call directly with a non-String.

1.1.0

• Stable release for null safety.
• Added stringBeforeLength and stringAfterLength to CharacterRange.
• Added CharacterRange.at constructor.
• Added getRange(start, end) and characterAt(pos) to Characters as alternative to .take(end).skip(start) and getRange(pos, pos + 1).
• Change some positional parameter names from other to characters.

1.0.0

• Core APIs deemed stable; package version set to 1.0.0.
• Added split methods on Characters and CharacterRange.

0.5.0

• Change [codeUnits] getter to [utf16CodeUnits] which returns an iterable. This avoids leaking that the underlying string has efficient UTF-16 code unit access in the API, and allows the same interface to be just as efficiently implemented on top of UTF-8.

0.4.0

• Added an extension method on String to allow easy access to the Characters of the string:

  print('The first character is: ' + myString.characters.first)
                  

• Updated Dart SDK dependency to Dart 2.6.0

0.3.1

• Added small example in example/main.dart
• Enabled pedantic lints and updated code to resolve issues.

0.3.0

• Updated API which does not expose the underlying string indices.

0.1.0

• Initial release