Sparkles

Appearance

Unicode analysis reference ​

sparkles.base.text.analysis provides a fixed-workspace pipeline for Unicode normalization, canonical combining-class ordering, simple/full case folding, mark removal, word-boundary flags, stopword removal, and strict-plus-opaque UTF-8 decoding. Its property data is owned Unicode 18.0.0 data, not the compiler's potentially different Unicode version.

Manifest-driven Unicode data ​

The reviewed inventory is libs/base/tools/unicode/manifest.json. It records all 35 source, test, and license artifacts under their real ucd/ or emoji/ paths, their actual SHA-256 hashes, algorithm editions, generator revision 1, and generated schema revision 1. The manifest's exact-byte SHA-256 is the generated unicodeManifestIdentity:

text
df3659783f974cb439f4f6436dc4e72f0d06313a45b865c04921dba1d938abcc

Only the manifest is tracked as raw-input metadata. The inventory at libs/base/tools/unicode/18.0.0/ is ignored, locally cached data. Default generation fetches it when the root is absent, stages and authenticates all artifacts, then installs the directory atomically before generating:

sh
dub run --single libs/base/tools/gen_unicode_tables.d -- \
  --manifest libs/base/tools/unicode/manifest.json \
  --ucd-dir libs/base/tools/unicode/18.0.0 \
  --out-file libs/base/src/sparkles/base/text/unicode_tables.d

Warm regeneration can explicitly forbid acquisition with --no-network. A missing root then fails without fetching; an existing incomplete or tampered inventory always fails authentication and is never silently repaired:

sh
dub run --single libs/base/tools/gen_unicode_tables.d -- \
  --no-network --manifest libs/base/tools/unicode/manifest.json \
  --ucd-dir libs/base/tools/unicode/18.0.0 \
  --out-file libs/base/src/sparkles/base/text/unicode_tables.d

--acquire remains an explicit acquisition-only operation into an absent root; it cannot be combined with --no-network. For example:

sh
dub run --single libs/base/tools/gen_unicode_tables.d -- \
  --acquire --ucd-dir /tmp/unicode18-acquired

The Nix conformance data derivation provisions the same inventory through manifest-driven fixed-output downloads, not checked-in raw bytes. Normative execution uses those immutable provisioned inputs with no network access.

Every artifact, including conformance corpora and the Unicode license, is authenticated before parsing. Unknown aliases/values, malformed fields, conflicting explicit ranges, unmatched First/Last pairs, and cyclic mappings fail before installed output changes. The sole generated module is staged, read back, and atomically renamed only after the complete emission succeeds. It retains the entire Unicode License V3 notice. Generated bytes contain no timestamp, absolute source path, compiler probe, or hash-map iteration order.

The generator's unittests cover byte/number parsing, layered @missing defaults, aliases, conflicting ranges, First/Last pairs, recursive and Hangul decomposition, cycles, SHA-256 mismatch/truncation, and staged-lookup boundaries. A reproduction recipe (run after integration, not a reported verification result):

sh
mkdir -p /tmp/unicode18-a /tmp/unicode18-b
LC_ALL=C dub run --single libs/base/tools/gen_unicode_tables.d \
  --compiler=ldc2 -bunittest -- \
  --no-network --manifest libs/base/tools/unicode/manifest.json \
  --ucd-dir libs/base/tools/unicode/18.0.0 \
  --out-file /tmp/unicode18-a/unicode_tables.d
LC_ALL=C.UTF-8 dub run --single libs/base/tools/gen_unicode_tables.d \
  --compiler=dmd -bunittest -- \
  --no-network --manifest libs/base/tools/unicode/manifest.json \
  --ucd-dir libs/base/tools/unicode/18.0.0 \
  --out-file /tmp/unicode18-b/unicode_tables.d
cmp /tmp/unicode18-a/unicode_tables.d /tmp/unicode18-b/unicode_tables.d

Generated property interfaces ​

Named constant-time staged lookups expose GraphemeBreakClass, IndicConjunctBreakClass, GeneralCategory, EastAsianWidthClass, WordBreakClass, SentenceBreakClass, LineBreakClass, BidiClass, Script, JoiningType, and JoiningGroup. Their functions are respectively graphemeBreakClass, indicConjunctBreakClass, generalCategory, eastAsianWidthClass, wordBreakClass, sentenceBreakClass, lineBreakClass, bidiClass, script, joiningType, and joiningGroup. unicodeCoreProperties performs one fused staged lookup for grapheme, Indic-conjunct, extended-pictographic, category, and East Asian width properties. No terminal width policy is embedded in raw properties.

UnicodeProperty plus unicodePropertyValue, unicodePropertyValueName, and unicodeProperty expose all consumed enumerated and binary properties, including normalization quick checks, casing context properties, composition exclusions, default-ignorables, and emoji flags. Values are identified by the release's short PropertyValueAliases spellings, not compiler enums. Named membership helpers retain isEastAsianWide, isEastAsianAmbiguous, isEmojiVsBase, isUnicodeMark, and isUnicodeUppercase, and add isUnicodeControl, isUnicodeFormat, and isExtendedPictographic.

Mapping APIs use UnicodeMappingSpan(offset, length, present) and corresponding *Value(offset) accessors. Presence distinguishes a mapped deletion from an absent identity mapping. Existing canonical/compatibility decomposition, canonical composition, combining class, and simple/full fold seams remain. Additional mappings include full lower/title/uppercase, Turkic folding, NFKC casefold/simple-casefold, FC_NFKC closure, and script extensions. An absent script-extension span means the singleton script(cp) value. contextualCaseRules retains language/context predicates and lower/title/upper mapping spans; contextualCaseValue reads their scalar data. bidiBracket, bidiMirror, and bidiBracketType expose paired/mirrored scalars and bracket type (0 none, 1 opening, 2 closing). emojiSequences, EmojiSequenceKind, and emojiSequenceValue retain the RGI, variation, qualification, and component inventory.

Raw property queries accept Unicode code points, including surrogate category data, and assert the U+10FFFF bound. Scalar algorithms must reject surrogate inputs before querying. Generated page/index arithmetic is validated across the entire code-point domain against the parsed raw representation. All runtime lookups are @safe pure nothrow @nogc.

Conformance and upgrades ​

The harness no longer has independent width/compiler-segmentation versions. Its configuration uses the generated manifest identity, and authenticates the selected manifest and every fetched corpus. --ucd-dir names the inventory root containing ucd/ and emoji/, not a flattened directory:

sh
dub run --root libs/base/tools/text-conformance --config=offline -- \
  --layers 0,1,2 --no-network \
  --manifest libs/base/tools/unicode/manifest.json \
  --ucd-dir libs/base/tools/unicode/18.0.0

The harness's width oracle owns sorted, half-open CodepointRanges; interval insertion is @safe and merges overlapping or adjacent ranges. Its oracle unittests are exercised separately from the executable corpus sweep:

sh
dub test :text-conformance --compiler=ldc2 -- -t 1
dub test :text-conformance --compiler=dmd -- -t 1

A release, algorithm, schema, or reviewed-byte change requires a newly reviewed manifest and generator revision update. The generator rejects unreviewed manifest identities. Regenerate the complete module in one change; never mix partially upgraded property families. Applications must include unicodeManifestIdentity in property/analysis/boundary/mapping cache keys and reject earlier identities before reading cached results; matchesUnicodeManifest provides an exact comparison. The conformance cache is likewise namespaced by manifest identity.

Source spans ​

TextUnit.value is a scalar or opaqueByteBase + byte; sourceStart and sourceEnd are half-open byte offsets enclosing its contributors. This envelope can include unrelated bytes after canonical reordering: it is not an exact selection or highlighting range. Use workspace.contributingSourceSpans(unit) for the sorted, deduplicated exact contributor set, and workspace.deletions for removed source spans anchored at output boundaries. Expansion retains the original contributor set on each output; composition merges contributor sets without filling gaps.

The lower-level UnicodeTransformView exposes units, spans, deletions and contributingSourceSpans(index). Each UnicodeDeletion records its source span and output-unit boundary. Opaque bytes remain explicit non-scalar units and terminate Unicode context. Do not feed them to scalar encoding.

Owned algorithms and workspace contracts ​

The normative pipeline is owned UTF decoding, UAX #15 normalization, owned Unicode casing, optional renormalization, canonical decomposition/mark removal and restoration of the selected form, then UAX #29 word boundaries and optional stopword deletion. AnalysisOptions makes normalization, casing, locale, folding, renormalization and lexicon policy explicit. Root locale is ""; supported locale identifiers are "tr", "az" and "lt". Unsupported locale and invalid options return errors rather than substituting another policy.

AnalysisWorkspace!(N, S, W, P) separately bounds final units (N), decomposed starter/combining segments (S), intermediate units (W), and provenance spans/deletion records per bank (P). Defaults are S = 64, W = N + S, P = 4 * W; they are storage choices, not Unicode limits. An intermediate expansion can exhaust W even when final output fits N. Failures include outputFull, segmentTooLong, workspaceFull, sourceTooLong, invalidOptions and unsupportedLocale. Check AnalysisResult.succeeded before reading output; failure publishes empty output, not a silently shortened successful transform. Reuse invalidates all prior output and provenance borrows.

Direct modules expose the same cores without allocating:

  • boundaries: wordBoundaries and sentenceBoundaries consume whole token input; scratch requires n entries for n input tokens and boundary output requires n + 1 entries, including start/end positions.
  • line_break: owned default UAX #14 opportunities, distinct from selecting actual wrapped lines.
  • bidi: whole-paragraph resolution followed by line-local reordering. Paragraph variable-length workspace slices require one entry per original token, including removed controls; line scratch requires one entry per selected original logical token. Bidi ordering is not font shaping.
  • normalization: normalizeText implements all four forms. Scratch items holds the entire decomposed input; ordering holds the longest reordered nonstarter run; boundaries needs input-unit count plus one when deletions exist.
  • casing: caseTransform requires contexts[n] and boundaryMap[n+1]; titlecasing additionally requires tokens[n], words[n+1] and wordWorkspace[n]. Full expansions and contextual rules use the original normalized input context, not a prematurely edited prefix.
  • transform: caller-owned units, exact source-span and deletion arrays plus exactly one UnicodeTransformEpoch. Successful publication yields a borrowed immutable view. Starting another operation invalidates earlier views, even when that operation fails; valid() checks publication and generation.

Scratch and output arenas must be disjoint as specified by each operation. The caller owns their lifetime; independent concurrent operations require independent arenas. UnicodeResult reports capacity/input/overlap/overflow failures and required capacity where available. Never consume an unpublished view or retain provenance after its workspace is reused.

Typed source maps ​

source_map distinguishes SourceByteOffset, ScalarIndex, Utf16Offset, GraphemeIndex, CellPosition, TransformedByteOffset and TransformedUnitIndex; none implicitly converts into another. SourceMapKey identifies immutable source identity/revision, encoding, malformed-input policy, Unicode manifest, transform identity/revision and cell policy/revision. Callers assign complete transform identities including locale, pipeline and lexicon revisions. A source revision must never be mutated under the same key.

SourceMapWorkspace borrows boundary arenas and exactly one disjoint SourceMapEpoch; source geometry needs scalar-count plus one entries, transformed geometry needs output-unit count plus one. Builds rescan the whole source rather than certifying Unicode context with fixed lookbehind. Rebuilding invalidates old snapshots even on failure. A published SourceMapView borrows workspace and transform storage: keep them alive and unchanged throughout use. Queries require the current key and distinguish staleSource, stalePolicy and staleView.

MapAffinity.exact rejects non-boundaries and ambiguous hits; before/after select the lower/upper boundary. Wide-cell interiors snap to whole-cluster endpoints. Zero-advance clusters can share a cell position: before/after select different exact coordinate hits while exact-only rejects the tie. CellPolicy.none rejects cell geometry; terminalKitty revision 1 uses whole clusters, zero-width controls/separators and zero-width context-free tabs. Opaque UTF-8 with none supports byte/grapheme maps but not scalar/UTF-16 geometry.

Projection envelopes (MapResult.lower/upper) are not contributor sets. MapRelationship distinguishes exact contributor rows, deletion rows with empty output intervals, and emptyOutputEndpoints; the latter preserves original start/end and is not a contributor hull. Use exact relationship/source span queries for selection, including reordered noncontiguous contributors.

ASCII versus root Unicode casing ​

case_text.asciiLower and asciiUpper implement the byte-ASCII policy: only ASCII letters change, and non-ASCII or malformed bytes are preserved without decoding. Identifier case_style likewise uses ASCII boundaries and ASCII recasing; it is not a Unicode word/title algorithm.

case_text.unicodeLower, unicodeUpper, unicodeTitle and unicodeFold implement strict root-Unicode convenience operations and allocate their result. They validate the complete UTF-8 input, retain full expansions and contextual rules, and throw UnicodeCaseTextException for malformed input rather than returning a transformed prefix. Folding is default non-Turkic folding, not normalization. For caller-owned @nogc storage or explicit locales use casing.caseTransform instead.