Open Chinese Convert: opencc::internal::Utf8SkipTable Struct Reference

Open Chinese Convert 1.4.2+g71621a0e1.dirty A project for conversion between Traditional and Simplified Chinese

Loading...
Searching...
No Matches
Public Member Functions | Public Attributes | List of all members
opencc::internal::Utf8SkipTable Struct Reference

Per-byte lookup table describing which byte values may begin a dictionary key. More...

`#include <src/Utf8SkipScan.hpp>`

## Public Member Functions
bool | CharLevel () const
void | EnableCharLevel ()
void | DisableCharLevel ()
| Permanently falls back to lead-byte filtering, e.g.
void | MarkCharCandidate (uint32_t codePoint)
bool | IsCharCandidate (uint32_t codePoint) const
void | MarkAllCandidates ()
void | Finalize ()
| Must be called after candidate[] is filled and before the table is used.

## Public Attributes
bool | candidate [256] = {}
| candidate[b] is true when byte value b may begin a dictionary key (or is the lead byte of an ideographic description operator), so the conversion loop must run a full prefix lookup at that position.
bool | asciiHasCandidates = false
| True when any ASCII byte value is a candidate; disables the bulk ASCII-run scan.
std::vector< uint64_t > | bmpCandidates
| Optional character-level refinement: one bit per BMP code point (U+0000..U+FFFF, 8 KiB).

## Detailed Description

Per-byte lookup table describing which byte values may begin a dictionary key.

The conversion hot loop uses it to consume runs of characters that cannot possibly match any key without paying for a trie lookup per character, and to scan the common all-ASCII case a word at a time.

## Member Function Documentation

## ◆ DisableCharLevel()

void opencc::internal::Utf8SkipTable::DisableCharLevel | ( |  | ) |
| inline

Permanently falls back to lead-byte filtering, e.g.

when a key's first character is truncated or invalid and cannot be represented as a code point. Byte-level candidates accumulated so far remain valid.

## ◆ Finalize()

void opencc::internal::Utf8SkipTable::Finalize | ( |  | ) |
| inline

Must be called after candidate[] is filled and before the table is used.

Ideographic description operators must always stop the scan: the conversion loop groups IDS sequences without consulting the dictionary, and skipping over an operator would lose that grouping. The operator set is defined by UTF8Util::IdeographicDescriptionOperatorArity() and bounded by UTF8Util::kFirst/kLastIdeographicDescriptionOperator.

## Member Data Documentation

## ◆ bmpCandidates

std::vector<uint64_t> opencc::internal::Utf8SkipTable::bmpCandidates

Optional character-level refinement: one bit per BMP code point (U+0000..U+FFFF, 8 KiB).

When non-empty, 2- and 3-byte UTF-8 characters are filtered by their exact code point instead of their lead byte, so e.g. a rare CJK character sharing its lead byte with common dictionary keys can still be skipped. 4-byte characters keep lead-byte filtering. Character-level mode is on exactly when this vector is non-empty; there is deliberately no separate flag to keep in sync.

The documentation for this struct was generated from the following file:

- Utf8SkipScan.hpp