Type Editor

---

Type Editor / @type-editor/changeset / tokenizer/tokenize-fragment / tokenizeFragment

Function: tokenizeFragment()

function tokenizeFragment<T>(fragment, encoder, start, end, target): T[];

Defined in: tokenizer/tokenize-fragment.ts:19

Convert the given range of a fragment to tokens for diff comparison. Recursively processes the fragment tree, encoding text characters and node boundaries.

Type Parameters

Type Parameter
T

Parameters

Parameter	Type	Description
fragment	Fragment	The fragment to tokenize.
encoder	TokenEncoder<T>	The encoder to use for converting nodes and characters to tokens.
start	number	The start offset within the fragment.
end	number	The end offset within the fragment.
target	T[]	The array to append tokens to.

Returns

T[]

The target array with all tokens appended.