This package is considered a duplicate. The official version of this package is found at:https://macmillancontentscience.r-universe.dev/morphemepiece

Package: morphemepiece 1.2.3

Jonathan Bratt

morphemepiece: Morpheme Tokenization

Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.

Authors:Jonathan Bratt [aut, cre], Jon Harmon [aut], Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning [cph]

morphemepiece_1.2.3.tar.gz
morphemepiece_1.2.3.zip(r-4.7)morphemepiece_1.2.3.zip(r-4.6)morphemepiece_1.2.3.zip(r-4.5)
morphemepiece_1.2.3.tgz(r-4.6-any)morphemepiece_1.2.3.tgz(r-4.5-any)
morphemepiece_1.2.3.tar.gz(r-4.7-any)morphemepiece_1.2.3.tar.gz(r-4.6-any)
morphemepiece_1.2.3.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
card.svg |card.png
morphemepiece/json (API)
NEWS

# Install 'morphemepiece' in R:

install.packages('morphemepiece', repos = c('https://jonthegeek.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/macmillancontentscience/morphemepiece/issues

On CRAN:

5.04 score 11 stars 9 scripts 243 downloads 10 exports 37 dependencies

Last updated from:bc071b1a03. Checks:7 NOTE, 2 OK. Indexed: no.

Target	Result	Time
linux-devel-x86_64	NOTE	128
source / vignettes	OK	193
linux-release-x86_64	NOTE	157
macos-release-arm64	NOTE	77
macos-oldrel-arm64	NOTE	102
windows-devel	NOTE	80
windows-release	NOTE	76
windows-oldrel	NOTE	78
wasm-release	OK	117

Exports:load_lookup load_or_retrieve_lookup load_or_retrieve_vocab load_vocab morphemepiece_cache_dir morphemepiece_lookup morphemepiece_tokenize morphemepiece_vocab prepare_vocab set_morphemepiece_cache_dir

Dependencies:bit bit64 cachem cli clipr cpp11 crayon digest dlr fastmap fastmatch fs glue hms lifecycle magrittr memoise morphemepiece.data piecemaker pillar pkgconfig prettyunits progress purrr R6 rappdirs readr rlang stringi stringr tibble tidyselect tzdb utf8 vctrs vroom withr

Generating a Vocabulary and Lookup

Rendered fromgenerating_vocab.Rmdusingknitr::rmarkdownon May 26 2026.

Last update: 2021-10-26
Started: 2021-07-29

Testing the fall-through algorithm

Rendered fromalgorithm_test.Rmdusingknitr::rmarkdownon May 26 2026.

Last update: 2021-09-06
Started: 2021-07-29

Citation

Development and contributors

Readme and manuals

Help Manual

Help page	Topics
morphemepiece: Morpheme Tokenization	morphemepiece-package
Load a morphemepiece lookup file	load_lookup
Load a lookup file, or retrieve from cache	load_or_retrieve_lookup
Load a vocabulary file, or retrieve from cache	load_or_retrieve_vocab
Load a vocabulary file	load_vocab
Retrieve Directory for Morphemepiece Cache	morphemepiece_cache_dir
Tokenize Sequence with Morpheme Pieces	morphemepiece_tokenize
Format a Token List as a Vocabulary	prepare_vocab
Set a Cache Directory for Morphemepiece	set_morphemepiece_cache_dir