camxes-rs is a PEG parser generator with the camxes Lojban grammars built in,
plus an optional second engine. Each engine is a Cargo feature (peg, on by
default, and earley); at least one must be enabled:
- PEG engine (
peg, default) — packrat-style PEG interpreter: ordered choice, greedy repetition, no backtracking into a committed choice. - Earley engine (
earley) — the vendored CLL (Complete Lojban Language) pipeline of tergenzalvi insrc/cll: Earley-parsed CFG word shapes + CLL constraint filters + stream segmentation for morphology, and the CLL Chapter 21 EBNF grammar (src/cll/data/cll21.ebnf) over selma'o tokens for syntax. It does not use the camxes PEG grammars. It also accepts the older CLL word forms (obsolete words such askuarka), flagged by rule.
Both engines are exposed through the same Lojban APIs: morphology::analyze
(word split + classes) and syntax::parse / syntax::accepts (whole texts).
API documentation: docs.rs/camxes-rs.
[dependencies]
camxes-rs = "2.0.0" # PEG engine only
# camxes-rs = { version = "2.0.0", features = ["earley"] } # PEG + Earley / CLL
# camxes-rs = { version = "2.0.0", default-features = false, features = ["earley"] } # Earley only| Cargo feature | Default | Adds |
|---|---|---|
peg |
yes | the PEG engine (peg::grammar::Peg), the camxes grammar texts (grammars), the grammar AST (peg::rule), the stack guard (peg::stack), Engine::Peg for morphology and syntax |
earley |
no | the vendored cll module (CLL morphology, cmavo lexicon, Chapter 21 parser) and Engine::Earley for morphology and syntax |
Always compiled: the engine-independent morphology and syntax APIs and
the parse-tree types (peg::parsing). Building with neither feature is a
compile error. Engine::default() is Engine::Peg when peg is enabled,
else Engine::Earley.
| Path | What it is |
|---|---|
src/peg/** |
PEG engine: grammar/ (Peg::new, Peg::parse, bootstrap grammar of the PEG notation), rule/ (rule tree and its matcher), transformer/ (grammar text → rules), parsing.rs (ParseResult, ParseNode, Span, errors), annotations.rs (#: PUSH/#: PEEK-DIFF/#: POP-EQ handling), stack.rs (stack guard). Everything except parsing.rs needs feature peg |
src/grammars.rs |
Embedded PEG grammars (feature peg): LOJBAN_MORPHOLOGY_GRAMMAR (camxes word-level morphology, start text), LOJBAN_GRAMMAR (full camxes syntax from ilmentufa, start text) |
src/morphology/mod.rs |
Engine-independent morphology API and result types (analyze, Morphology, MorphWord, WordClass, WordDialect, Dialect, Engine) |
src/morphology/peg.rs |
PEG adapter (feature peg): LOJBAN_MORPHOLOGY_GRAMMAR on the PEG engine |
src/morphology/tree.rs |
Words read off a camxes morphology parse tree (PEG adapter) |
src/morphology/earley.rs |
Earley adapter (feature earley): CLL word analysis and segmentation from src/cll, tergenzalvi MorphClass → camxes word type, rule-based standard/obsolete decision |
src/syntax.rs |
Whole-text syntax check (parse, accepts): LOJBAN_GRAMMAR on the PEG engine, cll21.ebnf on the Earley engine |
src/cll/** |
Vendored tergenzalvi (feature earley): phonology/ (letters, diphthongs, consonant pair/triple tables, CLL ch. 3), shapes/ (CV shape recognition), cfg/ (morphology CFG built in cfg/grammar.rs + its Earley recognizer), constraints/ (CLL ch. 4 filters: tosmabru, slinku'i, lujvo hyphens, rafsi/fu'ivla, stress, pauses, class exclusion), lex/ (stream segmentation, tokens), tables/ (rafsi data), grammar/ (CLL ch. 21: cmavo lexicon, lexemes, EBNF compiler, Earley parser, elision, SI/SA/SU erasure), forms.rs (cmavo break-up rule, CLL §4.6/§4.7) |
src/cll/data/cll21.ebnf |
CLL Chapter 21 EBNF grammar (compiled at runtime by cll::grammar) |
src/cll/data/cmavo_selmaho.tsv |
cmavo → selma'o lexicon (used for MorphWord::selmaho and the ch. 21 lexer) |
src/examples/cmaxes-test.rs |
Example: run LOJBAN_MORPHOLOGY_GRAMMAR over examples/lujvo_tests.tsv |
examples/morphology_corpus.rs |
Example: PEG vs Earley over the dictionary corpus, timings, per-word report |
tests/grammar_basics.rs |
Syntax test: basic constructions on both engines, zoi/la'o delimiters |
tests/dictionary_corpus.rs, tests/data/ |
Dictionary corpus + committed morphology parity expectations (repository only, not in the crate package) |
benches/parser.rs |
Criterion benchmarks (grammar startup, parsing) |
- PEG grammar and morphology:
src/grammars.rs(LOJBAN_GRAMMAR,LOJBAN_MORPHOLOGY_GRAMMAR), run bysrc/peg/**; morphology adaptersrc/morphology/peg.rs. - Earley grammar and morphology: only the vendored tergenzalvi code in
src/cll/**. Morphology: the word-shape CFG insrc/cll/cfg/grammar.rs, the filters insrc/cll/constraints/, segmentation insrc/cll/lex/, camxes-compatibility rules insrc/cll/camxes.rs; adaptersrc/morphology/earley.rs. Syntax: the CLL Chapter 21 grammarsrc/cll/data/cll21.ebnfover the selma'o ofsrc/cll/data/cmavo_selmaho.tsv(camxes_rs::cll::grammar).
Grammars are plain text, one rule per line, loaded at runtime with
Peg::new(start_rule, text). The embedded grammars are (start_rule, text)
tuples.
| Syntax | Meaning |
|---|---|
rule <- e |
definition |
e1 e2 |
sequence |
e1 / e2 |
ordered choice: e2 is tried only if e1 fails |
e*, e+, e? |
greedy repetition / option (never gives back input) |
&e, !e |
and/not predicates (zero-width lookahead) |
( … ) |
grouping |
'abc', "abc" |
literal |
[abc], [a-z] |
character class / range |
. |
any character |
rule <- e #: PUSH |
back-reference annotation: PUSH, PEEK-DIFF, POP-EQ (see below) |
The grammar text has no comment syntax apart from these annotations.
Back-references (ilmentufa's annotations, used by camxes for
zoi/la'o): a #: PUSH rule remembers the word it matched, a
#: PEEK-DIFF rule fails if its match equals the remembered word, and a
#: POP-EQ rule matches only if its match equals it (then forgets it).
Words are compared after normalization (ASCII case, pauses ., commas and
spaces ignored, h ≡ '; src/peg/annotations.rs). The PEG engine keeps a delimiter stack that is restored when a
sequence fails and after predicates; PEEK-DIFF / POP-EQ rules are not
memoized. A PUSH rule must be closed by a POP-EQ rule in the same rule
body. So in camxes (zoi_word spaces)* zoi_close stops at the first word
equal to the opening delimiter instead of swallowing it.
ParseResult(_, end, result) gives the end of the match of the start rule
(the PEG matches a prefix) and the tree of ParseNode::NonTerminal /
ParseNode::Terminal nodes with byte spans.
#[cfg(feature = "peg")]
{
use camxes_rs::peg::grammar::Peg;
let grammar = r#"
expression <- term (('+' / '-') term)*
term <- factor (('*' / '/') factor)*
factor <- number / '(' expression ')'
number <- [0-9]+
"#;
let parser = Peg::new("expression", grammar).unwrap();
let result = parser.parse("2+3*4");
assert!(result.2.is_ok());
assert_eq!(result.1, 5); // bytes matched
}The vendored tergenzalvi code follows the CLL text rather than camxes:
- Morphology (CLL ch. 3–4): a context-free overgenerating grammar of
word shapes (cmavo, gismu, lujvo, fu'ivla, cmevla), built in
src/cll/cfg/grammar.rsand recognized with its own Earley parser, is intersected with constraint filters that each cite a CLL section (src/cll/constraints/: tosmabru and slinku'i tests, lujvo hyphenation, rafsi/fu'ivla shape rules, stress and pause handling, class exclusion). Continuous text is segmented bycll::lex::stream. - Syntax (CLL ch. 21):
src/cll/data/cll21.ebnfis compiled at runtime into a CFG over selma'o terminals (cll::grammar::compile_ebnf), with CLL's...repetition,[ ]options,/X#/elidable terminators and&selections; text is lexed into selma'o viacmavo_selmaho.tsv, SI/SA/SU erasure runs on the lexeme stream, andzoi/la'oquotes are cut out before morphology (the body is one opaqueFOREIGNlexeme; the quote closes at the first word equal to the delimiter).syntax::parsereturns the Chapter 21 derivation asParseNodes named after the EBNF rules and selma'o (sumti,KOhA,BRIVLA,FOREIGN, …); a text with a word the CLL morphology rejects is a syntax error.
No local changes. src/cll is identical to tergenzalvi's src/ apart
from crate paths and the data-file location; scripts/check_cll_sync.sh [PATH_TO_TERGENZALVI] diffs the two trees with that glue normalised and
prints nothing but an "in sync" line when they match. Where CLL is silent,
both repositories fill the gap the same way (each case cites the CLL text
it relies on): experimental cmavo with three or more vowels and y
apostrophe chains (§4.2), apostrophe vowel chains in brivla (§3.4), y
chains in names (§4.8), in the camxes dialect (see Dialect
options) lujvo recognized by rafsi form with the
§4.5 restrictions, consonant-only names such as
.d. or .rl. when every adjacent pair and triple is permissible (§3.7;
mz, ndj are not), the literal slinku'i test of §4.7 rule 3, the §4.6/§4.7 rule
that a word breaking up into a cmavo followed by cmavo/gismu/lujvo is not a
brivla (lerenanmu = le + re + nanmu), digits as PA cmavo, and
punctuation-only tokens (-) skipped. A text with a word the morphology
rejects has no derivation. tergenzalvi's MorphClass is mapped to the camxes
word types outside src/cll, in src/morphology/camxes.rs; the
words on which the two engines still disagree are listed in
Differences between the PEG (camxes) and Earley (CLL)
engines.
#[cfg(feature = "earley")]
{
use camxes_rs::cll::grammar::parse_text;
use camxes_rs::cll::shapes::{recognize, MorphClass};
assert!(recognize("creka").classes.contains(&MorphClass::Gismu));
assert!(parse_text("mi viska le gerku").accepted);
assert!(!parse_text("zoi gy. hello .dy cu glico").accepted);
}The CLL pipeline takes a MorphOptions (re-exported as
camxes_rs::morphology::MorphOptions) with one dialect switch,
camxes_dialect, plus the separate §4.16 proposal flag rafsi_fuhivla:
| Setting | Strict CLL (camxes_dialect: false) |
camxes dialect (camxes_dialect: true) |
|---|---|---|
| Lujvo rafsi | only rafsi of the CLL-era (1994) list | also rafsi outside the list, recognised by §4.5 shape (CVC, CVV/CV'V, CCV, 4-letter + y, final 5-letter); the §4.6 hyphen and §4.7 tosmabru/slinku'i rules still apply |
| Brivla-stem rafsi | none | as in camxes: a consonant-final brivla/fu'ivla stem + y (afxanyru'u), a vowel-final stem + 'y (baxmasi'ygu'e), y' before a vowel-initial piece (pedy'utka), a brivla-shaped last piece (bemymantodi); pieces keep §3.7 clusters and §3.4–3.5 vowel groups |
| Names | §4.8: no la, lai, doi inside a name (volapuk → vo la puk) |
la, lai, doi allowed (volapuk, laus, tigbalan) |
They always change together: there is no way to get one without the
other. Presets: MorphOptions::default() (strict CLL + §4.16, tergenzalvi's
default), MorphOptions::settled() (strict CLL, no §4.16) and
MorphOptions::camxes() (camxes dialect + §4.16); with_camxes_dialect(bool)
sets the switch on any of them. Engine::Earley uses
morphology::earley_options(), which is camxes(), so it agrees with the
PEG engine on names, post-1994 rafsi and brivla-stem rafsi; morphology::analyze_with_options
and syntax::parse_with_options take strict CLL or any other set. The
switch only decides which words are accepted and how text splits;
WordDialect (next section) stays rule-based on the camxes syllable rules,
so volapuk is a standard cmevla, and a word is never Obsolete merely
because the camxes dialect admitted it. Consonant-only names (below),
vowel grouping (§3.4–3.5, with on-glides) and commas as syllable breaks
(§3.3) are the same in both dialects.
#[cfg(feature = "earley")]
{
use camxes_rs::morphology::{analyze, analyze_with_options, Engine, MorphOptions};
// Engine::Earley (camxes dialect): a name with `la`, a lujvo with the
// post-CLL rafsi `gam`.
assert_eq!(analyze("volapuk", Engine::Earley).unwrap().render(), "volapuk/cmevla");
assert_eq!(analyze("gamkemdau", Engine::Earley).unwrap().render(), "gamkemdau/lujvo");
// Strict CLL turns off both at once.
let strict = MorphOptions::default();
assert!(analyze_with_options("volapuk", strict).map_or(true, |m| m.render() != "volapuk/cmevla"));
assert!(analyze_with_options("gamkemdau", strict).map_or(true, |m| m.render() != "gamkemdau/lujvo"));
}Engine::Earley decides each word's WordDialect by rule, without word
lists:
the CLL pipeline analyses the input (Dialect::Cll), and each word is
Obsolete when it breaks the camxes syllable rules
(is_obsolete_form in src/morphology/camxes.rs, a port of the camxes syllables /
zifcme rules: names must be camxes zifcme, other words must start with a
camxes onset and split into camxes syllables), Standard otherwise. On the
dictionary corpus a word is obsolete exactly when the PEG engine rejects it.
Typical obsolete forms: on-glide diphthongs iV/uV inside brivla and names
(kuarka, tiuanJOUS), n before an affricate (djinundji; allowed by CLL
§3.7, banned by camxes), diphthong + apostrophe + vowel (gondii'i) and
CiV'V'V cmavo (cie'a'u). Gismu rules are the same in both dialects, so no
gismu is ever obsolete; words the PEG engine accepts are always standard.
morphology::analyze(input, engine) returns a normalized Morphology, the
same shape for both engines (internal trees are not exposed):
words: Vec<MorphWord>— each withtext, bytespan,class(WordClass::{Cmavo, Gismu, Lujvo, Fuhivla, Cmevla}),selmaho(Some("KOhA")etc. for cmavo when theearleyfeature provides the lexicon, elseNone) andword_dialect(Standard/Obsolete); helpersis_obsolete()andtype_label()("gismu","obsolete fu'ivla", …).dialect: Dialect—Camxes(PEG engine) orCll(Earley engine).signature()(engine-comparable(text, class)list),has_obsolete(),render()(le/cmavo nu/cmavo klama/gismu).
#[cfg(feature = "peg")]
{
use camxes_rs::morphology::{analyze, Dialect, Engine, WordClass};
let m = analyze("lenu klama", Engine::Peg).unwrap();
assert_eq!(m.render(), "le/cmavo nu/cmavo klama/gismu");
assert_eq!(m.dialect, Dialect::Camxes);
assert_eq!(m.words[2].class, WordClass::Gismu);
assert!(analyze("kuarka", Engine::Peg).is_err()); // obsolete form
}#[cfg(feature = "earley")]
{
use camxes_rs::morphology::{analyze, Dialect, Engine, WordDialect};
// Same words and classes as the PEG engine for standard words …
let m = analyze("lenu klama", Engine::Earley).unwrap();
assert_eq!(m.render(), "le/cmavo nu/cmavo klama/gismu");
assert_eq!(m.words[0].selmaho.as_deref(), Some("LE"));
assert_eq!(m.dialect, Dialect::Cll);
// … and the older CLL dialect, flagged by rule.
let m = analyze("lojbo zei kuarbli", Engine::Earley).unwrap();
let labels: Vec<String> = m.words.iter().map(|w| w.type_label()).collect();
assert_eq!(labels, ["gismu", "cmavo", "obsolete fu'ivla"]);
assert_eq!(m.dialect, Dialect::Cll);
assert_eq!(m.words[2].word_dialect, WordDialect::Obsolete);
assert!(m.has_obsolete());
}Engine::available() lists the engines compiled in; "peg" / "earley"
parse with str::parse::<Engine>().
syntax::parse(text, engine) checks a complete text and returns the tree or
a SyntaxError with a position; syntax::accepts returns a bool. The PEG
engine runs LOJBAN_GRAMMAR (letters are lowercased first; the start rule
text must cover the input); the Earley engine runs the CLL Chapter 21
grammar src/cll/data/cll21.ebnf (start text) over selma'o tokens, with
elidable terminators. The two engines' trees differ (camxes vs Chapter 21
rule names).
use camxes_rs::morphology::Engine;
use camxes_rs::syntax;
let engine = Engine::default(); // `Engine::Peg` with the default features
assert!(syntax::accepts("lo prenu poi klama cu tavla", engine));
assert!(!syntax::accepts("lo cu klama", engine));
let tree = syntax::parse("mi klama", engine).unwrap();
assert!(!tree.is_empty());
// zoi/la'o quotes need the same delimiter word on both sides (both engines).
for &engine in Engine::available() {
assert!(syntax::accepts("zoi gy. hello world .gy glico", engine));
assert!(syntax::accepts("mi klama la'o dy. New York .dy", engine));
assert!(!syntax::accepts("zoi gy. hello .dy glico", engine));
assert!(!syntax::accepts("mi cusku zoi gy. hello", engine));
}
#[cfg(feature = "earley")]
assert!(syntax::accepts("lo prenu poi klama cu tavla", Engine::Earley));tests/grammar_basics.rs lists the constructions tested on both engines.
# PEG morphology grammar over examples/lujvo_tests.tsv (or a TSV path)
cargo run --example cmaxes-test [path.tsv]
# PEG vs Earley morphology over the dictionary corpus, with timings;
# --raw times the bare word recognisers (camxes PEG grammar vs CLL
# morph_parse), --words w1,w2 checks single words
cargo run --release --features earley --example morphology_corpus -- [--raw] [--words w1,w2]morphology_corpus also writes a per-word report (word, dictionary type,
PEG result, Earley result, status) to target/corpus-results.tsv.
cargo test # PEG engine
cargo test --release --features earley # both engines + corpus parity (release: the CLL pipeline is slow in debug)
cargo bench [--features earley] # criterion benchmarks
cargo test --no-default-features --features earley # Earley-only build
python3 scripts/fetch_lensisku_corpus.py # refresh tests/data from lensiskutests/dictionary_corpus.rs runs every lensisku English headword (wiki
page-title entries excluded; tests/data/lensisku-en-words.tsv.gz) through
both engines and compares with tests/data/morphology-parity.tsv, which
records for each headword the expected words, classes, obsolete flags and
selma'o, each engine's acceptance, the rule-based dialect, and whether the
dictionary's obsolete label agrees with it (label_check). The engines
must agree on every PEG-accepted headword except the documented known
differences in tests/data/known-differences.tsv (word, category, exact
Earley result; see Differences between the PEG (camxes) and Earley (CLL)
engines); a
new difference, a changed Earley result or a listed word that now agrees
fails the test. After an
intended change, regenerate it and review the diff:
CAMXES_BLESS_PARITY=1 cargo test --release --features earley --test dictionary_corpusbenches/parser.rs measures grammar construction and parsing of short and
medium Lojban texts: LOJBAN_GRAMMAR on the PEG engine; with
--features earley, compiling cll21.ebnf and syntax::parse on the
Earley engine. The test data, corpus test, corpus example and
scripts live in the repository only and are excluded from the crates.io
package.
The PEG engine logs through the log crate:
env_logger::builder().filter_level(log::LevelFilter::Debug).init();or RUST_LOG=debug cargo run --example cmaxes-test.
- Back-references: delimiters are compared after normalization (case,
pauses, commas,
h/'). The PEG engine checksPEEK-DIFF/POP-EQwith its delimiter stack (PUSH and POP-EQ rules in the same rule body); the Earley engine cutszoi/la'oquotes out before lexing, closing at the first word equal to the delimiter. - Word types: on the dictionary corpus the engines disagree on 37 of the 31,219 PEG-accepted headwords; each is a documented camxes/CLL difference (next section).
- Trees: the PEG tree uses camxes rule names, the Earley tree CLL Chapter 21 rule names; they are not comparable. The parity test compares word split, classes and selma'o only.
- Dictionary labels: 226 headwords labelled obsolete in lensisku have a
standard form (e.g.
matriocka) and one standard-labelled word (gudjrati) is obsolete by rule; the parity file records these aslabel_checkmismatches. - The vendored CLL Chapter 21 layer approximates some BU/ZEI interactions;
texts that the Chapter 21 grammar and camxes judge differently are listed
by the
dwecomparison (see Performance). - Errors, not panics: malformed grammar text (including non-ASCII) is a
GrammarError/CompileError, and no input string makes a parse panic (tests/robustness.rs). Recursion deeper than the current thread's stack allows (very deep nesting, left-recursive rules) fails withErrorKind::RecursionLimit; the stack size is queried on Linux and macOS, elsewhere 1 MiB is assumed (peg::stack,Peg::set_stack_budget). Worst-case cost is still high: the vendored CLL word analysis is cubic in the length of one unbroken word, so cap untrusted input length. Pegmemoizes in aRefCelland is notSync; use one parser per thread (assyntaxdoes). The CLL pipeline caches word analyses per thread.
The PEG engine implements camxes; the Earley engine implements the text of
the CLL (Complete Lojban Language) and fills gaps only where the CLL is
silent. Where they disagree, the Earley engine is not changed to match
camxes. Each row of tests/data/known-differences.tsv has one of the
categories below (the same reasons as DIFFERENCE_CATEGORIES in
tests/dictionary_corpus.rs). Counts are PEG-accepted dictionary headwords
(37 in total).
| Category (count) | Kind | Examples (PEG → Earley) | CLL |
|---|---|---|---|
medial-pair (1) |
camxes bug | vamzma: lujvo → rejected |
§3.7: "the specific pairs cx, kx, xc, xk, and mz are forbidden"; the camxes morphology grammar does not check mz (the voicing clashes zt, fg of ciztolnei, refgau are now rejected by both engines, see below) |
hesitation (2) |
CLL rule | y → rejected; yyy: y + yy → rejected |
§19.14: .y. "differs from [English uh] in the requirement for pauses before and after" |
vowel-initial-pause (1) |
camxes bug | aierne: a + ierne → one fu'ivla aierne |
§4.9 rule 3: a word beginning with a vowel must be preceded by a pause, so a + ierne is not a valid split |
fuhivla-cluster (13) |
CLL rule (kept) | ku'aidzi, fau'arma, ri'oirkai, xai'otso → rejected |
§4.7 rule 1: every fu'ivla "must contain a consonant cluster in the first five letters of the word". The apostrophe is not a consonant (§3.6) and fu'ivla cannot contain y (rule 4), so in ku'aidzi the first five letters are k u ' a i and the only cluster, dz, is at letters 6–7: not a fu'ivla. camxes accepts a cluster anywhere; this stays a documented difference |
hyphen-not-required (20) |
CLL rule | geirzdi, toirsti, selda'ergau, relpi'irna'u: lujvo → fu'ivla |
§4.6: "It is illegal to add a hyphen at a place that is not required by this algorithm"; an r/n-hyphen follows only a CVV rafsi that needs it |
Resolved categories (both engines now agree):
- Brivla/fu'ivla-stem rafsi (former
rafsi-vowel-initial98,rafsi-shape55,y-after-vowel16,n-hyphen2). In the camxes dialect (Engine::Earley) a lujvo may also use the stem of any brivla as a rafsi, as the camxes grammar does (fuhivla_rafsi,hy_rafsi,y h?, a finalfuhivla): a consonant-final stem +y(afxan-y-ru'u,kexmair-y-bau), a vowel-final stem +'y(baxmasi-'y-gu'e),y'before a vowel-initial piece (ped-y'-utka) and a brivla-shaped last piece (bem-y-mantodi). Each piece must have §3.7 clusters, §3.4–3.5 vowel groups and apostrophes only between vowels; a vowel-initial piece needs a word start, an apostrophe or an on-glide (batky-uidje); short rafsi between stems keep the §4.6r/nhyphen rule. Strict CLL (MorphOptions::default()) still rejects all of them (src/cll/constraints/camxes_lujvo.rs). - Vowel grouping (former
vowel-grouping15). Vowels group from the left (§3.5), andi/ubefore a vowel is an on-glide starting the next syllable (§3.4: on-glides are "used only as stand-alone words and in Lojbanized names and borrowings"), sogleua=gle·ua,proteuu,paxtoua,iuctcouoare fu'ivla in both dialects. A pair that is neither a diphthong nor vowel + glide (ea) still needs a comma or apostrophe. - Comma (former
comma7). The camxes morphology grammar treated a comma as a pause (pause_0 <- [ ,.]+), solijdrba'a,iwaslijdrba'a+i. It now treats a comma as a syllable break inside a word (CLL §3.3): before a vowel it separates syllables (h <- [,'] &nucleus), before a consonant or an on-glide it is skipped, and only a comma next to a space or the end of the input is a pause.lijdrba'a,i,OTa,uas,bodga,ias,nu,IORK,tai,UAN,xali,uyd,klanrniut,niare now one word in both engines (the Earley engine also retries a word without the commas that do not separate vowels). The only other dictionary change isbie,uaRUC, now accepted by the PEG engine (a standard name). - Earlier:
name-la(8) andconsonant-name(5) — the camxes dialect allowslain names, and consonant-only names with permissible clusters (§3.7) are accepted; the voicing clashesciztolnei,refgauare rejected by both engines.
The other direction: 173 dictionary headwords are accepted only by the
Earley engine. They are the older CLL dialect ("obsolete" words such as
kuarka, tiuanJOUS, djinundji, gondii'i), flagged
WordDialect::Obsolete by rule (see Standard vs obsolete words).
Whole texts (dwe corpus). On the 1,699-text dwe comparison corpus
the PEG engine accepts 1,678 texts and the Earley engine 1,676 (all of them
also PEG-accepted). The two texts only the PEG engine accepts (reported, not
fixed):
ti marce ge le se vecnu na'o ku gi le remna ca'a ku— Earley: "no CLL Chapter 21 derivation oftext". A forethought termset (ge … gi …joining terms) needsnu'iin the CLL grammar (Ch. 14/21termset); camxes allows it without.zo co'oi poi do pu cusku cu ranxi le ka se smuni zo coi vau le ka se smuni zo co'o— Earley:cmavo "co'oi" not in lexicon.co'oiis not in the CLL cmavo table (cmavo_selmaho.tsv); camxes accepts any cmavo-shaped word.
The texts ending in zo d and la c are now accepted by both engines
(consonant-only names, CLL §3.7).
Measured on the build machine (release build, single thread) over the 31,414
lensisku headwords with examples/morphology_corpus.rs:
| Workload | PEG | Earley (CLL pipeline) |
|---|---|---|
morphology::analyze, all headwords |
1,029 words/s | 343 words/s |
bare word recogniser (--raw: camxes grammar / cll::constraints::morph_parse) |
1,118 inputs/s | 296 inputs/s |
Whole-text syntax (cargo bench --features earley; the Earley numbers are
warm-cache: the CLL word analysis is cached per thread):
| Benchmark | PEG | Earley |
|---|---|---|
grammar startup (Peg::new(LOJBAN_GRAMMAR) / compile_ebnf(cll21.ebnf)) |
195 ms | 1.2 ms |
mi prami do (11 bytes) |
5.6 ms | 0.47 ms |
mi prami do .i do prami mi × 20 (540 bytes) |
100 ms | 0.58 ms |
1,699 example sentences (dwe corpus, syntax::parse, cold cache) |
53 rows/s | 433 rows/s |
Compile a grammar once and reuse it (syntax caches both). The slowest
headwords for the CLL pipeline are very long cmavo clusters (0.5 s); long
camxes lujvo, which the CLL pipeline used to break up word by word (up to
3.5 s), are now read whole as brivla-stem lujvo, which also raised the
Earley analyze rate from 214 to 343 words/s on the same machine. Numbers
vary with machine load.
Results: PEG accepts 31,219 headwords, Earley 31,376; the engines agree on word split and class on 31,182, the other 37 PEG-accepted headwords are the documented known differences; 173 headwords are accepted only by Earley (obsolete CLL forms) and 22 by neither.
MIT — see LICENSE. The camxes grammars are derived from
ilmentufa / camxes.js
(© Masato Hagiwara, MIT); src/cll is vendored from
tergenzalvi by the same author as
this crate. PEG: Bryan Ford, Parsing Expression Grammars.