Repository navigation
Install hyphen-* packages when babel can't define the language - #516
Conversation
…515) With some babel options (e.g. nswissgerman from `lang: de-CH`), missing hyphenation patterns trigger the fatal error "You haven't defined the language 'ngerman' yet" instead of a warning, so check_extra() never ran and parse_packages() found nothing to install. Detect this error in parse_packages() and map the language to its hyphen-* package. The mapping from language names to hyphen-* packages is now shared by check_babel(), check_polyglossia() and the new detect_hyphen(), and also covers pattern names that differ from package names in texlive.tlpdb (e.g. ngerman/swissgerman -> hyphen-german).
Language names parsed from a LaTeX log end up as package names passed to tlmgr via system2(), which uses a shell on Unix. hyphen_pkgs() now only returns names matching ^hyphen-[a-z0-9-]+$, which covers every hyphenation name in language.dat and texlive.tlpdb (checked against the full CTAN tlpdb). check_babel() and check_polyglossia() no longer call tlmgr_install() when nothing is left, since tlmgr_install(character()) returns 0 and would trigger a pointless rerun.
assamese, bengali, gujarati, hindi, kannada, malayalam, marathi, oriya, pali, panjabi, tamil, and telugu are all defined by the hyphen-indic package, so e.g. hindi used to lead to the nonexistent hyphen-hindi.
amharic, geez, persian, patois, francais, polygreek, portuges, slovene, and espanol are synonyms in texlive.tlpdb for languages in the hyphen-ethiopic, -farsi, -french, -greek, -portuguese, -slovenian, and -spanish packages, so they used to lead to nonexistent packages.
9a397e7 to
0997537
Compare
…ge (#15029) With `lang: de-CH` and no German hyphenation patterns installed, babel stops with a fatal "You haven't defined the language 'ngerman' yet" error. Missing patterns were only detected from babel warnings after a successful compile, so this failure path had no package to install. On this error, install the `hyphen-*` package for the language by name. Language names map to their package through a table built from the `execute AddHyphen` entries of texlive.tlpdb, so names such as `swissgerman` (hyphen-german) and `hindi` (hyphen-indic) resolve to the package that provides them. The table is shared with the existing babel warning detection. - Install the hyphenation package on the babel undefined-language error - Map 37 pattern and synonym names to the package that provides them - Drop language names that are not valid `hyphen-[a-z0-9-]+` packages - Add a smoke test that removes hyphen-german and babel-german before rendering, run on CI only See rstudio/tinytex#515 and rstudio/tinytex#516, which carry the same fix for the tinytex R package. Fixes #15027
| m = c( | ||
| ngerman = 'german', swissgerman = 'german', pinyin = 'chinese', | ||
| ukenglish = 'english', british = 'english', usenglishmax = 'english', | ||
| bokmal = 'norwegian', norsk = 'norwegian', nynorsk = 'norwegian', | ||
| schoolfinnish = 'finnish', monogreek = 'greek', ibycus = 'ancientgreek', | ||
| classiclatin = 'latin', liturgicallatin = 'latin', | ||
| mongolianlmc = 'mongolian', serbianc = 'serbian', | ||
| # older names (synonyms in texlive.tlpdb) | ||
| amharic = 'ethiopic', geez = 'ethiopic', persian = 'farsi', | ||
| patois = 'french', francais = 'french', polygreek = 'greek', | ||
| portuges = 'portuguese', slovene = 'slovenian', espanol = 'spanish', | ||
| # all of these are in the hyphen-indic package | ||
| assamese = 'indic', bengali = 'indic', gujarati = 'indic', hindi = 'indic', | ||
| kannada = 'indic', malayalam = 'indic', marathi = 'indic', oriya = 'indic', | ||
| pali = 'indic', panjabi = 'indic', tamil = 'indic', telugu = 'indic' | ||
| ) |
There was a problem hiding this comment.
Is there a way to retrieve these names programmatically (from language.dat/texlive.tlpdb) so we don't need to hardcode them here?
There was a problem hiding this comment.
Yes I did get all them from texlive.tlpdb but I think I needed to download it and it was not locally available in my install. Though it was quick.
So I can see if I can extract the logic to either generate this list on a schedule basis or before each release we do, instead of dynamically.
but you're right - this list came from an analysis of some data I made.
There was a problem hiding this comment.
I looked again at where these names came from, and I don't think we can read them from the local files.
language.dat and the local texlive.tlpdb only contain the packages that are installed. The package we are looking for is the missing one, so it is never in there. The full list is only in the remote texlive.tlpdb, in the execute AddHyphen name=... synonyms=... lines of each package.
tlmgr does keep a cached copy of the remote db (tlpkg/texlive.tlpdb.main.<md5>, ~20MB) and it has all the names. But we delete these files ourselves when building the bundles, and also with options(tinytex.delete_tlpdb = TRUE):
Lines 258 to 262 in 0997537
So we can't count on it, and we would still need the table as a fallback. Using tlmgr dump-tlpdb --remote instead would download the full db each time this error happens, which seems too much for this.
What we could do is generate the table instead: a script in tools/ that downloads texlive.tlpdb.xz from CTAN, parses the AddHyphen lines, and writes the table as R code, keeping only the names whose package is not hyphen-<name>. We could run it before each release, or in a scheduled workflow that opens a PR when the table changes. I tried a quick version and it gives the same 37 entries as here, plus the 4 german-x-* names from dehyph-exptl that I listed as not handled in the PR description.
I would do this in a follow-up PR, and we can discuss there the best way ?
The hyphenation table in parse-error.ts was a hand copy of tinytex's
and had drifted. With `\begin{hyphenrules}{ngerman-x-latest}` and no
dehyph-exptl installed, babel stops with "You haven't defined the
language 'ngerman-x-latest' yet". The package was always built as
`hyphen-<name>`, so tlmgr was asked for the non-existent
`hyphen-ngerman-x-latest` and the render failed.
Detection now follows tinytex main (rstudio/tinytex#517, follow-up to
rstudio/tinytex#516), and builds on #15029:
- The map holds full package names, with the 4 dehyph-exptl entries, and
the language name is validated instead of the resulting package name.
- A language given as a description keeps its first word, so "German
(current orthography)" gives hyphen-german.
- The babel regex only reads warning blocks, so a (babel) continuation
of an Info message can't produce a package.
- Every warned language is installed, not only the first, and
polyglossia warnings are detected.
- The daily TinyTeX workflow also compares tinytex's hyphen.json. It
reports a skipped comparison when the download fails, and an invalid
hyphen.json fails the first run instead of being cached.
The chinese skip list (#10291), the `\l@nil` fallback and unquoted names
in the babel error regex are kept.
Fixes #15035
See #15027, rstudio/tinytex#518
2 participants
When a document uses a babel option whose hyphenation patterns are not installed, e.g.
lang: de-CH(babel optionnswissgerman), babel can stop with an error instead of a warning:latexmk()only looked for the warning variant of this problem, so nothing was installed and the compilation failed. Installing hyphen-german by hand fixes it (#515).Checked against the full TeX Live database
Both the filter and the mapping table were checked against the full
texlive.tlpdbfrom CTAN: 65hyphen-*packages, 88AddHyphenlines and 98 distinct names once synonyms are included.hyphen_pkgs()does). Names inlanguage.datcan contain digits and hyphens (german-x-2024-02-28), so a letters-only pattern would have been wrong.hyphen-<name>.Not in this PR
check_glossaries()andcheck_datetime2()also pass names from the log totlmgr_install()without validation.tlmgr_install()andcheck_installed(). On Windowstlmgris a.batfile, so&,|and%would need separate handling.german-x-latest,ngerman-x-latestand their dated variants pass the filter but are not in ahyphen-*package. They are defined by dehyph-exptl, so they still give a package name that does not exist.Fixes #515