Skip to content

Repository files navigation

Signal K Plugin Registry

Automated testing and quality scoring for Signal K server plugins.

Results: https://signalk.org/signalk-plugin-registry/

What It Does

  • Discovers Signal K plugins and webapps from npm (keywords: signalk-node-server-plugin, signalk-webapp, signalk-embeddable-webapp — the ones the App Store lists by)
  • Tests each plugin against Signal K server: install, load, activate, detect providers, security audit
  • Stores results persistently in results.json — only retests on version changes
  • Publishes a static JSON API via GitHub Pages
  • Runs nightly or on manual trigger

Scoring

Each plugin is scored out of 100 points:

Tier What Points
Install npm install --ignore-scripts succeeds 20
Load Constructor returns a valid plugin object 15
Activate start() completes with schema defaults 15
Schema Plugin exposes a JSON configuration schema 5
Tests Plugin's own test suite passes 25
Security No npm audit vulnerabilities 20
Changelog CHANGELOG file or GitHub Release for the version −5 if missing
Screenshots signalk.screenshots array in package.json −5 if missing
Core dep freshness Declared ranges allow the latest same-major release of core Signal K packages −80 if held back
Legacy baconjs Any baconjs range in dependencies / peerDependencies can resolve to 3.x −15 if it cannot
Legacy React Embedded webapp bundles register React ≥19 −15 if <19
Bundled plugins No runtime dependency is another App Store plugin or webapp −30 if one is

Security scoring breakdown: 20 points for a clean audit, 15 if only moderate vulnerabilities, 10 if high (no critical), 0 if any critical vulnerabilities.

Changelog detection prefers a CHANGELOG.md (or CHANGES.md / HISTORY.md) in the published tarball; if absent, falls back to the repository's public GitHub Releases atom feed (https://github.com/<owner>/<repo>/releases.atom, no token needed) and looks for a release whose tag or entry title carries the installed version. Tags may carry a package-name prefix, so the monorepo conventions (pkg@1.2.3, pkg-v1.2.3, pkg/v1.2.3) are recognised alongside a plain v1.2.3. The version must stand on its own, so 1.2.3 never matches a 1.2.30 or 1.2.3-beta.1 release. See signalk-server PR #2615 for the release-notes convention.

Screenshots detection requires at least one string entry under signalk.screenshots in package.json.

A webapp without a server plugin — a package carrying only signalk-webapp or signalk-embeddable-webapp, not signalk-node-server-plugin — is scored on the same scale. It has nothing to load, activate or configure, so the Load, Activate and Schema tiers (35 points) are not applicable and count as passed, and it gets a webapp badge instead of loads / activates. Every other check applies as for a plugin. The plugin-ci penalty does not apply either, because plugin-ci.yml requires the plugin keyword. The server's own admin UI (@signalk/server-admin-ui) is not scored.

Test scoring runs the plugin's suite from its published tarball, then falls back to a shallow clone of the source repo (devDependencies aren't published, so most suites only run from source). The clone checks out the version being scored: the commit npm recorded when it was published (gitHead), else its release tag (v<version>, <version> or <name>@<version>), and only when neither exists the default branch. The tested commit and which of the three it came from are recorded as source_test_ref in the plugin's results and shown when hovering the test status. In the source fallback the harness honors the build/test commands a plugin declares to the canonical plugin-ci.yml reusable workflow, so it mirrors the CI it already tracks for the plugin-ci badge rather than guessing. This is what lets webapp-class plugins (Angular/React/Vue) be scored correctly — their plain build/test scripts are often a non-exiting bundler and a watch-mode runner, which a naive guess would score "not runnable". A declared command may be an npm script invocation (npm run <script>, npm test) or an && chain of them, and the time limit covers the whole chain. When a plugin declares no such commands, or one of another shape, the harness falls back to its own heuristics.

A plugin published from a subdirectory of its repository is scored there rather than at the repository root, provided it declares npm's standard repository.directory field. Test discovery, the build and the test run all happen in that subdirectory, so a sibling package's tests are never mistaken for the plugin's. Dependencies install at the repository root — which is what npm workspaces expect — and the subdirectory is installed separately when it isn't a workspace. Note that such plugins currently cannot run the reusable plugin-ci.yml workflow (it has no working-directory input), so they still take the no-plugin-ci penalty; that fix belongs upstream.

Core dep freshness checks the plugin's declared dependencies and peerDependencies (never devDependencies — those don't reach user installs) against the latest npm release of each core Signal K package: @signalk/server-api, @canboat/canboatjs, @canboat/ts-pgns, @signalk/n2k-signalk, @signalk/nmea0183-signalk, @signalk/streams. A range that targets the same major as the latest release but cannot resolve to it (e.g. ~2.9.0 or an exact 2.9.0 when latest is 2.30.0) holds the package back in every user's ~/.signalk install and costs −80. Ranges on an older major are not flagged (the plugin may legitimately not support the new major yet), and neither are git/file/URL specs, dist-tags, or invalid ranges. If the npm registry lookup fails, the check is skipped — a registry flake never costs points.

Legacy runtime deps cover two libraries the server provides at a fixed major and only bridges for older plugin builds through temporary compatibility shims. baconjs: the server moved to 3.x in 2.24.0 and redirects every require('baconjs') to its own copy (signalk-server#2487); a dependencies / peerDependencies range that cannot resolve to any 3.x release (e.g. ^0.7.88, ^1.0.1) costs −15. React: the admin UI is React 19 and renders Module Federation remotes built against React 16 through an isolated bridge (signalk-server#2451); for plugins with an embedded-webapp keyword (signalk-embeddable-webapp, signalk-plugin-configurator, signalk-node-server-addon) the remote entry the server injects into the admin UI (public/remoteEntry.js, or remoteEntry.js at the package root without a public/) and the chunks reachable from it are scanned for the shared React version they register, and a major below 19 costs −15. Files nothing references — stale builds, unrelated entries — are not consulted, and remotes that consume the host's React without shipping their own are not flagged. Both checks are pure file inspection — no plugin code runs — and an unreadable package or bundle is indeterminate, never a penalty.

Bundled plugins checks the plugin's dependencies, optionalDependencies and non-optional peerDependencies (npm installs all three) for packages the App Store lists — anything published with the signalk-node-server-plugin, signalk-webapp or signalk-embeddable-webapp keyword. npm installs such a dependency into the user's ~/.signalk tree at the range this plugin chose, where the server loads it as a plugin of its own: an old major (^2.0.2 when the plugin is at 3.x) runs there instead of the current release, and uninstalling the bundling plugin can take it away. Any such dependency costs −30, once per plugin. Declare companions in signalk.requires (the App Store offers to install them together with the plugin) or signalk.recommends (suggested on the detail page) instead — see the publishing guide. The check reads the package.json npm installed for each dependency, so git, tarball and npm: alias specs are judged by the package they actually install. A dependency on the plugin itself is not counted, and one that was not installed or has an unreadable manifest is skipped.

Plotter extensions — plugins that register a plotterExtensions resource provider to contribute widgets, panels, buttons and map layers to chart plotters such as Freeboard-SK — get the plotter-extension badge and two checks from the provider guide. A package without the signalk-category-plotter-extensions keyword costs −10: the keyword lists it in the App Store's Plotter Extensions category (server 2.36.0 and later) and is how plotters find installable extensions. While the plugin runs, the harness calls the provider's listResources() the way a host does and checks the manifests it returns; any of these costs −10 once: no manifest, a missing name, an apiVersion other than "1" (the host drops such a manifest), no contribution, a widget size other than 1x1/2x1/1x2/2x2, a configPanel, button or action panel or image-overlay runtime the manifest does not declare, or an asset URL that is not server-relative or sits under /plugins/, which the server serves to admins only. A plugin whose start() fails is not checked further, since activation already accounts for that. The keyword without a registered manifest is not penalised.

Provider detection (resources, weather, history, autopilot, radar) is tracked as an informational badge but does not affect the score — most plugins are not expected to register providers.

Badges

Badge Meaning
compatible Installs successfully
loads Plugin constructor succeeds
webapp Webapp without a server plugin; Load, Activate and Schema do not apply
activates start() completes without error (with schema defaults)
has-providers Registers at least one provider (informational)
tested Plugin has its own test suite and it passes
tests-failing Plugin has tests but they fail (-5 penalty)
npm-audit-ok No npm audit vulnerabilities (20 pts)
audit-moderate Has moderate vulnerabilities (15 pts)
audit-high Has high vulnerabilities (10 pts)
audit-critical Has critical vulnerabilities (0 pts)
has-changelog CHANGELOG file or matching GitHub Release is available
has-screenshots Declares at least one signalk.screenshots entry
holds-back-core-deps A declared range pins a core Signal K package below its latest same-major release (−80 penalty)
legacy-baconjs A dependencies / peerDependencies range for baconjs cannot resolve to 3.x (−15 penalty)
legacy-react Embedded webapp bundle registers React <19 (−15 penalty)
bundles-plugins A runtime dependency is another App Store plugin or webapp (−30 penalty)
plotter-extension Registers a plotterExtensions manifest (informational)
missing-plotter-keyword Registers a plotter extension without the signalk-category-plotter-extensions keyword (−10 penalty)
invalid-plotter-manifest The plotter extension manifest would be dropped by a host or names a panel, runtime or asset it cannot load (−10 penalty)
broken Failed to install

API

Results are published to GitHub Pages:

  • index.json — summary of all plugins, sorted by score
  • plugins/<name>.json — full detail for one plugin

Per-plugin records include upstream informational metrics fetched once per nightly run so signalk-server installs can display them without each boat hitting api.github.com (60/hr unauthenticated limit). Fields:

  • stars — GitHub repository stargazer count
  • open_issues — open issue count (includes PRs)
  • contributors — contributor count
  • downloads_per_week — npm weekly downloads
  • github_url — normalised https://github.com/<owner>/<repo> URL

All fields are optional — any combination of GitHub API limits, missing repositories, or npm throttling can leave them absent. Consumers should render — or a similar empty-state when a field is not present.

Manual Testing

Test a single plugin locally:

npm ci
npx ts-node test-harness/runner.ts <plugin-name> [version]

Test against a local plugin source:

npx ts-node test-harness/detect-providers.ts /path/to/plugin

Requesting a Re-Score

If you maintain a plugin and want a fresh score without waiting for the nightly run — for example right after publishing a fix — open a re-score request:

  1. Go to New issue and choose Request a plugin re-score.
  2. Enter your npm package name (e.g. signalk-my-plugin or @scope/signalk-my-plugin) and submit.

A bot validates the request, runs a single-plugin scan, comments the new score and badges on the issue, then closes it. This usually takes a few minutes.

Your plugin must be published to npm and carry the signalk-node-server-plugin keyword, or signalk-webapp / signalk-embeddable-webapp for a webapp (the keywords the Signal K app store uses to discover packages). If it isn't a recognised plugin, the bot replies explaining why and closes the issue without running a scan.

To re-run later on the same issue — after shipping another change — comment /rescore <npm-name>.

You don't need write access to this repository; anyone can open the request.

Triggering CI

Maintainers can also trigger a scan directly. Go to Actions > "Nightly Plugin Registry Scan" > Run workflow:

  • changed_only — only test plugins with new versions since last run
  • all_plugins — retest everything
  • single_plugin — test one specific plugin by npm name

CI Security Model

Plugins are untrusted code. The CI pipeline is designed so that even a deliberately malicious plugin cannot steal secrets, exfiltrate data, tamper with results, or attack third parties.

Job isolation

The workflow has four jobs. Only the last two have any permissions:

Job Permissions Runs plugin code?
plan {} (none) No
test {} (none) Yes
merge-results contents: write No
publish contents: write No

The test job uses persist-credentials: false so no token exists in git config either.

Network isolation (firejail)

All plugin code execution is wrapped in firejail --net=none:

  • require() + start() (provider detection) — runs as a sandboxed subprocess
  • npm test from published packages and from cloned source repos

This prevents plugin code from making any outbound network requests — no data exfiltration, no phoning home for second-stage payloads, no participation in attacks on third parties.

Filesystem isolation

Firejail runs with --read-only=/home --read-only=/etc --read-only=/var. Plugin code cannot modify the workspace, git history, results.json, or npm cache. Only /tmp (where plugin workdirs live) is writable.

The sandbox also runs with --ignore="noexec /tmp", which lifts the noexec /tmp that firejail's default profile would otherwise apply. Without it, plugins that load a native addon at require-time (sharp/libvips, canvas, prebuilt better-sqlite3, …) fail dlopen with failed to map segment from shared object and flatline at ~30 for an environment-only reason. This is a deliberate, scoped relaxation: the sandbox's whole job is to execute the plugin's own untrusted JavaScript, so the attacker already has arbitrary code execution before /tmp is touched — noexec /tmp only blocks the binary-mmap path while leaving the JS path open, so it adds no real boundary here. The load-bearing containment — --net=none and the read-only mounts — is unchanged.

SIGNALK_REGISTRY_TEST env var (for plugin authors)

The harness sets SIGNALK_REGISTRY_TEST=1 in the environment of every npm test invocation (both the tarball pass and the source-fallback pass). Plugin authors whose tests need network or a working container daemon can detect the harness and self-skip those tests:

// node:test example
import { describe, it } from "node:test";
describe("my integration test", () => {
  it("pulls an image", { skip: !!process.env.SIGNALK_REGISTRY_TEST }, async () => {
    // …
  });
});

vitest and mocha have equivalent skip mechanisms (describe.skip(condition), if (cond) this.skip()). Tests that don't honor the env var still run and are subject to --net=none; they typically fail and cost the plugin 30 score points (no +25 "tested" badge AND a −5 "tests-failing" penalty). The clean alternative is to split unit and integration tests into separate scripts (e.g. npm test for unit, npm run test:integration for the rest) so npm test is safe to run in any sandbox — that's what signalk-container does since 1.10.1.

Supply chain protection

All plugin dependency installs use --ignore-scripts to block postinstall/preinstall lifecycle scripts from transitive dependencies. The Signal K server itself is installed normally since it is trusted first-party code.

Artifact validation

The merge job validates every result entry before committing. Each slot must have a valid composite score (0-100), known badge names, ISO timestamp, and boolean installs field. Malformed entries are rejected.

Best-score-wins

When a plugin is retested, the new result only replaces the old one if its score is equal or higher. A transient CI failure (GitHub 500, npm registry blip) cannot downgrade a plugin that was previously passing.

Stale result retest

Plugins whose results are older than 7 days are automatically retested on the nightly run. This recovers from transient failures and catches new npm audit vulnerabilities.

Known Limitations

  • Plugins that need real hardware (CAN bus, serial ports) will not activate
  • Plugins that require credentials or external services in config will not activate
  • start() is tested with schema defaults extracted from the plugin's schema property — this matches what the admin UI sends. Plugins that need external services (databases, credentials) will still fail activation.
  • activates_without_config is tracked as an informational field (not scored) showing whether start({}) with empty config succeeds
  • The app shim logs unstubbed method accesses — check unstubbed_accesses in results to identify shim gaps
  • Plugins that load a native addon at require-time (sharp/libvips, canvas, prebuilt better-sqlite3, …) are supported — the sandbox lifts noexec /tmp so the addon binary can map (see Filesystem isolation above). They still need a prebuilt binary for linux-x64 on Node 24 and 26; addons that compile from source at install time fail because installs run with --ignore-scripts.

Contributing

See AGENTS.md for the contributor / AI agent guide: layout, the security invariants the workflow enforces, slot-key + score-shape rules, and how to reproduce CI failures locally (the firejail-vs-host gotcha is the most common surprise).

Releases

Packages

Contributors

Languages