Automated testing and quality scoring for Signal K server plugins.
Results: https://signalk.org/signalk-plugin-registry/
- Discovers Signal K plugins and webapps from npm (keywords:
signalk-node-server-plugin,signalk-webapp,signalk-embeddable-webapp— the ones the App Store lists by) - Tests each plugin against Signal K server: install, load, activate, detect providers, security audit
- Stores results persistently in
results.json— only retests on version changes - Publishes a static JSON API via GitHub Pages
- Runs nightly or on manual trigger
Each plugin is scored out of 100 points:
| Tier | What | Points |
|---|---|---|
| Install | npm install --ignore-scripts succeeds |
20 |
| Load | Constructor returns a valid plugin object | 15 |
| Activate | start() completes with schema defaults |
15 |
| Schema | Plugin exposes a JSON configuration schema | 5 |
| Tests | Plugin's own test suite passes | 25 |
| Security | No npm audit vulnerabilities | 20 |
| Changelog | CHANGELOG file or GitHub Release for the version | −5 if missing |
| Screenshots | signalk.screenshots array in package.json |
−5 if missing |
| Core dep freshness | Declared ranges allow the latest same-major release of core Signal K packages | −80 if held back |
| Legacy baconjs | Any baconjs range in dependencies / peerDependencies can resolve to 3.x |
−15 if it cannot |
| Legacy React | Embedded webapp bundles register React ≥19 | −15 if <19 |
| Bundled plugins | No runtime dependency is another App Store plugin or webapp | −30 if one is |
Security scoring breakdown: 20 points for a clean audit, 15 if only moderate vulnerabilities, 10 if high (no critical), 0 if any critical vulnerabilities.
Changelog detection prefers a CHANGELOG.md (or CHANGES.md / HISTORY.md) in the published tarball; if absent, falls back to the repository's public GitHub Releases atom feed (https://github.com/<owner>/<repo>/releases.atom, no token needed) and looks for a release whose tag or entry title carries the installed version. Tags may carry a package-name prefix, so the monorepo conventions (pkg@1.2.3, pkg-v1.2.3, pkg/v1.2.3) are recognised alongside a plain v1.2.3. The version must stand on its own, so 1.2.3 never matches a 1.2.30 or 1.2.3-beta.1 release. See signalk-server PR #2615 for the release-notes convention.
Screenshots detection requires at least one string entry under signalk.screenshots in package.json.
A webapp without a server plugin — a package carrying only signalk-webapp or signalk-embeddable-webapp, not signalk-node-server-plugin — is scored on the same scale. It has nothing to load, activate or configure, so the Load, Activate and Schema tiers (35 points) are not applicable and count as passed, and it gets a webapp badge instead of loads / activates. Every other check applies as for a plugin. The plugin-ci penalty does not apply either, because plugin-ci.yml requires the plugin keyword. The server's own admin UI (@signalk/server-admin-ui) is not scored.
Test scoring runs the plugin's suite from its published tarball, then falls back to a shallow clone of the source repo (devDependencies aren't published, so most suites only run from source). The clone checks out the version being scored: the commit npm recorded when it was published (gitHead), else its release tag (v<version>, <version> or <name>@<version>), and only when neither exists the default branch. The tested commit and which of the three it came from are recorded as source_test_ref in the plugin's results and shown when hovering the test status. In the source fallback the harness honors the build/test commands a plugin declares to the canonical plugin-ci.yml reusable workflow, so it mirrors the CI it already tracks for the plugin-ci badge rather than guessing. This is what lets webapp-class plugins (Angular/React/Vue) be scored correctly — their plain build/test scripts are often a non-exiting bundler and a watch-mode runner, which a naive guess would score "not runnable". A declared command may be an npm script invocation (npm run <script>, npm test) or an && chain of them, and the time limit covers the whole chain. When a plugin declares no such commands, or one of another shape, the harness falls back to its own heuristics.
A plugin published from a subdirectory of its repository is scored there rather than at the repository root, provided it declares npm's standard repository.directory field. Test discovery, the build and the test run all happen in that subdirectory, so a sibling package's tests are never mistaken for the plugin's. Dependencies install at the repository root — which is what npm workspaces expect — and the subdirectory is installed separately when it isn't a workspace. Note that such plugins currently cannot run the reusable plugin-ci.yml workflow (it has no working-directory input), so they still take the no-plugin-ci penalty; that fix belongs upstream.
Core dep freshness checks the plugin's declared dependencies and peerDependencies (never devDependencies — those don't reach user installs) against the latest npm release of each core Signal K package: @signalk/server-api, @canboat/canboatjs, @canboat/ts-pgns, @signalk/n2k-signalk, @signalk/nmea0183-signalk, @signalk/streams. A range that targets the same major as the latest release but cannot resolve to it (e.g. ~2.9.0 or an exact 2.9.0 when latest is 2.30.0) holds the package back in every user's ~/.signalk install and costs −80. Ranges on an older major are not flagged (the plugin may legitimately not support the new major yet), and neither are git/file/URL specs, dist-tags, or invalid ranges. If the npm registry lookup fails, the check is skipped — a registry flake never costs points.
Legacy runtime deps cover two libraries the server provides at a fixed major and only bridges for older plugin builds through temporary compatibility shims. baconjs: the server moved to 3.x in 2.24.0 and redirects every require('baconjs') to its own copy (signalk-server#2487); a dependencies / peerDependencies range that cannot resolve to any 3.x release (e.g. ^0.7.88, ^1.0.1) costs −15. React: the admin UI is React 19 and renders Module Federation remotes built against React 16 through an isolated bridge (signalk-server#2451); for plugins with an embedded-webapp keyword (signalk-embeddable-webapp, signalk-plugin-configurator, signalk-node-server-addon) the remote entry the server injects into the admin UI (public/remoteEntry.js, or remoteEntry.js at the package root without a public/) and the chunks reachable from it are scanned for the shared React version they register, and a major below 19 costs −15. Files nothing references — stale builds, unrelated entries — are not consulted, and remotes that consume the host's React without shipping their own are not flagged. Both checks are pure file inspection — no plugin code runs — and an unreadable package or bundle is indeterminate, never a penalty.
Bundled plugins checks the plugin's dependencies, optionalDependencies and non-optional peerDependencies (npm installs all three) for packages the App Store lists — anything published with the signalk-node-server-plugin, signalk-webapp or signalk-embeddable-webapp keyword. npm installs such a dependency into the user's ~/.signalk tree at the range this plugin chose, where the server loads it as a plugin of its own: an old major (^2.0.2 when the plugin is at 3.x) runs there instead of the current release, and uninstalling the bundling plugin can take it away. Any such dependency costs −30, once per plugin. Declare companions in signalk.requires (the App Store offers to install them together with the plugin) or signalk.recommends (suggested on the detail page) instead — see the publishing guide. The check reads the package.json npm installed for each dependency, so git, tarball and npm: alias specs are judged by the package they actually install. A dependency on the plugin itself is not counted, and one that was not installed or has an unreadable manifest is skipped.
Plotter extensions — plugins that register a plotterExtensions resource provider to contribute widgets, panels, buttons and map layers to chart plotters such as Freeboard-SK — get the plotter-extension badge and two checks from the provider guide. A package without the signalk-category-plotter-extensions keyword costs −10: the keyword lists it in the App Store's Plotter Extensions category (server 2.36.0 and later) and is how plotters find installable extensions. While the plugin runs, the harness calls the provider's listResources() the way a host does and checks the manifests it returns; any of these costs −10 once: no manifest, a missing name, an apiVersion other than "1" (the host drops such a manifest), no contribution, a widget size other than 1x1/2x1/1x2/2x2, a configPanel, button or action panel or image-overlay runtime the manifest does not declare, or an asset URL that is not server-relative or sits under /plugins/, which the server serves to admins only. A plugin whose start() fails is not checked further, since activation already accounts for that. The keyword without a registered manifest is not penalised.
Provider detection (resources, weather, history, autopilot, radar) is tracked as an informational badge but does not affect the score — most plugins are not expected to register providers.
| Badge | Meaning |
|---|---|
compatible |
Installs successfully |
loads |
Plugin constructor succeeds |
webapp |
Webapp without a server plugin; Load, Activate and Schema do not apply |
activates |
start() completes without error (with schema defaults) |
has-providers |
Registers at least one provider (informational) |
tested |
Plugin has its own test suite and it passes |
tests-failing |
Plugin has tests but they fail (-5 penalty) |
npm-audit-ok |
No npm audit vulnerabilities (20 pts) |
audit-moderate |
Has moderate vulnerabilities (15 pts) |
audit-high |
Has high vulnerabilities (10 pts) |
audit-critical |
Has critical vulnerabilities (0 pts) |
has-changelog |
CHANGELOG file or matching GitHub Release is available |
has-screenshots |
Declares at least one signalk.screenshots entry |
holds-back-core-deps |
A declared range pins a core Signal K package below its latest same-major release (−80 penalty) |
legacy-baconjs |
A dependencies / peerDependencies range for baconjs cannot resolve to 3.x (−15 penalty) |
legacy-react |
Embedded webapp bundle registers React <19 (−15 penalty) |
bundles-plugins |
A runtime dependency is another App Store plugin or webapp (−30 penalty) |
plotter-extension |
Registers a plotterExtensions manifest (informational) |
missing-plotter-keyword |
Registers a plotter extension without the signalk-category-plotter-extensions keyword (−10 penalty) |
invalid-plotter-manifest |
The plotter extension manifest would be dropped by a host or names a panel, runtime or asset it cannot load (−10 penalty) |
broken |
Failed to install |
Results are published to GitHub Pages:
index.json— summary of all plugins, sorted by scoreplugins/<name>.json— full detail for one plugin
Per-plugin records include upstream informational metrics fetched once per
nightly run so signalk-server installs can display them without each boat
hitting api.github.com (60/hr unauthenticated limit). Fields:
stars— GitHub repository stargazer countopen_issues— open issue count (includes PRs)contributors— contributor countdownloads_per_week— npm weekly downloadsgithub_url— normalisedhttps://github.com/<owner>/<repo>URL
All fields are optional — any combination of GitHub API limits, missing
repositories, or npm throttling can leave them absent. Consumers should
render — or a similar empty-state when a field is not present.
Test a single plugin locally:
npm ci
npx ts-node test-harness/runner.ts <plugin-name> [version]Test against a local plugin source:
npx ts-node test-harness/detect-providers.ts /path/to/pluginIf you maintain a plugin and want a fresh score without waiting for the nightly run — for example right after publishing a fix — open a re-score request:
- Go to New issue and choose Request a plugin re-score.
- Enter your npm package name (e.g.
signalk-my-pluginor@scope/signalk-my-plugin) and submit.
A bot validates the request, runs a single-plugin scan, comments the new score and badges on the issue, then closes it. This usually takes a few minutes.
Your plugin must be published to npm and carry the signalk-node-server-plugin
keyword, or signalk-webapp / signalk-embeddable-webapp for a webapp (the
keywords the Signal K app store uses to discover packages). If
it isn't a recognised plugin, the bot replies explaining why and closes the
issue without running a scan.
To re-run later on the same issue — after shipping another change — comment
/rescore <npm-name>.
You don't need write access to this repository; anyone can open the request.
Maintainers can also trigger a scan directly. Go to Actions > "Nightly Plugin Registry Scan" > Run workflow:
- changed_only — only test plugins with new versions since last run
- all_plugins — retest everything
- single_plugin — test one specific plugin by npm name
Plugins are untrusted code. The CI pipeline is designed so that even a deliberately malicious plugin cannot steal secrets, exfiltrate data, tamper with results, or attack third parties.
The workflow has four jobs. Only the last two have any permissions:
| Job | Permissions | Runs plugin code? |
|---|---|---|
| plan | {} (none) |
No |
| test | {} (none) |
Yes |
| merge-results | contents: write |
No |
| publish | contents: write |
No |
The test job uses persist-credentials: false so no token exists in git config either.
All plugin code execution is wrapped in firejail --net=none:
require()+start()(provider detection) — runs as a sandboxed subprocessnpm testfrom published packages and from cloned source repos
This prevents plugin code from making any outbound network requests — no data exfiltration, no phoning home for second-stage payloads, no participation in attacks on third parties.
Firejail runs with --read-only=/home --read-only=/etc --read-only=/var. Plugin code cannot modify the workspace, git history, results.json, or npm cache. Only /tmp (where plugin workdirs live) is writable.
The sandbox also runs with --ignore="noexec /tmp", which lifts the noexec /tmp that firejail's default profile would otherwise apply. Without it, plugins that load a native addon at require-time (sharp/libvips, canvas, prebuilt better-sqlite3, …) fail dlopen with failed to map segment from shared object and flatline at ~30 for an environment-only reason. This is a deliberate, scoped relaxation: the sandbox's whole job is to execute the plugin's own untrusted JavaScript, so the attacker already has arbitrary code execution before /tmp is touched — noexec /tmp only blocks the binary-mmap path while leaving the JS path open, so it adds no real boundary here. The load-bearing containment — --net=none and the read-only mounts — is unchanged.
The harness sets SIGNALK_REGISTRY_TEST=1 in the environment of every npm test invocation (both the tarball pass and the source-fallback pass). Plugin authors whose tests need network or a working container daemon can detect the harness and self-skip those tests:
// node:test example
import { describe, it } from "node:test";
describe("my integration test", () => {
it("pulls an image", { skip: !!process.env.SIGNALK_REGISTRY_TEST }, async () => {
// …
});
});vitest and mocha have equivalent skip mechanisms (describe.skip(condition), if (cond) this.skip()). Tests that don't honor the env var still run and are subject to --net=none; they typically fail and cost the plugin 30 score points (no +25 "tested" badge AND a −5 "tests-failing" penalty). The clean alternative is to split unit and integration tests into separate scripts (e.g. npm test for unit, npm run test:integration for the rest) so npm test is safe to run in any sandbox — that's what signalk-container does since 1.10.1.
All plugin dependency installs use --ignore-scripts to block postinstall/preinstall lifecycle scripts from transitive dependencies. The Signal K server itself is installed normally since it is trusted first-party code.
The merge job validates every result entry before committing. Each slot must have a valid composite score (0-100), known badge names, ISO timestamp, and boolean installs field. Malformed entries are rejected.
When a plugin is retested, the new result only replaces the old one if its score is equal or higher. A transient CI failure (GitHub 500, npm registry blip) cannot downgrade a plugin that was previously passing.
Plugins whose results are older than 7 days are automatically retested on the nightly run. This recovers from transient failures and catches new npm audit vulnerabilities.
- Plugins that need real hardware (CAN bus, serial ports) will not activate
- Plugins that require credentials or external services in config will not activate
start()is tested with schema defaults extracted from the plugin'sschemaproperty — this matches what the admin UI sends. Plugins that need external services (databases, credentials) will still fail activation.activates_without_configis tracked as an informational field (not scored) showing whetherstart({})with empty config succeeds- The app shim logs unstubbed method accesses — check
unstubbed_accessesin results to identify shim gaps - Plugins that load a native addon at require-time (
sharp/libvips,canvas, prebuiltbetter-sqlite3, …) are supported — the sandbox liftsnoexec /tmpso the addon binary can map (see Filesystem isolation above). They still need a prebuilt binary forlinux-x64on Node 24 and 26; addons that compile from source at install time fail because installs run with--ignore-scripts.
See AGENTS.md for the contributor / AI agent guide: layout, the security invariants the workflow enforces, slot-key + score-shape rules, and how to reproduce CI failures locally (the firejail-vs-host gotcha is the most common surprise).