CTIS: CTEM Ingest Schema

CTIS is the JSON format security tools use to send assets, findings and dependencies (SBOM) to a CTEM platform such as OpenCTEM. Product documentation: https://docs.openctem.io. This repository is the single source of truth for:

• The specification: docs/spec.md, normative, with a field reference generated from the schema.
• JSON Schemas (schemas/v1/): draft-07, language-agnostic.
• Go types (root package): what OpenCTEM decodes reports into.
• Severity (severity/) and fingerprint (fingerprint/) helpers.
• Converters: SARIF and recon (subfinder, dnsx, naabu, httpx, katana) output to CTIS.
• Examples (examples/): one report per finding type, validated in CI.
• Web URLs (weburl/): parse, normalise, template and redact web URLs.
• Politeness keys (politeness/): which hosts a scan job touches and when two jobs touch the same host.
• Repository source (source/): the repository, branch and exact commit a platform code scan fetches.
• Per-target outcomes (outcome/): how each target of a scan task ended (done, failed, skipped, unfinished), so only unfinished targets are run again.
• Capability taxonomy (capability/): the closed list of acts a scan tool performs and the contract of each (docs/capabilities.md).
• Tool settings (settings/): the closed JSON Schema subset a tool manifest declares its configuration in, the validator, and layered resolution of the values a scan runs with.
• Tool options (tooloption/): the common execution options of scan tools (timeout, rate, concurrency, retries), their bounds and how layers may narrow them.
• Feed transfer (transfer/, its own module): download of signed, chunked feed bundles from ordered origins with retries, resume and fall-back, and the bundle format, writer and consumer.

The current specification version is 1.6. See CHANGELOG.md.

Installation (Go)

go get github.com/openctemio/ctis

The module has no dependencies outside the Go standard library.

import (
    "github.com/openctemio/ctis"
    "github.com/openctemio/ctis/fingerprint"
    "github.com/openctemio/ctis/severity"
)

// Produce
report := ctis.NewReport() // version 1.6, $schema set, timestamp now
report.Tool = &ctis.Tool{Name: "my-scanner", Version: "1.0.0", Capabilities: []string{"sast"}}
report.Assets = append(report.Assets, ctis.Asset{ID: "repo", Type: ctis.AssetTypeRepository, Value: "github.com/org/repo"})
report.Findings = append(report.Findings, ctis.Finding{
    Type:        ctis.FindingTypeVulnerability,
    Title:       "SQL injection",
    Severity:    ctis.Severity(severity.FromCVSS(8.8)),
    AssetRef:    "repo",
    RuleID:      "sqli",
    Location:    &ctis.FindingLocation{Path: "app/db.go", StartLine: 42, Snippet: snippet},
    Fingerprint: fingerprint.GenerateSASTStable("app/db.go", "sqli", "db.Find", snippet),
})
if err := report.Validate(); err != nil {
    log.Fatal(err)
}

// Consume: decode strictly, as OpenCTEM does, then validate.
dec := json.NewDecoder(r)
dec.DisallowUnknownFields()
var in ctis.Report
if err := dec.Decode(&in); err != nil { /* reject */ }
if err := in.Validate(); err != nil { /* reject */ }

// Convert SARIF (Semgrep, CodeQL, Trivy, ...)
report, err := ctis.FromSARIF(sarifBytes, nil)

Schemas

Schema	Description
report.json	Report envelope: version, metadata, tool, assets, findings, dependencies, endpoints
asset.json	Assets (domains, IPs, hosts, repositories, cloud, Web3, ...)
finding.json	Security findings, with their web location (web) and typed evidence (evidence_items)
dependency.json	SBOM dependencies
endpoint.json	Web endpoints: a method and a path on an origin, parameters by name only (1.6)
evidence-item.json	Typed evidence of a finding: HTTP exchanges, text, command output, file excerpts, screenshots, with marked sensitive spans (1.6)
web3-asset.json	Web3 asset details
web3-finding.json	Web3 vulnerability details

Each schema's $id resolves:

https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/report.json
https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/asset.json
https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/finding.json
https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/dependency.json
https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/endpoint.json
https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/evidence-item.json
https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/web3-asset.json
https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/web3-finding.json

Replace main with a release tag (vX.Y.Z) for an immutable copy. Every object sets additionalProperties: false except the properties bags: put producer-specific data in properties.

Validating CTIS reports

report.json references the other schemas, so load all eight.

Python

pip install "jsonschema[format-nongpl]"

import glob, json
from jsonschema import Draft7Validator, FormatChecker
from referencing import Registry, Resource
from referencing.jsonschema import DRAFT7

schemas = [json.load(open(p)) for p in glob.glob("schemas/v1/*.json")]
registry = Registry().with_resources(
    (s["$id"], Resource.from_contents(s, default_specification=DRAFT7)) for s in schemas
)
report_schema = next(s for s in schemas if s["$id"].endswith("/report.json"))
validator = Draft7Validator(report_schema, registry=registry, format_checker=FormatChecker())

errors = list(validator.iter_errors(json.load(open("my-report.json"))))
for e in errors:
    print("/".join(map(str, e.path)), e.message)

scripts/validate_schemas.py does this for the examples in CI.

Node.js

npm install ajv ajv-formats

const Ajv = require('ajv');
const addFormats = require('ajv-formats');
const fs = require('fs');

const ajv = new Ajv({ allErrors: true });
addFormats(ajv);
for (const f of fs.readdirSync('schemas/v1')) {
  ajv.addSchema(JSON.parse(fs.readFileSync(`schemas/v1/${f}`, 'utf8')));
}
const validate = ajv.getSchema('https://raw.githubusercontent.com/openctemio/ctis/main/schemas/v1/report.json');
if (!validate(myReport)) console.log(validate.errors);

Schema validation checks shape. Report.Validate() in Go also checks what a schema cannot: unique IDs, asset_ref resolution, the spec version and score ranges (spec section 3.3).

Importing other tools' exports

The importer package converts exported files to CTIS 1.6: Nessus, Qualys (with the KnowledgeBase), DefectDojo Generic JSON, CycloneDX, SPDX, osv-scanner, CSAF, OpenVEX, SARIF 2.1.0, the ZAP traditional report (JSON and XML), generic bug-bounty reports (bounty_report, JSON and CSV), and the native JSON of trivy, grype, nuclei, semgrep, opengrep, betterleaks, gitleaks and vuls. Detect cannot tell opengrep JSON from semgrep JSON (same shape), so a caller that ran opengrep sets Options.Format to opengrep; its findings keep the fingerprint a semgrep run gave them. Detect names bounty report JSON by its top-level bounty_reports member but never CSV, which a caller forces with Options.Format; Detect names the format from the first bytes; Parse converts the file:

res, err := importer.Parse(ctx, file, importer.Options{})
// res.Report: the CTIS report; res.VEX: statements of a VEX document;
// res.Stats, res.Issues (with line numbers), res.Unmapped.

Each format has a mapping spec: every source field, the CTIS member that keeps it or why it is ignored on purpose (docs/importers, generated from the code). The tests fail when a fixture holds a field its spec does not list, or a spec maps a field no fixture holds. Inputs are treated as hostile: size, depth, element, text and record limits; no XML internal subsets, entities or external resources; UTF-8, US-ASCII or ISO-8859-1 only; OpenZip refuses traversal, links, nested archives and decompression bombs. Scan credentials and account names are never copied into a report.

Web URLs

The weburl package parses, normalises and templates web URLs, so a crawler, a scanner and the platform give one endpoint the same identity:

u, err := weburl.Parse("HTTPS://Shop.Example.COM:443/api/v1/orders/42/?token=x")
u.Origin()   // "https://shop.example.com"
u.Template() // "/api/v1/orders/{int}"
u.Params     // ["token"]: names only, never values
weburl.RedactURL("https://u:p@a.example/x?api_key=k#t") // "https://a.example/x?api_key="
weburl.PathHash("GET", u.Template())                      // the endpoint's dedup key

Parse refuses anything that is not a plain absolute http(s) URL: user info, control and bidi characters, malformed escapes, invalid ports, ambiguous numeric hosts and URLs over 2,048 bytes. A query value is never kept.

Politeness keys

The politeness package is the one definition of the per-host politeness keys of scan work, shared by the platform (which holds the keys while a job runs and sends them with the job) and the sensor runtime (which admits the job under the same keys):

keys := politeness.Keys([]string{"https://www.example.com/x", "10.0.0.5", "10.0.1.0/24"}, "")
// ["10.0.0.5" "in:10.0.0.0/24" "net:10.0.1.0/24" "www.example.com"]
politeness.Conflicts(keys) // the keys another job must not hold at the same time

A name, the host of a URL or a service, an address (with the marker of its /24 or /120 bucket) and a range (one key per bucket) each have a key. An address and a range scan of its bucket conflict, two addresses of one bucket do not. With a network id, the keys of private addresses and of names carry the network, so overlapping private ranges at two sites stay two sets of hosts; public addresses never do.

A port never makes a separate host: 203.0.113.5, 203.0.113.5:8443 and https://203.0.113.5/ all have the key 203.0.113.5. Group cuts a target list into groups of one host, so parts built from whole groups (the chunks of a step, the waves of a running job) never conflict with each other:

politeness.Group([]string{"203.0.113.5", "198.51.100.7", "203.0.113.5:8443"}, "")
// [["203.0.113.5" "203.0.113.5:8443"] ["198.51.100.7"]]

Repository source

The source package is the source of a platform code scan (a full scan of a protected branch): the platform puts a source.Repository in the job, the sensor fetches exactly that commit read-only, and both validate it with the same code:

{"kind": "github", "clone_url": "https://github.com/acme/api.git", "branch": "main",
 "commit": "4f2c…(40 or 64 hex)", "depth": 1, "credential": "scm",
 "capabilities": ["sast.code@1", "secrets.code@1", "sca.deps@1", "iac.misconfig@1"]}

Validate accepts only an HTTPS URL with a DNS host (no address, no user information, query or fragment, plain path segments), a branch git accepts as a ref (no leading -), a full commit id, a depth of 1 to 10000 and capabilities that take a repository as input. credential is the name of the credential the sensor asks the platform for when the job starts (a short-lived, read-only token for this repository); a token never travels in the job.

SAST code identity and data flow

• fingerprint.SnippetHash hashes the code a SAST finding flags, independent of its line and indentation. A producer that has the scanned tree but no snippet sends it as the partial fingerprint snippetHash/v1 (fingerprint.PartialSnippetHash) instead of the code, so a receiver can key the finding on the flagged code and a decision on it survives code moving above it.
• A finding's data_flow is its taint trace: source, intermediate steps and sink, each with file, lines, snippet and label. FromSARIF maps codeFlows to it, and the semgrep and opengrep importers keep their traces. NormalizeDataFlow bounds a trace (at most MaxDataFlowSteps, 100, steps), makes paths repository-relative (dropping a path that leaves the repository) and masks credentials in snippets and labels. A malformed trace leaves the finding without a data flow and never fails the report. The data flow never feeds the fingerprint.

Per-target outcomes

The outcome package is the shape a sensor uses to say how each target of a scan task ended, and the platform uses to settle targets one by one. A command result carries the list under target_outcomes:

[{"target": "https://a.example", "state": "done"},
 {"target": "https://b.example", "state": "failed", "class": "timeout"},
 {"target": "https://c.example", "state": "unfinished"}]

States are done, failed (class unreachable, timeout, tool_error or egress_refused: the sensor's network confinement refused every connection toward the target), skipped (class refused or excluded) and unfinished (the task stopped first). Done, failed and skipped settle a target; only unfinished targets are handed out again. outcome.Normalize(reported, taskTargets) reads a sensor's list against the task's own targets: unknown targets and invalid entries are dropped, and a task target without a valid entry is unfinished, so a sensor can never add work. outcome.Remaining returns the targets to run again.

Capability taxonomy

The capability package is the closed, versioned list of acts a scan tool can perform (discover.subdomains@1, scan.ports@1, sast.code@1, ...). The capability, not the tool, carries the facts about the act: its phase and CTEM stage, its tier floor (how intrusive it is at minimum), its typed input and output ports with the CTIS asset types they carry, its standard params, the CTIS paths every report of it must carry, and its MITRE ATT&CK, D3FEND and CAPEC references. A tool only declares which capabilities it implements.

c, ok := capability.Lookup("scan.ports@1")
c.Accepts("ip_address")          // true: an input port carries it
c.MayEmit("asset:repository")    // false: no port carries it
violations, err := c.Check(report, capability.CheckOptions{})

Check reports outputs the capability may not emit and records that miss a required path; producers run it before upload and receivers again at ingest. The reference page docs/capabilities.md is generated from the data.

Tool settings

The settings package is the tool settings contract: the closed subset of JSON Schema 2020-12 (plus x-octm-* annotations) in which a tool manifest declares its configuration, the validator, and the layered resolution of the values a task runs with. The platform validates what an administrator enters with it and the sensor validates the same values again before a task uses them; both import this one implementation, so the two never disagree.

s, err := settings.ParseSchema(manifestConfigJSON)
err = s.Validate(map[string]any{"rate_limit": 100})
eff, err := s.Resolve(
	settings.Layer{Source: settings.SourceSensor, Values: sensorValues},
	settings.Layer{Source: settings.SourceScan, Values: scanValues},
)
eff.Source("rate_limit") // which layer set the effective value

Annotations: x-octm-scope (sensor or scan: the lowest layer that may set the key), x-octm-tier (the safety class of a non-default value: T0 safe, T1 impact, T2 dangerous), x-octm-common (the key is a common tool option, see below), x-octm-flag (the tool's native flag, shown in a job's effective configuration), and presentation hints. Layers, in order: tenant policy, the sensor's configuration, a scan profile or workflow step, the scan, the platform's decision for one job; only tenant and sensor layers may set sensor-scope keys.

Tool options

The tooloption package defines the execution options almost every scan tool has, once, so every tool, the sensor runtime and the platform mean the same thing by each name:

Option	Unit	Bounds	Narrows
timeout	seconds per request or probe	1..300	no
rate	requests or packets per second	1..100000	down
concurrency	targets or requests at once	1..1000	down
retries	repeats of an unanswered request	0..10	down

A tool declares its options in the configuration schema of its tool manifest; a property that is a common option carries "x-octm-common": "<name>", must have the common type and may only narrow the common bounds (CheckProperty). Settings are layered (platform defaults, tenant policy, sensor configuration, scan or workflow step, the platform's decision for one job); for an option that narrows down, only an administrator's layer (tenant, sensor) may raise it above the tool's default, and a layer closer to the job may only lower the value in force (NarrowValue). proxy, user_agent and headers are reserved: the platform's egress and HTTP policy set them, never a tool option (Reserved).

o, _ := tooloption.Lookup("rate")
o.CheckProperty("integer", nil, &max) // a tool's "rate" property is compatible
v, ok := o.NarrowValue(100, 500)      // 100, false: a scan cannot raise the sensor's rate

Mapping any JSON tool output

The importer/mapping package turns the JSON or JSON Lines output of any command-line tool into CTIS through a declarative mapping file (openctem.io/mapping/v1), so wrapping a tool needs no parser code:

m, err := mapping.Load(mappingJSON)
report, stats, err := m.Apply(ctx, stdout, mapping.Options{Tool: &ctis.Tool{Name: "example-portscan"}})

The language is closed: predicates (exists, equals, in, RE2 matches), source paths, constants, named templates and a fixed list of transforms. It has no expressions, code, file or network access. Targets are checked against the CTIS types. Ids, links between records and the report's tool cannot be set. Input is bounded in size, depth, line length and record count. See docs/mapping.md.

Feed transfer

github.com/openctemio/ctis/transfer is a separate Go module (tagged transfer/vX.Y.Z) so the root module keeps zero dependencies; it needs the standard library and the Prometheus client. transfer fetches files from an ordered list of origins (release URL, mirrors, a local directory) with retries, Range resume, a content-addressed blob cache and per-origin circuits; transfer/bundle is the signed (DSSE, Ed25519), chunked JSON Lines bundle format, its writer and a resumable consumer.

go get github.com/openctemio/ctis/transfer

The caller supplies the HTTP client, and it must be SSRF-guarded: there is no unguarded default. The format is pinned by a golden test to the bytes of github.com/openctemio/sdk-go/pkg/transfer v0.1.0, where the package lived before; switching the import path changes nothing on the wire or on disk.

Asset types

Group	Types
External attack surface	domain, subdomain, ip_address, certificate
Applications	website, web_application, api, mobile_app, service
Code	repository
Cloud	cloud_account, compute, storage, database, serverless, container_registry
Infrastructure	host, server, container, kubernetes, kubernetes_cluster, kubernetes_namespace
Network	network, vpc, subnet, load_balancer, firewall
Identity / IAM	iam_user, iam_role, service_account
Recon results	http_service, open_port, discovered_url
Web3	smart_contract, wallet, token, nft_collection, defi_protocol, blockchain
Other	unclassified

Finding types

Type	Description
vulnerability	Code, dependency, host or web vulnerabilities
secret	Exposed secrets and credentials
misconfiguration	IaC, cloud and configuration issues
compliance	Compliance control failures
web3	Smart contract vulnerabilities

Versioning

• version in a report is the spec version, MAJOR.MINOR. Receivers accept any minor of their major.
• Minor versions only add optional members or values. Receivers decode strictly, so a producer must not send members newer than the receiver's minor: upgrade receivers first, then producers.
• Module v1.MINOR.x implements spec 1.MINOR.

Details: docs/spec.md, section 2.

Contributing

Every change to the format updates the schema, the Go types, the examples and the CHANGELOG together. go test ./... fails when the schema and the Go types disagree, when an example does not validate, or when the generated field reference in docs/spec.md is stale (refresh it with go test -run TestSpecFieldReference -update). The design record of the 1.4 to 1.6 additions, with the status of each proposal, is docs/proposals-1.4.md.

Security

Report vulnerabilities privately; see SECURITY.md.

License

Apache License 2.0. See LICENSE and NOTICE.