Get started with Kestra in 3 minutes.
- What user problem does this solve? Teams need to run pipelines against SDKs and dedicated runners including Apache Flink from orchestrated workflows instead of relying on manual console work, ad hoc scripts, or disconnected schedulers.
- Why would a team adopt this plugin in a workflow? It keeps Apache Beam steps in the same Kestra flow as upstream preparation, approvals, retries, notifications, and downstream systems.
- What operational/business outcome does it enable? It reduces manual handoffs and fragmented tooling while improving reliability, traceability, and delivery speed for processes that depend on Apache Beam.
- Provides plugin components under
io.kestra.plugin.beam. - Includes classes such as
RunPipeline,BeamSDK,BeamRunner,DataflowRunnerConfig.
- Full documentation can be found under: kestra.io/docs
- Documentation for developing a plugin is included in the Plugin Developer Guide
RunPipeline executes Beam YAML pipelines inline or from a file, forwards Beam counters, distributions, and gauges into Kestra metrics, and can target Direct, Flink, Spark, or Dataflow runners with the Java or Python SDK.
id: beam-direct
namespace: dev.beam
tasks:
- id: run-beam
type: io.kestra.plugin.beam.RunPipeline
sdk: JAVA
beamRunner: DIRECT
definition: |
pipeline:
transforms:
- type: ReadFromText
config:
path: "data.txt"
- type: CountApache 2.0 © Kestra Technologies
We release new versions every month. Give the main repository a star to stay up to date with the latest releases and get notified about future updates.
