diff --git a/README.md b/README.md index cfa7f20..8ad013a 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ A plugin marketplace for working with [Flyte](https://flyte.org) — in [Google Antigravity](https://antigravity.google), or any agent harness that supports [Agent Skills](https://agentskills.io). -21 skills, plus two **MCP servers** that let Claude search Flyte docs and act on your own +25 skills, plus two **MCP servers** that let Claude search Flyte docs and act on your own cluster. `uvx flyte-skills install` gets you the skills in any harness; `uvx flyte-agent-plugins mcp install` adds the servers, and installing the `flyte` plugin gets you both at once — see [Bundled MCP servers](#bundled-mcp-servers). @@ -19,7 +19,7 @@ This guide assumes that you have [uv installed](https://docs.astral.sh/uv/gettin uvx flyte-skills install ``` -That copies all 21 skills into whichever harness directories it finds on your machine — +That copies all 25 skills into whichever harness directories it finds on your machine — no arguments needed. (`pip install flyte-agent-plugins` then `flyte-agent-plugins install` works the same way.) @@ -80,7 +80,7 @@ convention shared across agent CLIs, so any harness honouring it picks the skill without a harness-specific flag. `--dir ` writes them anywhere else you like — both work the same with `--project`, `--dry-run`, and `uninstall`. -**Two packages.** `flyte-skills` and `flyte-agent-plugins` carry the same 21 skills, so +**Two packages.** `flyte-skills` and `flyte-agent-plugins` carry the same 25 skills, so every `install` command above works under either name. They differ in one thing: only `flyte-agent-plugins` ships the `mcp` subcommand, so reach for it when you want the MCP servers too. To pin a version, @@ -88,7 +88,7 @@ servers too. To pin a version, ## Install as a plugin (Claude Code, Codex, and Antigravity) -Installing the plugin instead of the skills gets you the same 21 skills **plus** both MCP +Installing the plugin instead of the skills gets you the same 25 skills **plus** both MCP servers, which are declared in `plugins/flyte/.mcp.json`. ### Claude Code @@ -156,20 +156,20 @@ The skills are plain [Agent Skills](https://agentskills.io) (`SKILL.md` + YAML frontmatter), so they work in any harness that supports the standard. `flyte-agent-plugins install` above covers all of them; each harness also has its own installer, which is what you want when you would rather track the repo than a release, or to install a single skill -rather than all 21. +rather than all 25. | Harness | Skills | MCP servers | `flyte-agent-plugins install` | |---|---|---|---| -| Claude Code | all 21 | both, automatically | `--target claude` | -| Codex CLI | all 21 | both, automatically | `--target agents` | -| Antigravity | all 21 | both, via the plugin | `--target agents --project` | +| Claude Code | all 25 | both, automatically | `--target claude` | +| Codex CLI | all 25 | both, automatically | `--target agents` | +| Antigravity | all 25 | both, via the plugin | `--target agents --project` | | Hermes | per-skill | none — add manually | `--target hermes` | -| opencode | all 21 | none — add manually | `--target opencode` | -| pi | all 21 | none — add manually | `--target pi` | +| opencode | all 25 | none — add manually | `--target opencode` | +| pi | all 25 | none — add manually | `--target pi` | ### Hermes -`flyte-agent-plugins install --target hermes` writes all 21 into `~/.hermes/skills/`, which Hermes +`flyte-agent-plugins install --target hermes` writes all 25 into `~/.hermes/skills/`, which Hermes auto-discovers with no registration step. Add `--project` for `/.hermes/skills/` — project skills need `hermes skills trust` before first use. @@ -253,6 +253,8 @@ Convert existing Flyte 1 (`flytekit`) code to Flyte 2. Distilled from the offici | [`flyte-migrate-data-io`](plugins/flyte/skills/flyte-migrate-data-io) | Migrate data types & I/O: `FlyteFile`/`FlyteDirectory`→`flyte.io.File`/`Dir`, `StructuredDataset`→`flyte.io.DataFrame`, dataclasses/Pydantic as task I/O. | | [`flyte-migrate-ml`](plugins/flyte/skills/flyte-migrate-ml) | Migrate ML workloads (training, HPO, GPU/deep learning, batch inference, end-to-end pipelines) and the new-in-v2 patterns (real-time serving, apps, sandboxed execution) they unlock. | | [`flyte-migrate-slurm`](plugins/flyte/skills/flyte-migrate-slurm) | Migrate Slurm (`sbatch`/`srun`) workloads: `#SBATCH` pragmas → `TaskEnvironment` config, job arrays → `flyte.map`, `--dependency` chains → plain Python, multi-node `srun` → `ClusteredTaskEnvironment`, `--requeue` → retries/checkpoints/spot. | +| [`flyte-migrate-plugins`](plugins/flyte/skills/flyte-migrate-plugins) | Map `flytekitplugins-*` to `flyteplugins-*` (Spark, Ray, Dask, PyTorch Elastic, BigQuery, Snowflake, Databricks, dbt, papermill, wandb, mlflow, pandera, Slurm), list the integrations with no v2 plugin and their replacements, and the API changes that break a literal port. | +| [`flyte-migrate-troubleshoot`](plugins/flyte/skills/flyte-migrate-troubleshoot) | Fix errors in freshly migrated code, indexed by error text: relative imports, missing `depends_on`, unwrapped DataFrames, `RuntimeUserError` instead of the original exception, notebooks missing from the pod, `https://` `File` inputs, `NamedTuple` outputs, and shell tasks that exit 1 with an empty message. Includes a compile → `--local` → remote debug loop. | Example: diff --git a/evals/scenarios/flyte-migrate-plugins/static.yaml b/evals/scenarios/flyte-migrate-plugins/static.yaml new file mode 100644 index 0000000..1c8e0ed --- /dev/null +++ b/evals/scenarios/flyte-migrate-plugins/static.yaml @@ -0,0 +1,5 @@ +id: flyte-migrate-plugins-static +skill: flyte-migrate-plugins +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/evals/scenarios/flyte-migrate-troubleshoot/static.yaml b/evals/scenarios/flyte-migrate-troubleshoot/static.yaml new file mode 100644 index 0000000..a444e20 --- /dev/null +++ b/evals/scenarios/flyte-migrate-troubleshoot/static.yaml @@ -0,0 +1,5 @@ +id: flyte-migrate-troubleshoot-static +skill: flyte-migrate-troubleshoot +tier: static +# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets). +# No agent, no LLM. Runs on every change. diff --git a/packaging/README.md b/packaging/README.md index 6c171a1..27a613d 100644 --- a/packaging/README.md +++ b/packaging/README.md @@ -4,10 +4,10 @@ The skills in `plugins/flyte/` are published from this one source of truth: | Registry | Package | Skills | `mcp` subcommand | Status | |---|---|---|---|---| -| PyPI | [`flyte-agent-plugins`](https://pypi.org/project/flyte-agent-plugins/) | all 21 | yes | published | -| PyPI | [`flyte-skills`](https://pypi.org/project/flyte-skills/) | all 21 | no | published | -| npm | `flyte-agent-plugins` | all 21 | yes | **built, not published** | -| npm | `flyte-skills` | all 21 | no | **built, not published** | +| PyPI | [`flyte-agent-plugins`](https://pypi.org/project/flyte-agent-plugins/) | all 25 | yes | published | +| PyPI | [`flyte-skills`](https://pypi.org/project/flyte-skills/) | all 25 | no | published | +| npm | `flyte-agent-plugins` | all 25 | yes | **built, not published** | +| npm | `flyte-skills` | all 25 | no | **built, not published** | **npm publishing is currently disabled** while distribution focuses on pip/uvx. The npm packages are still generated, packed, and validated on every CI run, so diff --git a/plugins/flyte/README.md b/plugins/flyte/README.md index eff3023..256c557 100644 --- a/plugins/flyte/README.md +++ b/plugins/flyte/README.md @@ -64,6 +64,11 @@ Convert existing Flyte 1 (`flytekit`) code to Flyte 2, distilled from the offici - **`flyte-migrate-slurm`** — Slurm (`sbatch`/`srun`) → Flyte 2: `#SBATCH` pragmas → `TaskEnvironment` config, job arrays → `flyte.map`, `--dependency` chains → plain Python, multi-node `srun` → `ClusteredTaskEnvironment`, `--requeue` → retries/checkpoints/spot. +- **`flyte-migrate-plugins`** — `flytekitplugins-*` → `flyteplugins-*`, integrations with no + v2 plugin and their replacements, `task_config=` → `plugin_config=`, and connector tasks. +- **`flyte-migrate-troubleshoot`** — errors in freshly migrated code, indexed by error text: + relative imports, missing `depends_on`, unwrapped DataFrames, `RuntimeUserError`, files + missing from the pod, `https://` `File` inputs, `NamedTuple` outputs, and shell tasks. ## Bundled MCP servers diff --git a/plugins/flyte/skills/flyte-migrate-config/SKILL.md b/plugins/flyte/skills/flyte-migrate-config/SKILL.md index 61cf157..74871a8 100644 --- a/plugins/flyte/skills/flyte-migrate-config/SKILL.md +++ b/plugins/flyte/skills/flyte-migrate-config/SKILL.md @@ -99,7 +99,7 @@ image = ( | Constructor | Use case | |---|---| | `Image.from_debian_base()` | Most common; includes the Flyte SDK | -| `Image.from_base(image_uri)` | Start from any existing image | +| `Image.from_base(image_uri)` | Start from any existing image (call `.clone(name=..., extendable=True)` before any `.with_*`) | | `Image.from_dockerfile(path)` | Complex custom builds | | `Image.from_uv_script(path)` | UV-based projects | @@ -116,10 +116,21 @@ Common chainable builder methods: `.with_pip_packages(...)`, `.with_requirements | `env` | `.with_env_vars()` | Method instead of param | | `commands` | `.with_commands()` | Method instead of param | | `copy` / `source_root` | `.with_source_file()` / `.with_source_folder()` | More explicit methods | -| `base_image` | `Image.from_base()` | Different constructor | +| `base_image` | `Image.from_base()` + `.clone(extendable=True)` | Different constructor; see below | | `builder` | Config file or `flyte.init()` | Global setting | | `platform` | `platform` (constructor) | Tuple: `("linux/amd64", "linux/arm64")` | +An image from `Image.from_base()` is not extendable, so chaining `.with_*` onto it fails when the module loads with `ValueError: Cannot add additional layers to a non-extendable image`. Clone it as extendable first: + +```python +# Flyte 1: ImageSpec(base_image="ghcr.io/flyteorg/rawcontainers-python:v2", packages=["pandas"]) +image = ( + Image.from_base("ghcr.io/flyteorg/rawcontainers-python:v2") + .clone(name="rawcontainers-python-pandas", extendable=True) + .with_pip_packages("pandas") +) +``` + For a private registry, create an image-pull secret and reference it: ```bash @@ -169,8 +180,8 @@ Caching is enabled at the env level with `cache="auto"` (or per-task on `@env.ta | Behavior | Description | |---|---| -| `"auto"` | Cache results and reuse if available | -| `"override"` | Always execute and overwrite the cache | +| `"auto"` | Cache results; the version is computed from the function body, so any code change invalidates the cache | +| `"override"` | Cache results under the explicit `version_override` string; the cache is invalidated only when you change it (the closest match to `cache_version`) | | `"disable"` | No caching (default for a `TaskEnvironment`) | ```python @@ -182,7 +193,7 @@ def cached_task(x: int) -> int: # Advanced control (replaces cache_version, serialize, ignored_inputs, ...) @env.task(cache=flyte.Cache( - behavior="auto", + behavior="override", # the SDK documents version_override for "override" only version_override="v1.0", serialize=True, ignored_inputs=("debug",), @@ -308,7 +319,7 @@ def main(trigger_time: datetime = datetime(2024, 1, 1, 2, 0)) -> str: return f"retrained model at {trigger_time.isoformat()}" ``` -Triggers support `flyte.Cron("0 9 * * *", timezone="America/New_York")` and `flyte.FixedRate(timedelta(hours=1))` as automations, plus convenience constructors like `flyte.Trigger.hourly()` and `flyte.Trigger.daily()`. +Triggers support `flyte.Cron("0 9 * * *", timezone="America/New_York")` and `flyte.FixedRate(60)` (an integer number of minutes, not a `timedelta`; a `timedelta` passes `flyte run` but fails at `flyte deploy`) as automations, plus convenience constructors like `flyte.Trigger.hourly()` and `flyte.Trigger.daily()`. ## CLI command mapping: pyflyte to flyte @@ -444,7 +455,7 @@ For API-key authentication in non-interactive environments, use `flyte.init_from 2. **Don't use `mem`, `ephemeral_storage`, or separate `requests`/`limits`** — use `memory`, `disk`, and a single `flyte.Resources` value that serves as both. 3. **Don't pass GPUs with `gpu="1"` plus `accelerator=`** — combine type and count into one `Type:count` string like `gpu="A100:2"`. 4. **Don't rebuild `ImageSpec`'s many constructor args** — start from a base (`Image.from_debian_base()`) and chain `.with_*` builder methods. -5. **Don't keep `cache_version="1.0"`** — use `cache="auto"` for the common case, or `flyte.Cache(version_override=...)` for advanced control. +5. **Don't keep `cache_version="1.0"`** — use `cache="auto"` for the common case, or `flyte.Cache(behavior="override", version_override=...)` to keep a manually bumped version. 6. **Don't read secrets via `current_context().secrets.get(...)`** — declare them on the `TaskEnvironment` and read the injected environment variable with `os.environ` / `os.getenv`. 7. **Don't recreate `LaunchPlan` + `CronSchedule`** — use `flyte.Trigger` with `flyte.Cron` attached to the task, and deploy it with `flyte deploy`. 8. **Don't run `pyflyte ... --remote`** — `flyte run` is remote by default; add `--local` explicitly for in-process runs. diff --git a/plugins/flyte/skills/flyte-migrate-control-flow/SKILL.md b/plugins/flyte/skills/flyte-migrate-control-flow/SKILL.md index 13dcab8..1b9bd1a 100644 --- a/plugins/flyte/skills/flyte-migrate-control-flow/SKILL.md +++ b/plugins/flyte/skills/flyte-migrate-control-flow/SKILL.md @@ -166,12 +166,18 @@ async def train_fold(max_depth: int) -> float: async def main(max_depth: int) -> float: try: return await train_fold(max_depth) - except ValueError as e: + except flyte.errors.RuntimeUserError as e: + # A failed sub-task does not re-raise its original exception type. + # It arrives as RuntimeUserError, with the original class name in e.code. + if e.code != "ValueError": + raise print(f"invalid hyperparameter ({e}); falling back to a safe default") # Recover with a safe default instead of failing the whole run. return await train_fold(max_depth=6) ``` +An `except ValueError:` around `await train_fold(...)` never matches, either on a cluster or with `flyte run --local`: the parent receives `flyte.errors.RuntimeUserError`, and the original exception's class name is in its `code`. Check `e.code` as above, or catch `Exception` when any failure should be handled (the usual replacement for `on_failure`). + Flyte 2 also exposes typed errors, so you can catch a specific failure and retry with more resources — a common need for memory-hungry training jobs: ```python @@ -234,6 +240,16 @@ def main(n: int, model_version: int) -> list[int]: return list(flyte.map(bound, range(n), concurrency=10)) ``` +`flyte.map` accepts a `partial` only if it leaves **exactly one** parameter unbound. A Flyte 1 `map_task(partial(...))` that mapped over several list inputs at once fails with `Partial function must leave exactly one parameter unspecified for mapping. Found 2 unspecified parameters`. Pass one iterable per parameter instead, positionally, zipped like the built-in `map`: + +```python +# Flyte 1: map_task(multi_input_task)(quantity=list_q, price=list_p, shipping=list_s) +@env.task +def main(list_q: list[int], list_p: list[float], shipping: float) -> list[float]: + # One iterable per parameter, in parameter order; repeat a constant to the same length. + return list(flyte.map(multi_input_task, list_q, list_p, [shipping] * len(list_q))) +``` + ### Flyte 2 (asyncio.gather) ```python diff --git a/plugins/flyte/skills/flyte-migrate-data-io/SKILL.md b/plugins/flyte/skills/flyte-migrate-data-io/SKILL.md index fa83f34..e886c3e 100644 --- a/plugins/flyte/skills/flyte-migrate-data-io/SKILL.md +++ b/plugins/flyte/skills/flyte-migrate-data-io/SKILL.md @@ -27,10 +27,36 @@ Flyte 2 renames the offloaded-data types and makes their I/O `async`, but the me | `flytekit.types.structured.StructuredDataset` | `flyte.io.DataFrame` | build with `from_df`, read with `open(...).all()` | | `@dataclass_json` + `@dataclass` | plain `@dataclass` | no mixin needed | | Pydantic `BaseModel` (+ config) | plain Pydantic `BaseModel` | works directly as task I/O | +| `np.ndarray` (offloaded as `.npy`), and other types with no transformer | No transformer; falls back to pickle | Pass large values as a `File` (see below) | + +### Values with no type transformer, such as `np.ndarray` + +Flyte 1 had a numpy transformer that offloaded arrays as `.npy` files. Flyte 2 has none, so an `np.ndarray`, like any type without a transformer, falls back to pickle. The pickle fallback decides whether to store the value inline or offload it by `sys.getsizeof`, and stores it inline below 10 KB. A sliced view, such as the halves of a train/test split, reports about 160 bytes however large the data, so it is stored inline. A large dataset passed this way fails with `Inputs exceed max_bytes limit of 10.0 MB, actual size: 23.4 MB`. + +Pass large arrays as a `File` instead: + +```python +import numpy as np +from flyte.io import File + +@env.task +async def load_data() -> File: + x_train, y_train = ... + np.savez("/tmp/data.npz", x_train=x_train, y_train=y_train) + return await File.from_local("/tmp/data.npz") + +@env.task +async def train(data: File) -> float: + arrays = np.load(await data.download()) + x_train, y_train = arrays["x_train"], arrays["y_train"] + ... +``` + +A pickled value also needs its library in the image of every task that receives it, including the calling task (see `flyte-migrate-tasks-workflows`). ## Offloaded Data: The Mental Model -`File`, `Dir`, and `DataFrame` are lightweight references (pointers) to data offloaded in blob storage — not the materialized bytes. In Flyte 2 the read/write operations are `async`: upload with `await File.from_local(local_path)`, read with `async with f.open("rb") as fh: await fh.read()`, build a frame with `flyte.io.DataFrame.from_df(df)` (sync constructor), and read it with `await fdf.open(pandas.DataFrame).all()`. +`File`, `Dir`, and `DataFrame` are lightweight references (pointers) to data offloaded in blob storage — not the materialized bytes. In Flyte 2 the read/write operations are `async`: upload with `await File.from_local(local_path)`, read with `async with f.open("rb") as fh: bytes(await fh.read())`, build a frame with `flyte.io.DataFrame.from_df(df)` (sync constructor), and read it with `await fdf.open(pandas.DataFrame).all()`. ## Files and Directories @@ -81,7 +107,8 @@ async def write_file(content: str) -> File: @env.task async def read_file(f: File) -> str: async with f.open("rb") as fh: - return (await fh.read()).decode("utf-8") + # read() returns an obstore Bytes buffer, which has no .decode(); convert to bytes first. + return bytes(await fh.read()).decode("utf-8") @env.task async def main(content: str) -> str: @@ -145,7 +172,20 @@ async def main() -> float: return await total_payroll(await make_df()) ``` -Add the dataframe dependencies (for example `pandas` and `pyarrow`) to the `TaskEnvironment` image. See [DataFrames](https://www.union.ai/docs/v2/flyte/user-guide/task-programming/dataframes) for more. +Flyte 1 converted a `pd.DataFrame` to a `StructuredDataset` implicitly when it flowed between tasks. Flyte 2 does not: inside a driver task, a sub-task that returns `pd.DataFrame` hands back a real pandas object, and passing it to a sub-task typed `flyte.io.DataFrame` fails with `Type conversion failed ... Did you forget to wrap your dataframe in a DataFrame instance?`. Wrap it at the call site: + +```python +@env.task +async def get_df() -> pd.DataFrame: + return pd.DataFrame({"salary": [50000, 60000]}) + +@env.task +async def main() -> float: + pandas_df = await get_df() + return await total_payroll(flyte.io.DataFrame.from_df(pandas_df)) +``` + +Add the dataframe dependencies (for example `pandas` and `pyarrow`) to the `TaskEnvironment` image. `pyarrow` is needed whenever a task returns a `pd.DataFrame` or a `flyte.io.DataFrame`, because both are written as Parquet; without it the task fails while encoding its output. See [DataFrames](https://www.union.ai/docs/v2/flyte/user-guide/task-programming/dataframes) for more. ## Dataclasses and Structured Types @@ -307,3 +347,4 @@ async def main() -> int: 4. **Don't materialize large data into task outputs** — return `File`, `Dir`, or `DataFrame` references, not the raw bytes or full frames. 5. **Don't forget the dataframe dependencies** — add `pandas` and `pyarrow` (or your engine) to the `TaskEnvironment` image so DataFrame I/O works remotely. 6. **Don't import from `flytekit.types.*`** — import `File` and `Dir` from `flyte.io`, and use `flyte.io.DataFrame`. +7. **Don't pass large `np.ndarray`s between tasks** — there is no numpy transformer; pickled arrays (especially sliced views) can be stored inline and break the 10 MB limit. Save to `.npz` and pass a `File`. diff --git a/plugins/flyte/skills/flyte-migrate-plugins/SKILL.md b/plugins/flyte/skills/flyte-migrate-plugins/SKILL.md new file mode 100644 index 0000000..b208454 --- /dev/null +++ b/plugins/flyte/skills/flyte-migrate-plugins/SKILL.md @@ -0,0 +1,192 @@ +--- +name: flyte-migrate-plugins +description: Maps Flyte 1 plugins and agents (flytekitplugins-*) to their Flyte 2 equivalents (flyteplugins-*), lists the v1 integrations that have no v2 plugin and what to use instead, and shows the three ways a v2 plugin attaches to a task (plugin_config on a TaskEnvironment, a standalone plugin task wrapped with TaskEnvironment.from_task, or a decorator). Use when migrating Flyte 1 code that imports flytekitplugins, uses task_config=, or uses a connector/agent task such as BigQueryTask, SnowflakeTask, Spark, Ray, Dask, PyTorch, NotebookTask, or DbtTask. Trigger words are flytekitplugins, flyteplugins, plugin, agent, connector, task_config, plugin_config, Spark, Ray, Dask, BigQuery, Snowflake, Databricks, papermill, NotebookTask, dbt, wandb, mlflow, pandera, kfpytorch, Elastic. +--- + +# Flyte 1 to 2 Migration: Plugins and Connectors + +Flyte 1 integrations ship as `flytekitplugins-` packages. Some have a Flyte 2 package, `flyteplugins-` (imported as `flyteplugins.`), often with a changed API. Many have none, and the right move is a plain `@env.task` that calls the underlying library. Check the table below before porting a plugin, and keep its API notes in mind: several changes import fine but break at runtime. + +The mapping and examples below come from migrating the plugin examples in flytesnacks to Flyte 2 (SDK and plugins 2.12.1). The "Verified" column says how far each one was tested. + +## Grounding References + +| Resource | URL | +|---|---| +| Migration guide | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/ | +| Integrations docs | https://www.union.ai/docs/v2/flyte/integrations/ | +| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ | +| Plugin source | https://github.com/flyteorg/flyte-sdk/tree/main/plugins | +| Flyte MCP tools | Available via `flyte-mcp` server (`search_flyte_sdk_examples` finds e.g. `bigquery_example.py`) | + +## v1 plugins with a Flyte 2 plugin + +Install the v2 package in the task image (`.with_pip_packages("flyteplugins-")`) and in the local environment you run `flyte run` from. + +| Flyte 1 | Flyte 2 package and API | How it attaches | Verified | +|---|---|---|---| +| `flytekitplugins-pandera` | `flyteplugins-pandera`; use `pandera.typing.DataFrame[Schema]` as before | Type transformer; install it in the image, no import needed | Ran on a cluster | +| `flytekitplugins-papermill` `NotebookTask` | `flyteplugins-papermill` `NotebookTask(task_environment=env, inputs={...}, outputs={...})` | Standalone task | Ran on a cluster | +| `flytekitplugins-mlflow` `@mlflow_autolog` | `flyteplugins-mlflow` `@mlflow_run(autolog=True, framework="keras")` | Decorator, outermost | Ran on a cluster | +| `flytekitplugins-wandb` `@wandb_init` | `flyteplugins-wandb` `@wandb_init(project=..., entity=...)`, `get_wandb_run()` | Decorator, outermost | Ran offline | +| `flytekitplugins-spark` | `flyteplugins-spark` `Spark(spark_conf=...)` | `plugin_config` | Imports and serializes | +| `flytekitplugins-dask` | `flyteplugins-dask` `Dask(scheduler=Scheduler(...), workers=WorkerGroup(...))` | `plugin_config` | Imports and serializes | +| `flytekitplugins-ray` | `flyteplugins-ray` `RayJobConfig(head_node_config=..., worker_node_config=[...])` | `plugin_config` | Imports and serializes | +| `flytekitplugins-kfpytorch` `Elastic` | `flyteplugins-pytorch` `Elastic(nnodes=..., nproc_per_node=...)` | `plugin_config` | Imports and serializes | +| `flytekitplugins-databricks` (`Databricks`, `DatabricksV2`) | `flyteplugins-databricks` `Databricks(spark_conf=..., databricks_conf=..., databricks_instance=...)` | `plugin_config` | Imports and serializes | +| `flytekitplugins-bigquery` | `flyteplugins-bigquery` `BigQueryTask(..., plugin_config=BigQueryConfig(ProjectID=...))` | Standalone task | Imports and serializes | +| `flytekitplugins-snowflake` | `flyteplugins-snowflake` `Snowflake(..., plugin_config=SnowflakeConfig(...))` | Standalone task | Imports and serializes | +| `flytekitplugins-dbt` | `flyteplugins-dbt` `DbtTask(task_environment=env, ...)` | Standalone task | Imports and serializes | +| `flytekitplugins-slurm` | `flyteplugins-slurm` `SlurmScriptTask`, `Slurm(...)` (see `flyte-migrate-slurm`) | Standalone task or `plugin_config` | Imports and serializes | +| `JSONLFile` (`flytekitplugins-openai`) | `flyteplugins-jsonl` `JsonlFile` | Type | Imports and serializes | + +"Imports and serializes" means the module imports and every task serializes the way `flyte deploy` would, but nothing was run against the service. Test against your own account before relying on it. + +## v1 integrations with no Flyte 2 plugin + +Do not look for a package; write a regular `@env.task` that uses the library directly, with credentials from `secrets=` on the environment. + +| Flyte 1 | Flyte 2 replacement | +|---|---| +| `flytekitplugins-kfpytorch` `PyTorch` (PyTorchJob with `Worker(replicas=N)`) | `flyteplugins-pytorch` `Elastic`, or `flyte.clustered.ClusteredTaskEnvironment` (see "Multi-node training" in `flyte-migrate-slurm`) | +| `flytekitplugins-kftensorflow` (TFJob), `flytekitplugins-kfmpi` (MPIJob, Horovod) | No equivalent; run single-node, or restructure around `Elastic` / a clustered environment | +| `flytekitplugins-openai` `ChatGPTTask`, `create_batch` | The `openai` client in a regular task. Note: `flyteplugins-openai` exists, but it only provides `function_tool` for the OpenAI Agents SDK; it is not a replacement | +| Athena, Hive, `SQLAlchemyTask`, `SQLite3Task`, `DuckDBQuery` | `pyathena`, `pyhive`, `sqlalchemy`, `sqlite3`, `duckdb` in a regular task | +| Great Expectations, whylogs, modin, memray, ONNX, Dolt | The library in a regular task; values without a type transformer fall back to pickle, or pass a `File`. Don't pass a modin `DataFrame` between tasks: the pickle fallback calls `sys.getsizeof` on it, which raises (on a cluster this surfaces only as a Ray crash). Convert with `._to_pandas()` and pass a pandas `DataFrame` | +| Comet ML, Neptune | `comet_ml.login()` / `neptune.init_run()` in the task | +| AWS Batch | Drop the config; tasks run as Kubernetes pods | +| SageMaker inference, mmcloud, Perian | `boto3` (or the vendor SDK) in a regular task, or plain tasks with `flyte.Resources` | +| `flytekitplugins-inference` (NIM, Ollama sidecars) | A `flyte.app.AppEnvironment` serving the model, with `TaskEnvironment(depends_on=[app])` and `app.endpoint` (see `flyte-sdk-app`) | +| Airflow agent (Airflow operators inside Flyte) | Re-implement the operator as a task (for example, a `FileSensor` becomes a polling task with `flyte.storage.exists`) | +| Airflow provider (Flyte operators inside Airflow) | An Airflow `PythonOperator` that calls `flyte.remote` | +| `flytekitplugins-flyteinteractive` `@vscode` | `flyte run --debug` (run-level only; no per-task decorator) | +| `flytekitplugins-flyteinteractive` `@jupyter` | No equivalent | +| `flytekitplugins-deck-standard` renderers | Hand-built HTML with `flyte.report` | + +## How a v2 plugin attaches to a task + +`task_config=` becomes `plugin_config=`, and `kwtypes(a=int)` becomes a plain dict `{"a": int}`. Beyond that, plugins attach in one of three ways. + +### 1. `plugin_config` on a `TaskEnvironment` (Spark, Dask, Ray, Elastic, Databricks, Slurm functions) + +The plugin config moves from each task to the environment. Tasks in a plugin environment are workers, so call them from a separate driver environment that lists the plugin environment in `depends_on`: + +```python +import flyte +from flyteplugins.spark import Spark + +image = ( + flyte.Image.from_base("apache/spark-py:v3.4.0") + .clone(name="spark", python_version=(3, 10), extendable=True) + .with_pip_packages("flyteplugins-spark") +) + +spark_env = flyte.TaskEnvironment( + name="spark_pi", + image=image, + resources=flyte.Resources(cpu="1", memory="2000M"), + plugin_config=Spark(spark_conf={"spark.executor.instances": "2"}), +) + +driver = flyte.TaskEnvironment(name="spark_driver", image=image, depends_on=[spark_env]) + +@spark_env.task +async def hello_spark(partitions: int) -> float: + # Flyte 1: flytekit.current_context().spark_session + sess = flyte.ctx().data["spark_session"] + ... + +@driver.task +async def main(partitions: int = 10) -> float: + return await hello_spark(partitions) +``` + +### 2. A standalone plugin task wrapped with `TaskEnvironment.from_task` (BigQuery, Snowflake, Slurm scripts) + +Query and connector tasks are objects, not `@env.task` functions, so they have no environment. Running one directly fails with `RuntimeSystemError: Task has no parent environment name`. Group them with `TaskEnvironment.from_task` and depend on that group: + +```python +import flyte +from flyte.io import DataFrame +from flyteplugins.bigquery import BigQueryConfig, BigQueryTask + +bq_query = BigQueryTask( + name="sql.bigquery.w_io", + inputs={"version": int}, # Flyte 1: kwtypes(version=int) + output_dataframe_type=DataFrame, + plugin_config=BigQueryConfig(ProjectID="my-project"), # Flyte 1: task_config= + query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 10;", +) + +bigquery_env = flyte.TaskEnvironment.from_task("bigquery_env", bq_query) + +env = flyte.TaskEnvironment( + name="bigquery_example", + image=flyte.Image.from_debian_base().with_pip_packages("flyteplugins-bigquery", "pandas", "pyarrow"), + depends_on=[bigquery_env], +) + +@env.task +async def main(version: int) -> DataFrame: + return await bq_query(version=version) +``` + +`NotebookTask` and `DbtTask` are also standalone, but they take `task_environment=env` directly, so they need no `from_task`. + +### 3. A decorator (wandb, mlflow) + +The plugin decorator must be the **outermost** decorator, above `@env.task`. This is the reverse of the usual Flyte 1 order: + +```python +from flyteplugins.wandb import get_wandb_run, wandb_init + +@wandb_init(project="my-project", entity="my-team") +@env.task +async def train() -> float: + run = get_wandb_run() + ... +``` + +## API changes that break a literal port + +These import and serialize fine, so only a run catches them. + +- **Snowflake:** `SnowflakeTask` becomes `Snowflake`. Query parameters use printf style, `%(nation_key)s`, instead of Go templates, `{{ .inputs.nation_key }}`. `SnowflakeConfig(user=...)` is now required. +- **wandb:** no `secret=` argument; the API key is read from the `WANDB_API_KEY` environment variable, so declare the secret with `flyte.Secret(key="wandb-api-key", as_env_var="WANDB_API_KEY")`. +- **mlflow:** `@mlflow_autolog(framework=mlflow.keras)` becomes `@mlflow_run(autolog=True, framework="keras")`. +- **dbt:** the three task classes (`DBTRun`, `DBTTest`, `DBTFreshness`) and their input/output dataclasses collapse into one `DbtTask`, which you call with a dbt command and which returns `list[DbtNodeResult]`. +- **Databricks:** both v1 `Databricks` (backend plugin) and `DatabricksV2` (agent) map to the same v2 `Databricks` class. The Spark session comes from `flyte.ctx().data["spark_session"]`. +- **Spark:** `current_context().spark_session` becomes `flyte.ctx().data["spark_session"]`. + +### papermill `NotebookTask` + +- **It is a sync task.** Calling it with `await nb(...)` from an `async` task fails with `SyncTaskCallInAsyncContextError`. Use `.aio()`: + + ```python + @env.task + async def main(v: float) -> float: + return await nb.aio(v=v) # not `await nb(v=v)` + ``` + +- **The notebook is not shipped with your code.** `flyte run` bundles only the Python modules it loads, so the pod fails with `FileNotFoundError: /home/flyte//.ipynb`. Bake the notebook into the image at that path. `dst` is the full destination file path: + + ```python + image = ( + flyte.Image.from_debian_base() + .with_pip_packages("flyteplugins-papermill") + .with_source_file(NB_DIR / "nb_simple.ipynb", dst="/home/flyte/papermill_plugin/nb_simple.ipynb") + ) + ``` + +- **It does not inherit the environment's `resources`.** Set `resources=` on the `NotebookTask` itself, or a memory-hungry notebook is OOMKilled. +- **DataFrame outputs fail** with `No transformers could reverse Flyte literal type structured_dataset_type` in 2.12.1. Write the frame to Parquet and output a `flyte.io.File` instead. +- **String outputs containing quotes break `record_outputs`.** Emit them with `display({"text/plain": record_outputs(...)}, raw=True)`. + +## Anti-Patterns + +1. **Don't install `flytekitplugins-*` in a Flyte 2 image.** They depend on `flytekit`; use the `flyteplugins-*` package or the underlying library. +2. **Don't assume a same-named package is a replacement.** `flyteplugins-openai` is an Agents SDK helper, not a port of `ChatGPTTask`. +3. **Don't call a plugin task without `depends_on`.** Wrap standalone tasks with `TaskEnvironment.from_task`, and list every plugin environment in the caller's `depends_on`. +4. **Don't keep `task_config=` or `kwtypes(...)`.** Use `plugin_config=` and a plain dict. +5. **Don't put a plugin decorator inside `@env.task`.** `@wandb_init` and `@mlflow_run` go on top. +6. **Don't trust a load check for connectors.** Query templating, credentials, and secrets only fail at runtime; run one small query before migrating the rest. diff --git a/plugins/flyte/skills/flyte-migrate-slurm/SKILL.md b/plugins/flyte/skills/flyte-migrate-slurm/SKILL.md index e23f165..63ae053 100644 --- a/plugins/flyte/skills/flyte-migrate-slurm/SKILL.md +++ b/plugins/flyte/skills/flyte-migrate-slurm/SKILL.md @@ -494,7 +494,50 @@ Do not start with the thing Slurm does best. 2. **Single-node training next** — one `TaskEnvironment`, one GPU resource string, checkpoints to object storage. At this point you've validated that your image builds, your data streams, and your logs are where you expect. 3. **Multi-node training last** — once images, data, and observability are proven. `ClusteredTaskEnvironment` is the piece with the most moving parts and the least tolerance for a half-migrated environment. -Nothing forces a big bang: Slurm and Flyte can run side by side indefinitely, and a Flyte task can `subprocess` out to `sbatch` during the overlap if you need a bridge. +Nothing forces a big bang: Slurm and Flyte can run side by side indefinitely. For the overlap, use the Slurm connector below rather than shelling out to `sbatch`. + +## Keeping jobs on Slurm: `flyteplugins-slurm` + +When a job should stay on the Slurm cluster, the `flyteplugins-slurm` connector submits it over SSH and tracks it from Flyte. It is also the Flyte 2 replacement for the Flyte 1 Slurm agent (`flytekitplugins-slurm`): + +| Flyte 1 (`flytekitplugins-slurm`) | Flyte 2 (`flyteplugins-slurm`) | +|---|---| +| `SlurmTask(task_config=SlurmRemoteScript(batch_script_path=...))` | `SlurmScriptTask(script="#!/bin/bash\nbash /path/to/job.sh", plugin_config=Slurm(...))` | +| `SlurmShellTask(script=..., task_config=Slurm(...))` | `SlurmScriptTask(script=..., plugin_config=Slurm(...), inputs={...})` | +| `@task(task_config=SlurmFunction(...))` | `TaskEnvironment(plugin_config=Slurm(...))` + `@env.task` | + +`Slurm(...)` takes the connection (`host`, `username`, `ssh_private_key` as a Flyte secret name) and the job options (`partition`, `nodes`, `ntasks`, `mem`, `time_limit`, `modules`, `sbatch_options={...}`, ...). + +A `SlurmScriptTask` is not created through an `@env.task` decorator, so it has no parent environment, and running it fails with `RuntimeSystemError: Task basic has no parent environment name`. Wrap it with `TaskEnvironment.from_task` and make the calling environment depend on that: + +```python +import flyte +from flyteplugins.slurm import Slurm, SlurmScriptTask + +slurm = Slurm( + host="login.hpc.example.com", + username="ubuntu", + ssh_private_key="slurm-ssh-private-key", # name of a Flyte secret + partition="debug", + sbatch_options={"job-name": "job0"}, +) + +job = SlurmScriptTask( + name="basic", + script="#!/bin/bash\nbash /home/ubuntu/echo.sh\n", + plugin_config=slurm, +) + +slurm_env = flyte.TaskEnvironment.from_task("slurm_scripts", job) + +driver = flyte.TaskEnvironment(name="slurm_driver", depends_on=[slurm_env]) + +@driver.task +async def main() -> None: + await job() +``` + +Function tasks that run on Slurm need `flyteplugins-slurm` in their image (`Image.from_debian_base().with_pip_packages("flyteplugins-slurm")`). For other plugins, see `flyte-migrate-plugins`. ## Gotchas @@ -527,5 +570,6 @@ Nothing forces a big bang: Slurm and Flyte can run side by side indefinitely, an - **`flyte-sdk-ml`** — greenfield ML authoring in Flyte 2 (training, HPO, inference) once the migration shape is clear. - **`flyte-migrate`** — the entry point if you _also_ have Flyte 1 (`flytekit`) code to port. +- **`flyte-migrate-plugins`** — `flytekitplugins-*` to `flyteplugins-*`, including the Slurm connector. - **`flyte-sdk-ship`** — image specs, dependency management, and reproducible builds, i.e. everything that replaces `module load`. - **`flyte-sdk-app`** — serving and endpoints, for the step after training that Slurm never covered. diff --git a/plugins/flyte/skills/flyte-migrate-tasks-workflows/SKILL.md b/plugins/flyte/skills/flyte-migrate-tasks-workflows/SKILL.md index 6e4f767..4fb9bc8 100644 --- a/plugins/flyte/skills/flyte-migrate-tasks-workflows/SKILL.md +++ b/plugins/flyte/skills/flyte-migrate-tasks-workflows/SKILL.md @@ -228,6 +228,57 @@ def my_task(x: int) -> int: return x ``` +### Calling tasks in another environment: `depends_on` + +In Flyte 1 each `@task` could set its own `container_image`, and a workflow could mix them freely. In Flyte 2, tasks with different images live in different `TaskEnvironment`s, and the environment of the calling task must declare the environments it calls in `depends_on`. Without it, the call fails at runtime with `Environment 'mindmeld' not found in image cache ... declare that dependency using 'depends_on=[...]'`: the callee's image was never resolved for the run. (Environments that share the same image object may happen to work without it; declare the dependency anyway.) + +```python +# Flyte 1: @task(container_image=sklearn_image) def get_data(): ... +mindmeld = flyte.TaskEnvironment( + name="mindmeld", + image=flyte.Image.from_debian_base().with_pip_packages("scikit-learn", "numpy"), +) + +borebuster = flyte.TaskEnvironment( + name="borebuster", + image=flyte.Image.from_debian_base().with_pip_packages("numpy"), + depends_on=[mindmeld], # the entrypoint below calls get_data, which runs in mindmeld +) + +@mindmeld.task +def get_data() -> list[float]: + ... + +@borebuster.task +def main() -> list[float]: + return get_data() +``` + +The same applies to tasks not created with an `@env.task` decorator, such as plugin tasks (see `flyte-migrate-plugins`) and shell tasks: wrap them in an environment with `flyte.TaskEnvironment.from_task("name", task_a, task_b)` and add that environment to `depends_on`. + +**`depends_on` must not form a cycle.** If the driver environment depends on a worker environment and the worker also calls back into a task in the driver environment (`driver.depends_on=[worker]` plus `worker.add_dependency(driver)`), `flyte run` fails at submit with `maximum recursion depth exceeded`. Move the shared task into a third environment that both depend on: + +```python +helpers = flyte.TaskEnvironment(name="helpers") # stringify() lives here +worker = flyte.TaskEnvironment(name="worker", depends_on=[helpers]) +driver = flyte.TaskEnvironment(name="driver", depends_on=[worker, helpers]) +``` + +**The caller's image must be able to deserialize what it receives.** A calling task materializes every sub-task output. If a task in another environment returns a value that falls back to pickle, such as a scikit-learn model, the caller's image needs that library too. Otherwise the caller fails with `Error converting input ... No module named sklearn`. This holds even when `depends_on` is correct. Add the library to the caller's image, or return a `File` (for example a saved model) instead of the object. + +### Mixing sync and async tasks + +Flyte 1 tasks were all called the same way. In Flyte 2, a task is sync or async depending on its `def`, and the call must match: + +| Caller | Callee | How to call | +|---|---|---| +| sync | sync | `result = t(x)` | +| async | async | `result = await t(x)` | +| async | sync (including plugin tasks like `NotebookTask`) | `result = await t.aio(x)`. A plain `t(x)` fails with `Sync task '...' was called in a blocking way from async code ... Use await t.aio(...)` | +| sync | async | Don't. The call returns an un-awaited coroutine, and the caller fails later with `Expected value of type but got ''`. Make the caller `async`, or the callee sync | + +A literal port of a Flyte 1 workflow is sync throughout, so the simplest rule is: keep a migrated pipeline all sync or all async, and use `.aio()` at the boundary when an async driver calls a sync task. + ## Parameter Mapping: `@task` to `TaskEnvironment` + `@env.task` | Flyte 1 `@task` parameter | Flyte 2 location | Notes | @@ -238,7 +289,7 @@ def my_task(x: int) -> int: | `environment` | `TaskEnvironment(env_vars=...)` | Env-level only | | `secret_requests` | `TaskEnvironment(secrets=...)` | Env-level only | | `cache` | Both | Can override at task level | -| `cache_version` | `flyte.Cache(version_override=...)` | In a `Cache` object | +| `cache_version` | `flyte.Cache(behavior="override", version_override=...)` | In a `Cache` object | | `retries` | `@env.task(retries=...)` | Task-level only | | `timeout` | `@env.task(timeout=...)` | Task-level only | | `interruptible` | Both | Can override at task level | @@ -256,3 +307,6 @@ For image, resource, secret, and caching detail, see the Task configuration migr 4. **Don't put heavy compute in the orchestrating task** — keep the entrypoint task focused on calling other tasks; push CPU/GPU/memory-intensive work into leaf tasks whose resources you can tune per environment. 5. **Don't set image, resources, or secrets on the `@env.task` decorator** — those are env-level and belong on `TaskEnvironment`. Only per-task overrides like `retries`, `timeout`, `cache`, and `short_name` go on `@env.task`. 6. **Don't keep passing every argument by keyword** — Flyte 2 task calls accept positional arguments (`say_hello(name)`). +7. **Don't call a task in another `TaskEnvironment` without `depends_on`** — the caller's environment must list the callee's environment, or the run fails with `Environment '...' not found in image cache`. +8. **Don't make `depends_on` circular** — when two environments call each other, move the shared tasks into a third environment that both depend on. +9. **Don't call an async task from a sync task** — it returns a coroutine instead of a result. From an async task, call sync tasks with `await t.aio(...)`. diff --git a/plugins/flyte/skills/flyte-migrate-troubleshoot/SKILL.md b/plugins/flyte/skills/flyte-migrate-troubleshoot/SKILL.md new file mode 100644 index 0000000..c9ba3cd --- /dev/null +++ b/plugins/flyte/skills/flyte-migrate-troubleshoot/SKILL.md @@ -0,0 +1,541 @@ +--- +name: flyte-migrate-troubleshoot +description: Diagnoses and fixes errors that show up when running Flyte 1 (flytekit) code freshly migrated to Flyte 2 (flyte), such as submit-time import failures, missing depends_on between environments, pandas DataFrames that are not wrapped, exceptions that are not caught, notebooks missing from the code bundle, File inputs from https URLs, NamedTuple outputs that lose their fields, shell tasks that fail with an empty error message, circular depends_on, sync/async call mismatches, flyte.map partials with several unbound inputs, oversized numpy inputs, and calling tasks missing a library to unpickle a sub-task output. Use when a migrated workflow fails on `flyte run` (locally or remotely) and the error matches one of the known migration failures. Trigger words are migration error, migrated code fails, attempted relative import with no known parent package, not found in image cache, depends_on, has no parent environment name, Did you forget to wrap your dataframe, RuntimeUserError, non-extendable image, Bytes object has no attribute decode, _LazyModule, FileNotFoundError ipynb, Timeout context manager, NamedTuple, has no attribute, ShellTask, shell.create, exit code 1, empty error message, maximum recursion depth exceeded, called in a blocking way, aio, coroutine object, Partial function must leave exactly one parameter, max_bytes limit, No module named, unexpected extra argument, troubleshoot, debug migration. +--- + +# Flyte 1 to 2 Migration: Troubleshooting + +This skill covers errors observed when running migrated code, not how to write it. Each entry gives the exact symptom, the cause, and a verified fix. Start with the debug loop below, then find the error in the index. + +Every entry here was reproduced by migrating the flytesnacks examples (153 entrypoints) to Flyte 2 (SDK 2.12.1) and running them on a cluster. About one entrypoint in five failed on its first run because of the migration; entries 1-11 are those causes. Entries 12-17 come from a second run with these skills (2026-10-29), where 7% still failed for new reasons. + +## Grounding References + +| Resource | URL | +|---|---| +| Migration guide | https://www.union.ai/docs/v2/flyte/user-guide/migration/flyte-2/ | +| Official docs | https://www.union.ai/docs/v2/flyte | +| SDK API reference | https://www.union.ai/docs/v2/union/api-reference/flyte-sdk/ | +| Flyte MCP tools | Available via `flyte-mcp` server | + +## Debug loop + +Run the cheapest check that can catch the error first: + +1. **Compile:** `python -m py_compile my_module.py` catches syntax errors left over from the rewrite. +2. **Run locally:** `flyte run --local my_module.py main --arg value` runs every task in-process. Import errors and Python-level errors (entries 3, 4, 5, and 8 below) reproduce here, with a full traceback, in seconds. +3. **Run remotely:** `flyte run my_module.py main --arg value`, then `flyte get logs ` for the task's output. For runs and logs in depth, see the `flyte-sdk-run` skill. Errors about images, environments, and files missing from the pod (entries 2, 6, and 7) only show up here. + +Note at which stage the error appears. A failure before a run is created (no `Created Run:` line) is a load-time error in your module, not a cluster problem. + +## Error index + +| Error text (or symptom) | Stage | Entry | +|---|---|---| +| `ImportError: attempted relative import with no known parent package` | submit | 1 | +| `Environment '...' not found in image cache ... depends_on` | runtime | 2 | +| `RuntimeSystemError: Task ... has no parent environment name` | submit | 2 | +| `Type conversion failed ... Value received: [RuntimeUserError(...)]` after `flyte.map` | runtime | 2 | +| `Type conversion failed ... Did you forget to wrap your dataframe in a DataFrame instance?` | runtime | 3 | +| An `except SomeError:` around a task call never matches | runtime | 4 | +| `'builtins.Bytes' object has no attribute 'decode'` | runtime | 5 | +| `ValueError: Cannot add additional layers to a non-extendable image` | submit | 6 | +| `'_LazyModule' object has no attribute 'DataFrame'`, `Unable to find a usable engine` | runtime | 6 | +| `FileNotFoundError: /home/flyte//.ipynb` (or another non-`.py` file) | runtime | 7 | +| `argument should be a str or an os.PathLike object ..., not 'File'` | submit | 8 | +| `HTTPFileSystem requires "requests" and "aiohttp" to be installed` | runtime | 8 | +| `Timeout context manager should be used inside a task` | runtime | 8 | +| `'latte' is not one of 'ESPRESSO', 'AMERICANO', 'LATTE', ...` | submit | 9 | +| `Got unexpected extra argument (true)` for a bool input | submit | 9 | +| `AttributeError: '' object has no attribute ''` on a `NamedTuple` output | runtime | 10 | +| Shell task fails with `exit code (1)` and an empty message | runtime | 11 | +| `Execution failed: maximum recursion depth exceeded` | submit | 12 | +| `Sync task '...' was called in a blocking way from async code` | runtime | 13 | +| `Expected value of type but got ''` | runtime | 14 | +| `Partial function must leave exactly one parameter unspecified for mapping` | runtime | 15 | +| `Inputs exceed max_bytes limit of 10.0 MB` | runtime | 16 | +| `Error converting input ... No module named ''` in the calling task | runtime | 17 | + +## 1. `ImportError: attempted relative import with no known parent package` + +### Symptom + +`flyte run` fails before any run is created: + +```text +Error invoking command: Failed to load basics/launch_plan.py: ImportError: attempted +relative import with no known parent package +``` + +### Cause + +Flyte 1 examples often import sibling modules relatively (`from .workflow import slope`). `flyte run ` loads the file as a standalone module, not as part of its package, so relative imports have no parent package to resolve against. Passing `--root-dir` does not change this. + +### Fix + +Use absolute imports rooted at the package, and run `flyte run` from the package's parent directory. + +```python +# Flyte 1 (and a literal migration): fails under `flyte run` +from .workflow import env, intercept, slope +``` + +```python +# Flyte 2: absolute import from the package root +from basics.workflow import env, intercept, slope +``` + +```bash +# Run from the directory that contains the `basics/` package +cd examples/basics +flyte run basics/imperative_workflow.py imperative_wf --x '[-3,0,3]' --y '[7,4,-2]' +``` + +The remote container receives the same source tree, so the absolute import also resolves at runtime. + +## 2. A task in another environment: `not found in image cache` / `has no parent environment name` + +### Symptom + +A task calls a task from a different `TaskEnvironment` and fails: + +```text +Environment 'mindmeld' not found in image cache. ... declare that dependency using 'depends_on=[...]' +``` + +Or a plugin, connector, or shell task created as an object (not with `@env.task`) fails before the run starts: + +```text +RuntimeSystemError: Task basic has no parent environment name +``` + +Inside `flyte.map`, the first error is hidden: the failures come back as list items, and the next task fails with `Type conversion failed ... Actual type: Value received: [RuntimeUserError('Environment ... not found in image cache ...`. + +### Cause + +Flyte 1 let each `@task` set its own `container_image`. In Flyte 2, tasks with different images live in different environments, and the calling task's environment must declare the environments it calls in `depends_on`. Task objects such as `SlurmScriptTask`, `BigQueryTask`, `Snowflake`, or a `ContainerTask` have no environment at all until you give them one. + +### Fix + +```python +mindmeld = flyte.TaskEnvironment(name="mindmeld", image=sklearn_image) + +borebuster = flyte.TaskEnvironment( + name="borebuster", + image=numpy_image, + depends_on=[mindmeld], # a task in borebuster calls get_data, which runs in mindmeld +) +``` + +For a task object, wrap it first: + +```python +slurm_env = flyte.TaskEnvironment.from_task("slurm_scripts", slurm_job, other_job) +driver = flyte.TaskEnvironment(name="driver", depends_on=[slurm_env]) +# Or, for an environment that already exists: driver.add_dependency(slurm_env) +``` + +See `flyte-migrate-tasks-workflows` and `flyte-migrate-plugins` for the full pattern. + +## 3. `Did you forget to wrap your dataframe in a DataFrame instance?` + +### Symptom + +```text +Type conversion failed for variable 'fdf'. Reason: Expected a DataFrame instance, but got + instead. Did you forget to wrap your dataframe in a DataFrame instance? +``` + +### Cause + +Flyte 1 converted a `pd.DataFrame` to a `StructuredDataset` implicitly. In a Flyte 2 driver task, a sub-task that returns `pd.DataFrame` hands back a real pandas object, and a sub-task typed `flyte.io.DataFrame` will not accept it. + +### Fix + +Wrap it at the call site: + +```python +pandas_df = await get_df() +total = await total_payroll(flyte.io.DataFrame.from_df(pandas_df)) +``` + +## 4. An `except` around a task call never matches + +### Symptom + +```python +try: + return await raises_exc(x) +except TypeError: # never matches + return -1 +``` + +The run fails with `flyte.errors.RuntimeUserError` (`code: "TypeError"`) raised from the `await` line. + +### Cause + +A failed sub-task does not re-raise its original exception type in the parent, either on a cluster or with `flyte run --local`. It arrives as `flyte.errors.RuntimeUserError`, and the original exception's class name is in `code`. + +### Fix + +```python +try: + return await raises_exc(x) +except flyte.errors.RuntimeUserError as e: + if e.code == "TypeError": + return -1 + raise +``` + +Catch `Exception` when any failure should be handled, as in a replacement for `on_failure`. Typed subclasses such as `flyte.errors.OOMError` can still be caught directly. + +## 5. `'builtins.Bytes' object has no attribute 'decode'` + +### Cause + +`File.open()` returns a handle whose `read()` gives an obstore `Bytes` buffer, not Python `bytes`. + +### Fix + +```python +async with f.open("rb") as fh: + text = bytes(await fh.read()).decode("utf-8") +``` + +## 6. Image errors + +### `ValueError: Cannot add additional layers to a non-extendable image` + +The module fails to load. An image from `Image.from_base(uri)` (the Flyte 1 `ImageSpec(base_image=...)`) cannot be extended directly. Clone it first: + +```python +image = ( + flyte.Image.from_base("ghcr.io/flyteorg/rawcontainers-python:v2") + .clone(name="rawcontainers-python-pandas", extendable=True) + .with_pip_packages("pandas") +) +``` + +### `'_LazyModule' object has no attribute 'DataFrame'` + +This message hides an `ImportError` inside the image: Flyte imports pandas lazily, and when the real import fails, the lazy module raises this instead. The usual cause is a dependency pin conflict. For example, pinning `numpy<2` while `pyarrow` is unpinned installs a `pyarrow` that requires numpy 2, so pandas cannot import. Pin both sides (`"numpy<2", "pyarrow<18"`), or build the image and run `python -c "import pandas"` in it to see the real error. + +### `ImportError: Unable to find a usable engine` while writing an output + +A task returning `pd.DataFrame` or `flyte.io.DataFrame` writes Parquet and needs `pyarrow` in its image. + +## 7. `FileNotFoundError` for a notebook or other non-Python file + +### Symptom + +```text +[Errno 2] No such file or directory: '/home/flyte/papermill_plugin/nb_simple.ipynb' +``` + +### Cause + +`flyte run` bundles only the Python modules it loads. Notebooks, SQL files, and data files next to your code are not shipped. Note also that the task's working directory is `/home/flyte`, not `/root` as in many Flyte 1 images, so CWD-relative paths from v1 code may point elsewhere. + +### Fix + +Bake the file into the image. `dst` is the destination **file** path (not a folder, despite the docstring; a folder path turns the directory into a file and breaks code-bundle extraction with `tar: ... Not a directory`): + +```python +image = ( + flyte.Image.from_debian_base() + .with_pip_packages("flyteplugins-papermill") + .with_source_file(NB_DIR / "nb_simple.ipynb", dst="/home/flyte/papermill_plugin/nb_simple.ipynb") +) +``` + +For a whole folder of assets, use `.with_source_folder(...)`. + +## 8. `File` inputs that point at an `https://` URL + +Three separate errors, each from Flyte 1 code that passed a public URL as a `FlyteFile`: + +| Error | Cause | +|---|---| +| `argument should be a str or an os.PathLike object ..., not 'File'` (no run created) | An entrypoint input with a `File` default value breaks `flyte run`'s argument parsing. | +| `HTTPFileSystem requires "requests" and "aiohttp" to be installed` | The default image lacks the packages fsspec needs for `https://`. | +| `Timeout context manager should be used inside a task` | `await f.download()` of an `https://` path fails in an async task (SDK 2.12.1). | + +### Fix + +Take the URL as a `str` and read it inside the task, either directly with the library (`pd.read_csv(url)`) or with a `File` and the sync API in a sync task, with `aiohttp` and `requests` in the image: + +```python +env = flyte.TaskEnvironment( + name="data", + image=flyte.Image.from_debian_base().with_pip_packages("pandas", "aiohttp", "requests"), +) + +@env.task +def normalize(csv_url: str) -> File: + local = File.from_existing_remote(csv_url).download_sync() + ... + return File.from_local_sync(out_path) +``` + +## 9. Enum and bool inputs on the CLI + +### Symptom + +```text +Error invoking command: 'latte' is not one of 'ESPRESSO', 'AMERICANO', 'LATTE', 'CAPPUCCINO'. +``` + +```text +Error invoking command: Got unexpected extra argument (true) +``` + +### Cause and fix + +The code needs no change in either case: + +- `pyflyte run` took an enum's value; `flyte run` takes the member **name**. Pass `--coffee_enum LATTE`. +- A `bool` input is a bare flag. Pass `--boolean_input` to set it to true, not `--boolean_input true`. + +## 10. `AttributeError: '' object has no attribute ''` on a NamedTuple output + +### Symptom + +A task that reads a `NamedTuple` output from another task by attribute fails at runtime. It also reproduces with `flyte run --local`: + +```text +File "basics/named_outputs.py", line 38, in simple_wf_with_named_outputs + intercept_value = intercept(x, y, slope_value.slope) +AttributeError: 'float' object has no attribute 'slope' +``` + +### Cause + +In Flyte 1, a workflow could read a named output by attribute (`slope_value.slope`). In Flyte 2, calling a task whose return type is a `NamedTuple` gives back: + +- for a **single-field** `NamedTuple`, the bare value (here a `float`); +- for a **multi-field** `NamedTuple`, a plain `tuple`, so `result.train_features` fails with `'tuple' object has no attribute 'train_features'`. + +The error is raised at the **call site**, not in the task that returns the `NamedTuple`. Changing the callee to return `slope_value(slope=...)` explicitly does not fix it. + +### Fix + +Use the returned value directly, or unpack a multi-field result by position. + +```python +# Flyte 1 style: fails in Flyte 2 +slope_value = slope(x, y) +intercept_value = intercept(x, y, slope_value.slope) +return slope_and_intercept_values(slope=slope_value.slope, intercept=intercept_value.intercept) +``` + +```python +# Flyte 2: a single-field NamedTuple output arrives as the bare value +slope_value = slope(x, y) +intercept_value = intercept(x, y, slope_value) +return slope_and_intercept_values(slope=slope_value, intercept=intercept_value) + +# A multi-field NamedTuple output arrives as a plain tuple +train_features, train_labels, test_features, test_labels = split_data(df) +``` + +Returning a multi-field `NamedTuple` from the top-level task still works. Be aware that the run's outputs are reported positionally as `o0`, `o1`, ... rather than by field name. + +## 11. Shell task fails with `exit code (1)` and an empty message + +### Symptom + +A task built with `flyte.extras.shell.create()` (the Flyte 2 replacement for `ShellTask`) fails, and the error carries no message from the script: + +```text +Pod failed. No message received from kubernetes. +[flyte-copilot-uploader] terminated with ExitCode 0. +[flyte-copilot-downloader] terminated with ExitCode 0. +[] terminated with exit code (1). Reason [Error]. Message: +. +``` + +The task's container log is empty; only the copilot sidecars log anything. + +### Why the message is empty + +The shell wrapper redirects the script's stdout and stderr to files (`/var/outputs/_stdout`, `/var/outputs/_stderr`), so nothing reaches the container log. To see what actually ran, read the container's arguments, which contain the script with every `{inputs.*}` and `{outputs.*}` placeholder already substituted: + +```bash +kubectl -n get pod -o jsonpath='{.spec.containers[0].args}' +``` + +Then check each path the script touches against the input paths the downloader logged (`Successfully copied ... to local [/var/inputs/...]`). + +### A common cause: output files are named after the output variable + +In Flyte 1, `OutputLocation(var="i", var_type=FlyteFile, location="{inputs.x}")` wrote the output to the input's path, so the file kept its original name (for example `test.txt`). In Flyte 2, a `File` output declared in `outputs={"i": File}` is always written to `{outputs.i}`, so downstream tasks receive a file named `i`. + +Any later script that derives a path from that file's name, such as `$(basename {inputs.x})`, now looks for the wrong file and exits with 1 under `set -e`. + +```python +# Flyte 1 +t1 = ShellTask( + name="task_1", + script="""echo "Showcasing Flyte's Shell Task." >> {inputs.x}""", + inputs=kwtypes(x=FlyteFile), + output_locs=[OutputLocation(var="i", var_type=FlyteFile, location="{inputs.x}")], +) +``` + +```python +# Flyte 2: write the output explicitly; downstream it is named "i" +from flyte.extras import shell +from flyte.io import File + +t1 = shell.create( + name="task_1", + image="debian:12-slim", + script=""" + echo "Showcasing Flyte's Shell Task." >> {inputs.x} + cp {inputs.x} {outputs.i} + """, + inputs={"x": File}, + outputs={"i": File}, +) +``` + +### Fix + +Make every name-dependent step use the file that actually flows through the pipeline. In the flytesnacks example, `task_2` copies `t1`'s output into a directory and `task_3` looks it up by basename, so pass `t1`'s output, not the original file, to `task_3`: + +```python +@env.task +async def shell_task_wf() -> File: + x, y = await create_entities() + t1_out = await t1(x=x) + t2_out = await t2(x=t1_out, y=y) + # Not `x=x`: the copied file is named after t1's output variable. + t3_out = await t3(x=t1_out, y=y, z=t2_out) + return t3_out +``` + +### Required wiring for shell tasks + +The `TaskEnvironment` whose tasks call shell tasks declares them in `depends_on`, as the SDK's shell examples do: + +```python +env = flyte.TaskEnvironment( + name="shell_task", + depends_on=[t1.env, t2.env, t3.env], +) +``` + +Shell tasks are async; `await` them from an `async` task. + +## 12. `maximum recursion depth exceeded` at submit + +### Cause + +The `depends_on` graph has a cycle. A typical case: the driver environment depends on a worker environment, and a worker task calls a helper that lives in the driver environment, so the worker was given `depends_on` (or `add_dependency`) back to the driver. + +### Fix + +Move the shared helper tasks into a third environment that both depend on: + +```python +helpers = flyte.TaskEnvironment(name="helpers") +worker = flyte.TaskEnvironment(name="worker", depends_on=[helpers]) +driver = flyte.TaskEnvironment(name="driver", depends_on=[worker, helpers]) +``` + +Every entrypoint in an environment on the cycle fails the same way, so fix the environments before re-running each entrypoint. + +## 13. `Sync task '...' was called in a blocking way from async code` + +### Symptom + +```text +Sync task 'simple-nb' was called in a blocking way from async code. This blocks the event loop and +can hang the run. Use `await simple-nb.aio(...)` instead. +``` + +### Cause + +An `async` task called a sync task directly. This is common with plugin tasks that are sync even though you `await` everything else, such as the papermill `NotebookTask`. + +### Fix + +```python +out = await nb.aio(v=f) # not `await nb(v=f)` or `nb(v=f)` +``` + +## 14. A task returns `` + +### Symptom + +```text +Expected value of type but got '' +``` + +### Cause + +A **sync** task called an **async** task. The call returns an un-awaited coroutine instead of running the task, and the error appears only when that coroutine is used as a value. + +### Fix + +Make the calling task `async` and `await` the call, or make the callee a plain `def` task. Keep a migrated pipeline all sync or all async (see `flyte-migrate-tasks-workflows`). + +## 15. `Partial function must leave exactly one parameter unspecified for mapping` + +### Cause + +Flyte 1 `map_task(partial(...))` could map over several list inputs at once. `flyte.map` maps over exactly one unbound parameter per `partial`. + +### Fix + +Drop the `partial` and pass one iterable per parameter, positionally: + +```python +results = list(flyte.map(multi_input_task, list_q, list_p, [shipping] * len(list_q))) +``` + +## 16. `Inputs exceed max_bytes limit of 10.0 MB` + +### Cause + +A large value without a type transformer, usually an `np.ndarray`, was passed between tasks. It falls back to pickle, which stores values inline when `sys.getsizeof` reports under 10 KB. Sliced arrays (views, as from a train/test split) report about 160 bytes regardless of their size, so the whole array is stored inline and breaks the 10 MB inline I/O limit. + +### Fix + +Save the arrays with `np.savez` and pass a `flyte.io.File` (see `flyte-migrate-data-io`). + +## 17. `No module named ''` while converting a task's input + +### Symptom + +The **calling** task fails while receiving a sub-task's output, even though the sub-task succeeded: + +```text +Error converting input o0 ... Expected Python type: typing.Any Exception: No module named 'sklearn' +``` + +### Cause + +The sub-task runs in another environment and returned a pickled object, such as a scikit-learn model. The calling task deserializes every sub-task output, so its own image needs the library too. A correct `depends_on` doesn't help here. + +### Fix + +Add the library to the calling environment's image, or return a `File` (for example the saved model) instead of the object. + +## Anti-Patterns + +1. **Don't debug a submit-time error on the cluster.** If no run was created, the module failed to load locally. Fix imports first. +2. **Don't skip `flyte run --local`.** Python-level errors such as the `NamedTuple`, DataFrame, and exception-type errors reproduce locally with a full traceback. +3. **Don't keep relative imports between workflow modules.** Use absolute imports from the package root. +4. **Don't call across environments without `depends_on`.** Wrap task objects with `TaskEnvironment.from_task` first. +5. **Don't pass a raw `pd.DataFrame` to a `flyte.io.DataFrame` input.** Wrap it with `flyte.io.DataFrame.from_df`. +6. **Don't catch the original exception type around a task call.** Catch `flyte.errors.RuntimeUserError` and check `.code`. +7. **Don't assume non-Python files reach the pod.** Bake notebooks and data files into the image. +8. **Don't use a `File` default or an async download for an `https://` URL.** Take a `str` and read it in the task. +9. **Don't read a `NamedTuple` output by attribute.** Use the value directly or unpack it by position. +10. **Don't assume a shell task's output keeps its input's file name.** It is named after the output variable. +11. **Don't rely on pod logs to explain a shell task failure.** The script's stderr is redirected to a file; inspect the substituted script in the container args. +12. **Don't make `depends_on` circular.** Put tasks that two environments both call into a third environment. +13. **Don't mix sync and async calls blindly.** From async code call sync tasks with `.aio()`; never call an async task from a sync one. +14. **Don't pass large `np.ndarray`s between tasks.** Pass an `.npz` `File`. +15. **Don't assume only the callee's image matters.** The caller needs every library required to unpickle what it receives. diff --git a/plugins/flyte/skills/flyte-migrate/SKILL.md b/plugins/flyte/skills/flyte-migrate/SKILL.md index c5a01db..f2e9ce2 100644 --- a/plugins/flyte/skills/flyte-migrate/SKILL.md +++ b/plugins/flyte/skills/flyte-migrate/SKILL.md @@ -224,6 +224,8 @@ Migrate by theme. Start with tasks and workflows, then jump to whatever the work - **`flyte-migrate-data-io`** — `FlyteFile`/`FlyteDirectory` → `flyte.io.File`/`Dir`, `StructuredDataset` → `flyte.io.DataFrame`, dataclasses, and ETL patterns. - **`flyte-migrate-ml`** — small-model training, hyperparameter optimization, deep learning, batch inference, and end-to-end pipelines. - **`flyte-migrate-slurm`** — Slurm/HPC workloads: `#SBATCH` directives → `TaskEnvironment` resources, job arrays → `flyte.map`, multi-node `srun` training → clustered environments. +- **`flyte-migrate-plugins`** — `flytekitplugins-*` → `flyteplugins-*`: which integrations have a v2 plugin, which don't and what to use instead, `task_config=` → `plugin_config=`, and wrapping connector tasks with `TaskEnvironment.from_task`. +- **`flyte-migrate-troubleshoot`** — migrated code fails on `flyte run`: an error index for relative imports, missing `depends_on`, unwrapped DataFrames, uncaught `RuntimeUserError`, files missing from the pod, `https://` `File` inputs, `NamedTuple` outputs, and shell tasks. Includes the compile → `--local` → remote debug loop. ## Hybrid v1 and v2 pipelines