flyte-migrate: add plugins and troubleshoot skills, fix migration guidance from flytesnacks runs

README.md

@@ -6,7 +6,7 @@ A plugin marketplace for working with [Flyte](https://flyte.org) — in
 [Google Antigravity](https://antigravity.google), or any agent harness that supports
 [Agent Skills](https://agentskills.io).
 
-21 skills, plus two **MCP servers** that let Claude search Flyte docs and act on your own
+25 skills, plus two **MCP servers** that let Claude search Flyte docs and act on your own
 cluster. `uvx flyte-skills install` gets you the skills in any harness;
 `uvx flyte-agent-plugins mcp install` adds the servers, and installing the `flyte` plugin
 gets you both at once — see [Bundled MCP servers](#bundled-mcp-servers).
@@ -19,7 +19,7 @@ This guide assumes that you have [uv installed](https://docs.astral.sh/uv/gettin
 uvx flyte-skills install
 ```
 
-That copies all 21 skills into whichever harness directories it finds on your machine —
+That copies all 25 skills into whichever harness directories it finds on your machine —
 no arguments needed. (`pip install flyte-agent-plugins` then `flyte-agent-plugins install` works the
 same way.)
 
@@ -80,15 +80,15 @@ convention shared across agent CLIs, so any harness honouring it picks the skill
 without a harness-specific flag. `--dir <path>` writes them anywhere else you like — both
 work the same with `--project`, `--dry-run`, and `uninstall`.
 
-**Two packages.** `flyte-skills` and `flyte-agent-plugins` carry the same 21 skills, so
+**Two packages.** `flyte-skills` and `flyte-agent-plugins` carry the same 25 skills, so
 every `install` command above works under either name. They differ in one thing: only
 `flyte-agent-plugins` ships the `mcp` subcommand, so reach for it when you want the MCP
 servers too. To pin a version,
 `uvx --from flyte-skills==<version> flyte-skills install`.
 
 ## Install as a plugin (Claude Code, Codex, and Antigravity)
 
-Installing the plugin instead of the skills gets you the same 21 skills **plus** both MCP
+Installing the plugin instead of the skills gets you the same 25 skills **plus** both MCP
 servers, which are declared in `plugins/flyte/.mcp.json`.
 
 ### Claude Code
@@ -156,20 +156,20 @@ The skills are plain [Agent Skills](https://agentskills.io) (`SKILL.md` + YAML
 frontmatter), so they work in any harness that supports the standard. `flyte-agent-plugins
 install` above covers all of them; each harness also has its own installer, which is what you
 want when you would rather track the repo than a release, or to install a single skill
-rather than all 21.
+rather than all 25.
 
 | Harness | Skills | MCP servers | `flyte-agent-plugins install` |
 |---|---|---|---|
-| Claude Code | all 21 | both, automatically | `--target claude` |
-| Codex CLI | all 21 | both, automatically | `--target agents` |
-| Antigravity | all 21 | both, via the plugin | `--target agents --project` |
+| Claude Code | all 25 | both, automatically | `--target claude` |
+| Codex CLI | all 25 | both, automatically | `--target agents` |
+| Antigravity | all 25 | both, via the plugin | `--target agents --project` |
 | Hermes | per-skill | none — add manually | `--target hermes` |
-| opencode | all 21 | none — add manually | `--target opencode` |
-| pi | all 21 | none — add manually | `--target pi` |
+| opencode | all 25 | none — add manually | `--target opencode` |
+| pi | all 25 | none — add manually | `--target pi` |
 
 ### Hermes
 
-`flyte-agent-plugins install --target hermes` writes all 21 into `~/.hermes/skills/`, which Hermes
+`flyte-agent-plugins install --target hermes` writes all 25 into `~/.hermes/skills/`, which Hermes
 auto-discovers with no registration step. Add `--project` for `<repo>/.hermes/skills/` —
 project skills need `hermes skills trust` before first use.
 
@@ -253,6 +253,8 @@ Convert existing Flyte 1 (`flytekit`) code to Flyte 2. Distilled from the offici
 | [`flyte-migrate-data-io`](plugins/flyte/skills/flyte-migrate-data-io) | Migrate data types & I/O: `FlyteFile`/`FlyteDirectory`→`flyte.io.File`/`Dir`, `StructuredDataset`→`flyte.io.DataFrame`, dataclasses/Pydantic as task I/O. |
 | [`flyte-migrate-ml`](plugins/flyte/skills/flyte-migrate-ml) | Migrate ML workloads (training, HPO, GPU/deep learning, batch inference, end-to-end pipelines) and the new-in-v2 patterns (real-time serving, apps, sandboxed execution) they unlock. |
 | [`flyte-migrate-slurm`](plugins/flyte/skills/flyte-migrate-slurm) | Migrate Slurm (`sbatch`/`srun`) workloads: `#SBATCH` pragmas → `TaskEnvironment` config, job arrays → `flyte.map`, `--dependency` chains → plain Python, multi-node `srun` → `ClusteredTaskEnvironment`, `--requeue` → retries/checkpoints/spot. |
+| [`flyte-migrate-plugins`](plugins/flyte/skills/flyte-migrate-plugins) | Map `flytekitplugins-*` to `flyteplugins-*` (Spark, Ray, Dask, PyTorch Elastic, BigQuery, Snowflake, Databricks, dbt, papermill, wandb, mlflow, pandera, Slurm), list the integrations with no v2 plugin and their replacements, and the API changes that break a literal port. |
+| [`flyte-migrate-troubleshoot`](plugins/flyte/skills/flyte-migrate-troubleshoot) | Fix errors in freshly migrated code, indexed by error text: relative imports, missing `depends_on`, unwrapped DataFrames, `RuntimeUserError` instead of the original exception, notebooks missing from the pod, `https://` `File` inputs, `NamedTuple` outputs, and shell tasks that exit 1 with an empty message. Includes a compile → `--local` → remote debug loop. |
 
 Example:
 

evals/scenarios/flyte-migrate-plugins/static.yaml

@@ -0,0 +1,5 @@
+id: flyte-migrate-plugins-static
+skill: flyte-migrate-plugins
+tier: static
+# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets).
+# No agent, no LLM. Runs on every change.

evals/scenarios/flyte-migrate-troubleshoot/static.yaml

@@ -0,0 +1,5 @@
+id: flyte-migrate-troubleshoot-static
+skill: flyte-migrate-troubleshoot
+tier: static
+# Static tier lints the SKILL.md itself (frontmatter, code fences, no secrets).
+# No agent, no LLM. Runs on every change.

packaging/README.md

@@ -4,10 +4,10 @@ The skills in `plugins/flyte/` are published from this one source of truth:
 
 | Registry | Package | Skills | `mcp` subcommand | Status |
 |---|---|---|---|---|
-| PyPI | [`flyte-agent-plugins`](https://pypi.org/project/flyte-agent-plugins/) | all 21 | yes | published |
-| PyPI | [`flyte-skills`](https://pypi.org/project/flyte-skills/) | all 21 | no | published |
-| npm | `flyte-agent-plugins` | all 21 | yes | **built, not published** |
-| npm | `flyte-skills` | all 21 | no | **built, not published** |
+| PyPI | [`flyte-agent-plugins`](https://pypi.org/project/flyte-agent-plugins/) | all 25 | yes | published |
+| PyPI | [`flyte-skills`](https://pypi.org/project/flyte-skills/) | all 25 | no | published |
+| npm | `flyte-agent-plugins` | all 25 | yes | **built, not published** |
+| npm | `flyte-skills` | all 25 | no | **built, not published** |
 
 **npm publishing is currently disabled** while distribution focuses on pip/uvx.
 The npm packages are still generated, packed, and validated on every CI run, so

plugins/flyte/README.md

@@ -64,6 +64,11 @@ Convert existing Flyte 1 (`flytekit`) code to Flyte 2, distilled from the offici
 - **`flyte-migrate-slurm`** — Slurm (`sbatch`/`srun`) → Flyte 2: `#SBATCH` pragmas →
   `TaskEnvironment` config, job arrays → `flyte.map`, `--dependency` chains → plain Python,
   multi-node `srun` → `ClusteredTaskEnvironment`, `--requeue` → retries/checkpoints/spot.
+- **`flyte-migrate-plugins`** — `flytekitplugins-*` → `flyteplugins-*`, integrations with no
+  v2 plugin and their replacements, `task_config=` → `plugin_config=`, and connector tasks.
+- **`flyte-migrate-troubleshoot`** — errors in freshly migrated code, indexed by error text:
+  relative imports, missing `depends_on`, unwrapped DataFrames, `RuntimeUserError`, files
+  missing from the pod, `https://` `File` inputs, `NamedTuple` outputs, and shell tasks.
 
 ## Bundled MCP servers
 

plugins/flyte/skills/flyte-migrate-config/SKILL.md

@@ -99,7 +99,7 @@ image = (
 | Constructor | Use case |
 |---|---|
 | `Image.from_debian_base()` | Most common; includes the Flyte SDK |
-| `Image.from_base(image_uri)` | Start from any existing image |
+| `Image.from_base(image_uri)` | Start from any existing image (call `.clone(name=..., extendable=True)` before any `.with_*`) |
 | `Image.from_dockerfile(path)` | Complex custom builds |
 | `Image.from_uv_script(path)` | UV-based projects |
 
@@ -116,10 +116,21 @@ Common chainable builder methods: `.with_pip_packages(...)`, `.with_requirements
 | `env` | `.with_env_vars()` | Method instead of param |
 | `commands` | `.with_commands()` | Method instead of param |
 | `copy` / `source_root` | `.with_source_file()` / `.with_source_folder()` | More explicit methods |
-| `base_image` | `Image.from_base()` | Different constructor |
+| `base_image` | `Image.from_base()` + `.clone(extendable=True)` | Different constructor; see below |
 | `builder` | Config file or `flyte.init()` | Global setting |
 | `platform` | `platform` (constructor) | Tuple: `("linux/amd64", "linux/arm64")` |
 
+An image from `Image.from_base()` is not extendable, so chaining `.with_*` onto it fails when the module loads with `ValueError: Cannot add additional layers to a non-extendable image`. Clone it as extendable first:
+
+```python
+# Flyte 1: ImageSpec(base_image="ghcr.io/flyteorg/rawcontainers-python:v2", packages=["pandas"])
+image = (
+    Image.from_base("ghcr.io/flyteorg/rawcontainers-python:v2")
+    .clone(name="rawcontainers-python-pandas", extendable=True)
+    .with_pip_packages("pandas")
+)
+```
+
 For a private registry, create an image-pull secret and reference it:
 
 ```bash
@@ -169,8 +180,8 @@ Caching is enabled at the env level with `cache="auto"` (or per-task on `@env.ta
 
 | Behavior | Description |
 |---|---|
-| `"auto"` | Cache results and reuse if available |
-| `"override"` | Always execute and overwrite the cache |
+| `"auto"` | Cache results; the version is computed from the function body, so any code change invalidates the cache |
+| `"override"` | Cache results under the explicit `version_override` string; the cache is invalidated only when you change it (the closest match to `cache_version`) |
 | `"disable"` | No caching (default for a `TaskEnvironment`) |
 
 ```python
@@ -182,7 +193,7 @@ def cached_task(x: int) -> int:
 
 # Advanced control (replaces cache_version, serialize, ignored_inputs, ...)
 @env.task(cache=flyte.Cache(
-    behavior="auto",
+    behavior="override",        # the SDK documents version_override for "override" only
     version_override="v1.0",
     serialize=True,
     ignored_inputs=("debug",),
@@ -308,7 +319,7 @@ def main(trigger_time: datetime = datetime(2024, 1, 1, 2, 0)) -> str:
     return f"retrained model at {trigger_time.isoformat()}"
 ```
 
-Triggers support `flyte.Cron("0 9 * * *", timezone="America/New_York")` and `flyte.FixedRate(timedelta(hours=1))` as automations, plus convenience constructors like `flyte.Trigger.hourly()` and `flyte.Trigger.daily()`.
+Triggers support `flyte.Cron("0 9 * * *", timezone="America/New_York")` and `flyte.FixedRate(60)` (an integer number of minutes, not a `timedelta`; a `timedelta` passes `flyte run` but fails at `flyte deploy`) as automations, plus convenience constructors like `flyte.Trigger.hourly()` and `flyte.Trigger.daily()`.
 
 ## CLI command mapping: pyflyte to flyte
 
@@ -444,7 +455,7 @@ For API-key authentication in non-interactive environments, use `flyte.init_from
 2. **Don't use `mem`, `ephemeral_storage`, or separate `requests`/`limits`** — use `memory`, `disk`, and a single `flyte.Resources` value that serves as both.
 3. **Don't pass GPUs with `gpu="1"` plus `accelerator=`** — combine type and count into one `Type:count` string like `gpu="A100:2"`.
 4. **Don't rebuild `ImageSpec`'s many constructor args** — start from a base (`Image.from_debian_base()`) and chain `.with_*` builder methods.
-5. **Don't keep `cache_version="1.0"`** — use `cache="auto"` for the common case, or `flyte.Cache(version_override=...)` for advanced control.
+5. **Don't keep `cache_version="1.0"`** — use `cache="auto"` for the common case, or `flyte.Cache(behavior="override", version_override=...)` to keep a manually bumped version.
 6. **Don't read secrets via `current_context().secrets.get(...)`** — declare them on the `TaskEnvironment` and read the injected environment variable with `os.environ` / `os.getenv`.
 7. **Don't recreate `LaunchPlan` + `CronSchedule`** — use `flyte.Trigger` with `flyte.Cron` attached to the task, and deploy it with `flyte deploy`.
 8. **Don't run `pyflyte ... --remote`** — `flyte run` is remote by default; add `--local` explicitly for in-process runs.

plugins/flyte/skills/flyte-migrate-control-flow/SKILL.md

@@ -166,12 +166,18 @@ async def train_fold(max_depth: int) -> float:
 async def main(max_depth: int) -> float:
     try:
         return await train_fold(max_depth)
-    except ValueError as e:
+    except flyte.errors.RuntimeUserError as e:
+        # A failed sub-task does not re-raise its original exception type.
+        # It arrives as RuntimeUserError, with the original class name in e.code.
+        if e.code != "ValueError":
+            raise
         print(f"invalid hyperparameter ({e}); falling back to a safe default")
         # Recover with a safe default instead of failing the whole run.
         return await train_fold(max_depth=6)
 ```
 
+An `except ValueError:` around `await train_fold(...)` never matches, either on a cluster or with `flyte run --local`: the parent receives `flyte.errors.RuntimeUserError`, and the original exception's class name is in its `code`. Check `e.code` as above, or catch `Exception` when any failure should be handled (the usual replacement for `on_failure`).
+
 Flyte 2 also exposes typed errors, so you can catch a specific failure and retry with more resources — a common need for memory-hungry training jobs:
 
 ```python
@@ -234,6 +240,16 @@ def main(n: int, model_version: int) -> list[int]:
     return list(flyte.map(bound, range(n), concurrency=10))
 ```
 
+`flyte.map` accepts a `partial` only if it leaves **exactly one** parameter unbound. A Flyte 1 `map_task(partial(...))` that mapped over several list inputs at once fails with `Partial function must leave exactly one parameter unspecified for mapping. Found 2 unspecified parameters`. Pass one iterable per parameter instead, positionally, zipped like the built-in `map`:
+
+```python
+# Flyte 1: map_task(multi_input_task)(quantity=list_q, price=list_p, shipping=list_s)
+@env.task
+def main(list_q: list[int], list_p: list[float], shipping: float) -> list[float]:
+    # One iterable per parameter, in parameter order; repeat a constant to the same length.
+    return list(flyte.map(multi_input_task, list_q, list_p, [shipping] * len(list_q)))
+```
+
 ### Flyte 2 (asyncio.gather)
 
 ```python

plugins/flyte/skills/flyte-migrate-data-io/SKILL.md

@@ -27,10 +27,36 @@ Flyte 2 renames the offloaded-data types and makes their I/O `async`, but the me
 | `flytekit.types.structured.StructuredDataset` | `flyte.io.DataFrame` | build with `from_df`, read with `open(...).all()` |
 | `@dataclass_json` + `@dataclass` | plain `@dataclass` | no mixin needed |
 | Pydantic `BaseModel` (+ config) | plain Pydantic `BaseModel` | works directly as task I/O |
+| `np.ndarray` (offloaded as `.npy`), and other types with no transformer | No transformer; falls back to pickle | Pass large values as a `File` (see below) |
+
+### Values with no type transformer, such as `np.ndarray`
+
+Flyte 1 had a numpy transformer that offloaded arrays as `.npy` files. Flyte 2 has none, so an `np.ndarray`, like any type without a transformer, falls back to pickle. The pickle fallback decides whether to store the value inline or offload it by `sys.getsizeof`, and stores it inline below 10 KB. A sliced view, such as the halves of a train/test split, reports about 160 bytes however large the data, so it is stored inline. A large dataset passed this way fails with `Inputs exceed max_bytes limit of 10.0 MB, actual size: 23.4 MB`.
+
+Pass large arrays as a `File` instead:
+
+```python
+import numpy as np
+from flyte.io import File
+
+@env.task
+async def load_data() -> File:
+    x_train, y_train = ...
+    np.savez("/tmp/data.npz", x_train=x_train, y_train=y_train)
+    return await File.from_local("/tmp/data.npz")
+
+@env.task
+async def train(data: File) -> float:
+    arrays = np.load(await data.download())
+    x_train, y_train = arrays["x_train"], arrays["y_train"]
+    ...
+```
+
+A pickled value also needs its library in the image of every task that receives it, including the calling task (see `flyte-migrate-tasks-workflows`).
 
 ## Offloaded Data: The Mental Model
 
-`File`, `Dir`, and `DataFrame` are lightweight references (pointers) to data offloaded in blob storage — not the materialized bytes. In Flyte 2 the read/write operations are `async`: upload with `await File.from_local(local_path)`, read with `async with f.open("rb") as fh: await fh.read()`, build a frame with `flyte.io.DataFrame.from_df(df)` (sync constructor), and read it with `await fdf.open(pandas.DataFrame).all()`.
+`File`, `Dir`, and `DataFrame` are lightweight references (pointers) to data offloaded in blob storage — not the materialized bytes. In Flyte 2 the read/write operations are `async`: upload with `await File.from_local(local_path)`, read with `async with f.open("rb") as fh: bytes(await fh.read())`, build a frame with `flyte.io.DataFrame.from_df(df)` (sync constructor), and read it with `await fdf.open(pandas.DataFrame).all()`.
 
 ## Files and Directories
 
@@ -81,7 +107,8 @@ async def write_file(content: str) -> File:
 @env.task
 async def read_file(f: File) -> str:
     async with f.open("rb") as fh:
-        return (await fh.read()).decode("utf-8")
+        # read() returns an obstore Bytes buffer, which has no .decode(); convert to bytes first.
+        return bytes(await fh.read()).decode("utf-8")
 
 @env.task
 async def main(content: str) -> str:
@@ -145,7 +172,20 @@ async def main() -> float:
     return await total_payroll(await make_df())
 ```
 
-Add the dataframe dependencies (for example `pandas` and `pyarrow`) to the `TaskEnvironment` image. See [DataFrames](https://www.union.ai/docs/v2/flyte/user-guide/task-programming/dataframes) for more.
+Flyte 1 converted a `pd.DataFrame` to a `StructuredDataset` implicitly when it flowed between tasks. Flyte 2 does not: inside a driver task, a sub-task that returns `pd.DataFrame` hands back a real pandas object, and passing it to a sub-task typed `flyte.io.DataFrame` fails with `Type conversion failed ... Did you forget to wrap your dataframe in a DataFrame instance?`. Wrap it at the call site:
+
+```python
+@env.task
+async def get_df() -> pd.DataFrame:
+    return pd.DataFrame({"salary": [50000, 60000]})
+
+@env.task
+async def main() -> float:
+    pandas_df = await get_df()
+    return await total_payroll(flyte.io.DataFrame.from_df(pandas_df))
+```
+
+Add the dataframe dependencies (for example `pandas` and `pyarrow`) to the `TaskEnvironment` image. `pyarrow` is needed whenever a task returns a `pd.DataFrame` or a `flyte.io.DataFrame`, because both are written as Parquet; without it the task fails while encoding its output. See [DataFrames](https://www.union.ai/docs/v2/flyte/user-guide/task-programming/dataframes) for more.
 
 ## Dataclasses and Structured Types
 
@@ -307,3 +347,4 @@ async def main() -> int:
 4. **Don't materialize large data into task outputs** — return `File`, `Dir`, or `DataFrame` references, not the raw bytes or full frames.
 5. **Don't forget the dataframe dependencies** — add `pandas` and `pyarrow` (or your engine) to the `TaskEnvironment` image so DataFrame I/O works remotely.
 6. **Don't import from `flytekit.types.*`** — import `File` and `Dir` from `flyte.io`, and use `flyte.io.DataFrame`.
+7. **Don't pass large `np.ndarray`s between tasks** — there is no numpy transformer; pickled arrays (especially sliced views) can be stored inline and break the 10 MB limit. Save to `.npz` and pass a `File`.