onnxruntime

[akhilchhetri]

I tried to install onnxruntime but its not resolved, So i wanted to be clear either.

1. onnxruntime package has the python wheels for android / ios?
2. If not, what are the steps needed to create a binary build for onnxruntime for android arm64-v8a / armeabi64-v7a?

[ap4499]

At my last look at Onnx on embedded devices/Android/iOS, you needed to compile binaries from source(ie. they haven't got arm64-v8a binaries).

I would build that wheel (C++), create the Python bindings, and then have it contained within your package.

I think the key question from a Flet POV would be whether you could place something like this within your reqs.txt / .toml requirements when working on an Android build:
CustomOnnxWheel @ file:///Users/USER/project_folder/GitHub/CustomOnnxWheel.whl

I would guess that you can - as you can on MacOS/Windows, and I use custom wheels/binaries. But I haven't done Flet -> Android.

Answering your question, I think the most relevant link to start this journey is this:
https://onnxruntime.ai/docs/tutorials/mobile/

[ap4499]

Although.. there might be an easier way of doing it if someone has already set this up for Flutter.

maybe this:
https://pub.dev/documentation/onnxruntime/latest/
https://pub.dev/packages/onnxruntime

Then you can use the extension mechanism that Flet introduced in 0.27: https://flet.dev/docs/extend/user-extensions/

[SamYuan1990]

I tested it today, https://github.com/SamYuan1990/i18n-agent-action/actions/runs/17670451604/job/50220801655?pr=104 related with microsoft/onnxruntime#26025 I am not sure if onnx runtime can provide a package.

[SamYuan1990]

hi @akhilchhetri , do you have any exp for me and others to use https://flet.dev/docs/extend/user-extensions/ as @ap4499 mentioned?

[SamYuan1990]

well, for me I am attempting sherpa_onnx for STT(sound to text). hope it help, https://github.com/SamYuan1990/flet_sherpa_onnx

[ndonkoHenri]

onnxruntime 1.27.0 is now supported on Android and iOS: https://pypi.flet.dev/onnxruntime

It's a CPU-only build — the CPU execution provider (MLAS, NEON-optimized). The NNAPI (Android) and CoreML (iOS) accelerator backends are off in this first release, which keeps Android and iOS running the exact same dependable baseline. Everything runs offline; the only network you need is a one-time model download.

Install

Add it to your deps:

# pyproject.toml
dependencies = [
    "flet",
    "onnxruntime",
    "numpy",
]

Note this is 64-bit only (arm64-v8a / x86_64) — wheels for armeabi-v7a arch/devices aren't built (at least, for now). To prevent your Android builds from failing, you will have to explicitly specify the target archs:

[tool.flet.android]
target_arch = ["arm64-v8a", "x86_64"]

# or 
flet build apk --arch arm64-v8a x86_64

Storage & the model file (the important bit)

.onnx models can be large, and app stores cap bundle size — so don't ship a big model inside the app. Download it once on first run into Flet's persistent storage (FLET_APP_STORAGE_DATA) and load it from there:

import os, urllib.request

data_dir = os.getenv("FLET_APP_STORAGE_DATA")
model_path = os.path.join(data_dir, "model.onnx")
if not os.path.exists(model_path):
    urllib.request.urlretrieve("https://<your-host>/model.onnx", model_path)

A genuinely small model (a few hundred KB) could be shipped with the app (for example, by storing it in the assets/ directory) and be read from the bundle. If you use a higher-level library on top of onnxruntime (e.g. rapidocr, fastembed), point its model cache at storage before importing it — e.g. os.environ["HF_HOME"] = os.getenv("FLET_APP_STORAGE_DATA") — so its auto-downloaded models land in writable, persistent space.

Example

import os
import numpy as np
import onnxruntime as ort

model_path = os.path.join(os.getenv("FLET_APP_STORAGE_DATA"), "model.onnx")

# Set intra_op_num_threads to your device's performance-core count (often 4)
# rather than leaving it unbounded — that's where mobile latency comes from.
opts = ort.SessionOptions()
opts.intra_op_num_threads = 4

sess = ort.InferenceSession(model_path, opts, providers=["CPUExecutionProvider"])
inp = sess.get_inputs()[0].name
x = np.asarray(data, dtype=np.float32)          # shape/dtype your model expects
(out,) = sess.run(None, {inp: x})

Run it off the UI thread

Loading a session and running inference are CPU-bound and can take a moment — do them in page.run_thread(...) so the UI stays responsive.

Notes for good results on mobile

• Only CPUExecutionProvider is available: Check with ort.get_available_providers(); requesting a provider that isn't there just falls back to CPU.
• Set SessionOptions().intra_op_num_threads to your device's performance-core count (often 4) rather than leaving it unbounded — that's where mobile latency actually comes from.
• Prefer quantized / small models (INT8, or small FP32 graphs): Classifiers, embedding models, small vision and speech models run well; multi-hundred-MB networks will load but be slow and memory-heavy. For LLMs specifically, GGUF via llama-cpp-python is the better fit.

Let me know if you face any issues!