question: How to use replica in BentoML? #4887
Unanswered
Dawnfz-Lenfeng
asked this question in
Q&A
Replies: 1 comment
|
No Ray engine needed — replicas are a deployment concern, not a service-code one, and BentoML doesn't use Ray for this. Two different knobs, don't mix them:
For your BentoVLLM service, I verified this against current main; your question is from the July 2024-era API, but the workers-vs-replicas split is the same. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I am using the BentoVLLM. How do I configure the service file to use the
replicafeature?Do I need to use the Ray engine for this? I'm not very familiar with these configurations.
All reactions