Metadata-Version: 2.4
Name: mlserver
Version: 1.7.1+rhaiv.10
Summary: MLServer
License: Apache-2.0
License-File: LICENSE
Author: Seldon Technologies Ltd.
Author-email: hello@seldon.io
Requires-Python: >=3.10,<3.13
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: aiofiles
Requires-Dist: aiohttp (>=3.14.1,<4)
Requires-Dist: aiokafka
Requires-Dist: click
Requires-Dist: fastapi (>=0.134.0,<0.137.0)
Requires-Dist: gevent
Requires-Dist: geventhttpclient (>=2.1.1)
Requires-Dist: grpcio (>=1.67.1)
Requires-Dist: numpy
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc (>=1.22.0,<2.0.0)
Requires-Dist: opentelemetry-instrumentation-fastapi (>=0.43b0)
Requires-Dist: opentelemetry-instrumentation-grpc (>=0.43b0)
Requires-Dist: opentelemetry-sdk (>=1.22.0,<2.0.0)
Requires-Dist: orjson
Requires-Dist: pandas
Requires-Dist: protobuf (>=5.27.2,<7.0.0)
Requires-Dist: py-grpc-prometheus
Requires-Dist: pydantic (>=2.7.1,<3.0.0)
Requires-Dist: pydantic-settings (>=2.3.0,<3.0.0)
Requires-Dist: python-dotenv
Requires-Dist: python-multipart (>=0.0.27,<0.0.33)
Requires-Dist: starlette (>=1.0.1)
Requires-Dist: starlette-exporter
Requires-Dist: tritonclient[http] (>=2.42)
Requires-Dist: urllib3 (>=2.7.0)
Requires-Dist: uvicorn
Requires-Dist: uvloop ; sys_platform != "win32" and sys_platform != "cygwin" and platform_python_implementation != "PyPy"
Description-Content-Type: text/markdown

# MLServer

An open source inference server for your machine learning models.

[![video_play_icon](https://user-images.githubusercontent.com/10466106/151803854-75d17c32-541c-4eee-b589-d45b07ea486d.png)](https://www.youtube.com/watch?v=aZHe3z-8C_w)

## Overview

MLServer aims to provide an easy way to start serving your machine learning
models through a REST and gRPC interface, fully compliant with [KFServing's V2
Dataplane](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/v2-protocol.html)
spec. Watch a quick video introducing the project [here](https://www.youtube.com/watch?v=aZHe3z-8C_w).

- Multi-model serving, letting users run multiple models within the same
  process.
- Ability to run [inference in parallel for vertical
  scaling](https://mlserver.readthedocs.io/en/latest/user-guide/parallel-inference.html)
  across multiple models through a pool of inference workers.
- Support for [adaptive
  batching](https://mlserver.readthedocs.io/en/latest/user-guide/adaptive-batching.html),
  to group inference requests together on the fly.
- Scalability with deployment in Kubernetes native frameworks, including
  [Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/graph/protocols.html#v2-kfserving-protocol) and
  [KServe (formerly known as KFServing)](https://kserve.github.io/website/modelserving/v1beta1/sklearn/v2/), where
  MLServer is the core Python inference server used to serve machine learning
  models.
- Support for the standard [V2 Inference Protocol](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/v2-protocol.html) on
  both the gRPC and REST flavours, which has been standardised and adopted by
  various model serving frameworks.

You can read more about the goals of this project on the [initial design
document](https://docs.google.com/document/d/1C2uf4SaAtwLTlBCciOhvdiKQ2Eay4U72VxAD4bXe7iU/edit?usp=sharing).

## Usage

You can install the `mlserver` package running:

```bash
pip install mlserver
```

Note that to use any of the optional [inference runtimes](#inference-runtimes),
you'll need to install the relevant package.
For example, to serve a `scikit-learn` model, you would need to install the
`mlserver-sklearn` package:

```bash
pip install mlserver-sklearn
```

For further information on how to use MLServer, you can check any of the
[available examples](#examples).

## Inference Runtimes

Inference runtimes allow you to define how your model should be used within
MLServer.
You can think of them as the **backend glue** between MLServer and your machine
learning framework of choice.
You can read more about [inference runtimes in their documentation
page](./docs/runtimes/index.md).

Out of the box, MLServer comes with a set of pre-packaged runtimes which let
you interact with a subset of common frameworks.
This allows you to start serving models saved in these frameworks straight
away.
However, it's also possible to **[write custom
runtimes](./docs/runtimes/custom.md)**.

Out of the box, MLServer provides support for:

| Framework     | Supported | Documentation                                                    |
| ------------- | --------- | ---------------------------------------------------------------- |
| Scikit-Learn  | ✅        | [MLServer SKLearn](./runtimes/sklearn)                           |
| XGBoost       | ✅        | [MLServer XGBoost](./runtimes/xgboost)                           |
| Spark MLlib   | ✅        | [MLServer MLlib](./runtimes/mllib)                               |
| LightGBM      | ✅        | [MLServer LightGBM](./runtimes/lightgbm)                         |
| CatBoost      | ✅        | [MLServer CatBoost](./runtimes/catboost)                         |
| ONNX          | ✅        | [MLServer ONNX](./runtimes/onnx)                                 |
| Tempo         | ✅        | [`github.com/SeldonIO/tempo`](https://github.com/SeldonIO/tempo) |
| MLflow        | ✅        | [MLServer MLflow](./runtimes/mlflow)                             |
| Alibi-Detect  | ✅        | [MLServer Alibi Detect](./runtimes/alibi-detect)                 |
| Alibi-Explain | ✅        | [MLServer Alibi Explain](./runtimes/alibi-explain)               |
| HuggingFace   | ✅        | [MLServer HuggingFace](./runtimes/huggingface)                   |

### Runtime Security Maintainer Note

MLServer enforces a trusted runtime implementation allowlist in
`mlserver/settings.py` (`ALLOWED_MODEL_IMPLEMENTATIONS`) when resolving model
implementations from `model-settings.json` and
`MLSERVER_MODEL_IMPLEMENTATION`.

If you add a new core runtime implementation shipped by this repository, you
must also:

1. Add the runtime import path to `ALLOWED_MODEL_IMPLEMENTATIONS`.
2. Add or update tests validating allowlist behavior.
3. Keep runtime docs/examples aligned with the implementation import path.

For third-party or project-specific custom runtimes, do not extend the global
allowlist. Use the image-scoped runtime workflow with `mlserver build`:

**Building Images with Custom Runtimes:**

```bash
# PRODUCTION mode (production): Allowlist specific custom runtimes
mlserver build . -t my-image \
  --allow-runtime models.MyRuntime \
  --runtime-path models.py

# DEVELOPMENT mode (development): Allow any runtime
mlserver build . -t my-dev-image --dev
```

### Runtime Security Modes

MLServer operates in one of two security modes for loading custom runtimes:

**PRODUCTION Mode (Production):**
- Enforced when a trusted runtimes allowlist file exists in the image
- Only explicitly allowlisted runtimes can be loaded
- Custom runtimes must be baked into the image with `--allow-runtime` and `--runtime-path`
- Provides strong security guarantees for production deployments

**DEVELOPMENT Mode (Development):**
- Active when no allowlist file exists (e.g., running `mlserver start` directly)
- Supports dynamic loading of custom runtimes directly from model folders
- Simply place your custom runtime `.py` file next to `model-settings.json`
- Convenient for rapid local development and testing
- **WARNING:** Should NEVER be used in production - allows arbitrary code execution

You can query the current security mode through the `/v2/runtimes` REST endpoint
or `RuntimeSecurity` gRPC method. See the [model-settings reference](./docs/reference/model-settings.md#querying-runtime-security-configuration)
and [custom runtimes guide](./docs/user-guide/custom.md) for details.

MLServer is licensed under the Apache License, Version 2.0. However please note that software used in conjunction with, or alongside, MLServer may be licensed under different terms. For example, Alibi Detect and Alibi Explain are both licensed under the Business Source License 1.1. For more information about the legal terms of products that are used in conjunction with or alongside MLServer, please refer to their respective documentation.

## Supported Python Versions

🔴 Unsupported

🟠 Deprecated: To be removed in a future version

🟢 Supported

🔵 Untested

| Python Version | Status |
| -------------- | ------ |
| 3.7            | 🔴     |
| 3.8            | 🔴     |
| 3.9            | 🔴     |
| 3.10           | 🟢     |
| 3.11           | 🟢     |
| 3.12           | 🟢     |
| 3.13           | 🔴     |

## Examples

To see MLServer in action, check out [our full list of
examples](./docs/examples/index.md).
You can find below a few selected examples showcasing how you can leverage
MLServer to start serving your machine learning models.

- [Serving a `scikit-learn` model](./docs/examples/sklearn/README.md)
- [Serving a `xgboost` model](./docs/examples/xgboost/README.md)
- [Serving a `lightgbm` model](./docs/examples/lightgbm/README.md)
- [Serving a `catboost` model](./docs/examples/catboost/README.md)
- [Serving an `onnx` model](./docs/examples/onnx/README.md)
- [Serving a `tempo` pipeline](./docs/examples/tempo/README.md)
- [Serving a custom model](./docs/examples/custom/README.md)
- [Serving an `alibi-detect` model](./docs/examples/alibi-detect/README.md)
- [Serving a `HuggingFace` model](./docs/examples/huggingface/README.md)
- [Multi-Model Serving with multiple frameworks](./docs/examples/mms/README.md)
- [Loading / unloading models from a model repository](./docs/examples/model-repository/README.md)

## Developer Guide

### Versioning

Both the main `mlserver` package and the [inference runtimes
packages](./docs/runtimes/index.md) try to follow the same versioning schema.
To bump the version across all of them, you can use the
[`./hack/update-version.sh`](./hack/update-version.sh) script.

We generally keep the version as a placeholder for an upcoming version.

For example:

```bash
./hack/update-version.sh 0.2.0.dev1
```

### Testing

To run all of the tests for MLServer and the runtimes, use:

```bash
make test
```

To run run tests for a single file, use something like:

```bash
tox -e py3 -- tests/batch_processing/test_rest.py
```

