Tiny configuration for Triton Inference Server


Keywords
grpc, http, triton, tensorrt, inference, server, service, client, nvidia, rtzr, mlops, triton-inference-server, tritonclient
License
BSD-3-Clause
Install
pip install tritony==0.0.16

Documentation

tritony - Tiny configuration for Triton Inference Server

Pypi CI Coverage Status

What is this?

If you see the official example, it is really confusing to use where to start.

Use tritony! You will get really short lines of code like example below.

import argparse
import os
from glob import glob
import numpy as np
from PIL import Image

from tritony import InferenceClient


def preprocess(img, dtype=np.float32, h=224, w=224, scaling="INCEPTION"):
    sample_img = img.convert("RGB")

    resized_img = sample_img.resize((w, h), Image.Resampling.BILINEAR)
    resized = np.array(resized_img)
    if resized.ndim == 2:
        resized = resized[:, :, np.newaxis]

    scaled = (resized / 127.5) - 1
    ordered = np.transpose(scaled, (2, 0, 1))
    
    return ordered.astype(dtype)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--image_folder", type=str, help="Input folder.")
    FLAGS = parser.parse_args()

    client = InferenceClient.create_with("densenet_onnx", "0.0.0.0:8001", input_dims=3, protocol="grpc")
    client.output_kwargs = {"class_count": 1}

    image_data = []
    for filename in glob(os.path.join(FLAGS.image_folder, "*")):
        image_data.append(preprocess(Image.open(filename)))

    result = client(np.asarray(image_data))

    for output in result:
        max_value, arg_max, class_name = output[0].decode("utf-8").split(":")
        print(f"{max_value} ({arg_max}) = {class_name}")

Release Notes

  • 23.08.30 Support optional with model input, parameters on config.pbtxt
  • 23.06.16 Support tritonclient>=2.34.0
  • Loosely modified the requirements related to tritonclient

Key Features

  • Simple configuration. Only $host:$port and $model_name are required.
  • Generating asynchronous requests with asyncio.Queue
  • Simple Model switching
  • Support async tritonclient

Requirements

$ pip install tritonclient[all]

Install

$ pip install tritony

Test

With Triton

./bin/run_triton_tritony_sample.sh
pytest -s --cov-report term-missing --cov=tritony tests/

Example with image_client.py

# Download Images from https://github.com/triton-inference-server/server.git
python ./example/image_client.py --image_folder "./server/qa/images"