GitHub
InternLM
HomePage
GitHub
Twitter

Table of Contents

v0.4.1

Get Started

Get Started

Build

Build from source

Benchmark

Profile Token Latency and Throughput
Profile Request Throughput
Profile API Server
Profile Triton Inference Server
Evaluate LLMs with OpenCompass

Supported Models

Supported Models

Inference

LLM Offline Inference Pipeline
VLM Offline Inference Pipeline

serving

Serving LLM with OpenAI Compatible Server
Serving VLM with OpenAI Compatible Server
Serving with Gradio
Request Distributor Server

Quantization

W4A16 Quantization
Key-Value(KV) Cache Quantization
W8A8 LLM Model Deployment

Advanced Guide

Architecture of TurboMind
Architecture of lmdeploy.pytorch
How to support new model in lmdeploy.pytorch
Context length extrapolation
Customized chat template
How to debug Turbomind
LMDeploy-QoS Introduce and Usage

API Reference

inference pipeline

Docs >
Overview: module code

Shortcuts

All modules for which code is available

lmdeploy.api
lmdeploy.messages
lmdeploy.model

© Copyright 2021-2024, OpenMMLab. Revision 14e99539.

Built with Sphinx using a theme provided by Read the Docs.

GitHub