Profile API Server

Profile API Server#

Before benchmarking the api_server, please install the lmdeploy precompiled package and download the script and the test dataset:

pip install lmdeploy
git clone --depth=1 https://github.com/InternLM/lmdeploy
cd lmdeploy/benchmark

Launch the server first (you may refer here for guide) and run the following command:

python3 benchmark/profile_restful_api.py --backend lmdeploy --num-prompts 5000 --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json

For detailed argument specification of profile_restful_api.py, please run the help command python3 benchmark/profile_restful_api.py -h.