使用样例

限制与约束

仅Atlas 800I A2 推理产品支持此特性。
LoRA权重个数上限受硬件显存限制，建议数量为小于等于10个。
不支持LoRA权重热加载。
仅Qwen1.5-14B、Qwen1.5-72B、LLaMa3.1-70B和Qwen2-72B支持Multi-LoRA特性。

操作步骤

本章节以LLaMa3.1 70B模型为例，简单介绍Multi-Lora如何使用。

下载基础模型和Lora权重后，在基础模型路径下新增lora_adapter.json文件配置预加载的Lora权重。

以使用两个lora权重为例：

cd ${基础模型权重}
vi lora_adapter.json
# 在此文件中配置adapter名称及路径，示例：
{"adapter1": "${Lora权重1路径}", "adapter2": "${Lora权重2路径}"}

配置服务化参数并启动服务化，服务化参数说明请参见配置参数说明章节。
```
cd ${mindie-service安装路径}
vi conf/config.json
./bin/mindieservice_daemon
```

使用以下指令发送请求。

其中"model"参数可以设置为模型权重路径或lora_adapter.json中配置的adapter名称。当"model"参数为模型权重路径时，不使用Lora权重进行推理。当"model"参数为lora_adapter.json中配置的adapter名称时，启用基础模型权重和指定的Lora权重进行推理。

curl https://127.0.0.1:1025/generate \
-H "Content-Type: application/json" \
--cacert ca.pem --cert client.pem  --key client.key.pem \
-X POST \
-d '{
"model": "${基础模型权重}",
"prompt": "Taxation in Puerto Rico -- The Commonwealth government has its own tax laws and Puerto Ricans are also required to pay some US federal taxes, although most residents do not have to pay the federal personal income tax. In 2009, Puerto Rico paid $3.742 billion into the US Treasury. Residents of Puerto Rico pay into Social Security, and are thus eligible for Social Security benefits upon retirement. However, they are excluded from the Supplemental Security Income.\nQuestion: is federal income tax the same as social security?\nAnswer:",
"max_tokens": 20,
"temperature": 0
}'

curl https://127.0.0.1:1025/generate \
-H "Content-Type: application/json" \
--cacert ca.pem --cert client.pem  --key client.key.pem \
-X POST \
-d '{
"model": "adapter1",
"prompt": "Taxation in Puerto Rico -- The Commonwealth government has its own tax laws and Puerto Ricans are also required to pay some US federal taxes, although most residents do not have to pay the federal personal income tax. In 2009, Puerto Rico paid $3.742 billion into the US Treasury. Residents of Puerto Rico pay into Social Security, and are thus eligible for Social Security benefits upon retirement. However, they are excluded from the Supplemental Security Income.\nQuestion: is federal income tax the same as social security?\nAnswer:",
"max_tokens": 20,
"temperature": 0
}'

父主题： Multi-Lora