MindIE原生文本/流式推理接口

接口功能

提供文本/流式推理处理功能。

接口格式

操作类型：POST

URL：https://{ip}:{port}/infer

• {ip}优先取启动命令参数中的{predict_ip}；如果没有配置该命令行参数，则取配置文件ms_coordinator.json的“predict_ip”参数。
• {port}优先取启动命令参数中的{predict_port}；如果没有配置该命令行参数，则取配置文件ms_coordinator.json的“predict_port”参数。
• 当ms_coordinator.json的"algorithm_type"参数配置为"cache_affinity"时，该接口不支持使用。

请求参数

使用样例

请求样例：

POST https://{ip}:{port}/infer

请求消息体：

• 单模态文本模型：

  {
      "inputs": "My name is Olivier and I",
      "stream": false,
      "parameters": {
          "temperature": 0.5,
          "top_k": 10,
          "top_p": 0.95,
          "max_new_tokens": 20,
          "do_sample": true,
          "seed": null,
          "repetition_penalty": 1.03,
          "details": true,
          "typical_p": 0.5,
          "watermark": false,
          "priority": 5,
          "timeout": 10
      }
  }

• 多模态模型：
  

  "image_url"参数的取值请根据实际情况进行修改。

  {
      "inputs": [
          {"type": "text", "text": "My name is Olivier and I"},
          {
              "type": "image_url",
              "image_url": "/xxxx/test.png"
          }
      ],
      "stream": false,
      "parameters": {
          "temperature": 0.5,
          "top_k": 10,
          "top_p": 0.95,
          "max_new_tokens": 20,
          "do_sample": true,
          "seed": null,
          "repetition_penalty": 1.03,
          "details": true,
          "typical_p": 0.5,
          "watermark": false,
          "priority": 5,
          "timeout": 10
      }
  }

响应样例：

• 文本推理（“stream”=false）：

  {
      "generated_text": "am a french native speaker. I am looking for a job in the hospitality industry. I",
      "details": {
          "finish_reason": "length",
          "generated_tokens": 20,
          "seed": 846930886
      }
  }

• 流式推理：
  • 流式推理1（“stream”=true，使用sse格式返回）：

    data: {"prefill_time":45.54,"decode_time":null,"token":{"id":626,"text":"am"}}
    
    data: {"prefill_time":null,"decode_time":128.32,"token":{"id":263,"text":" a"}}
    
    data: {"prefill_time":null,"decode_time":18.17,"token":{"id":5176,"text":" French"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":17739,"text":" photograph"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":261,"text":"er"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":2729,"text":" based"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":297,"text":" in"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":3681,"text":" Paris"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":29889,"text":"."}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":13,"text":"\n"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":29902,"text":"I"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":505,"text":" have"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":1063,"text":" been"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":27904,"text":" shooting"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":1951,"text":" since"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":306,"text":" I"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":471,"text":" was"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":29871,"text":" "}}
    
    data: {"prefill_time":null,"decode_time":16.80,"token":{"id":29896,"text":"1"}}
    
    data: {"prefill_time":null,"decode_time":16.80,"generated_text":"am a French photographer based in Paris.\nI have been shooting since I was 15","details":{"finish_reason":"length","generated_tokens":20,"seed":846930886},"token":{"id":29945,"text":null}}
    

  • 流式推理2（“stream”=true，配置项“fullTextEnabled”=true，使用sse格式返回）：

    data: {"prefill_time":33.55400085449219,"decode_time":null,"token":{"id":5122,"text":"："}}
    
    data: {"prefill_time":null,"decode_time":19.922000885009766,"token":{"id":7552,"text":"：）"}}
    
    data: {"prefill_time":null,"decode_time":12.803999900817871,"token":{"id":40,"text":"：）I"}}
    
    data: {"prefill_time":null,"decode_time":11.869000434875488,"token":{"id":2776,"text":"：）I'm"}}
    
    data: {"prefill_time":null,"decode_time":12.008000373840332,"generated_text":"：）I'm ","details":null,"token":{"id":4102,"text":null}}

输出说明