流式推理接口

接口功能

提供流式推理处理功能。

接口格式

操作类型：POST

URL：https://{ip}:{port}/v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/generate_stream

请求参数

使用样例

请求样例：

POST https://{ip}:{port}/v2/models/llama_65b/generate_stream

请求消息体：

• 单模态文本模型：

  {
      "id":"a123",
      "text_input": "My name is Olivier and I",
      "parameters": {
          "details": true,
          "do_sample": true,
          "max_new_tokens":5,
          "repetition_penalty": 1.1,
          "seed": 123,
          "temperature": 1,
          "top_k": 10,
          "top_p": 0.99,
          "batch_size":100,
          "typical_p": 0.5,
          "watermark": false,
          "perf_stat": true,
          "priority": 5,
          "timeout": 10
      }
  }

• 多模态模型：
  

  "image_url"参数的取值请根据实际情况进行修改。

  {
      "id":"a123",
      "text_input": [
          {"type": "text", "text": "My name is Olivier and I"},
          {
              "type": "image_url",
              "image_url": "/xxxx/test.png"
          }
      ],
      "parameters": {
          "details": true,
          "do_sample": true,
          "max_new_tokens":20,
          "repetition_penalty": 1.1,
          "seed": 123,
          "temperature": 1,
          "top_k": 10,
          "top_p": 0.99,
          "batch_size":100,
          "typical_p": 0.5,
          "watermark": false,
          "perf_stat": false,
          "priority": 5,
          "timeout": 10
      }
  }

响应样例：

• 响应样例1：

  data:{"id":"a123","model_name":"llama_65b","model_version":null,"text_output":"live","details":{"generated_tokens":1,"first_token_cost":null,"decode_cost":null,"perf_stat":[[5735,28]],"batch_size":1,"queue_wait_time":5082},"prefill_time":28,"decode_time":null}
  
  data:{"id":"a123","model_name":"llama_65b","model_version":null,"text_output":" in","details":{"generated_tokens":2,"first_token_cost":null,"decode_cost":null,"perf_stat":[[5735,28],[297,9]],"batch_size":1,"queue_wait_time":36},"prefill_time":null,"decode_time":9}
  
  data:{"id":"a123","model_name":"llama_65b","model_version":null,"text_output":" Paris","details":{"generated_tokens":3,"first_token_cost":null,"decode_cost":null,"perf_stat":[[5735,28],[297,9],[3681,8]],"batch_size":1,"queue_wait_time":30},"prefill_time":null,"decode_time":8}
  
  data:{"id":"a123","model_name":"llama_65b","model_version":null,"text_output":",","details":{"generated_tokens":4,"first_token_cost":null,"decode_cost":null,"perf_stat":[[5735,28],[297,9],[3681,8],[29892,7]],"batch_size":1,"queue_wait_time":23},"prefill_time":null,"decode_time":7}
  
  data:{"id":"a123","model_name":"llama_65b","model_version":null,"text_output":" France","details":{"finish_reason":"length","generated_tokens":5,"first_token_cost":null,"decode_cost":null,"perf_stat":[[5735,28],[297,9],[3681,8],[29892,7],[3444,7]],"batch_size":1,"queue_wait_time":24},"prefill_time":null,"decode_time":7}
  

• 响应样例2（配置项“fullTextEnabled”=true）：

  data:{"id":"endpoint_common_20","model_name":"Qwen1.5-14B-Chat","model_version":null,"text_output":"'m","details":{"generated_tokens":1,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":5092},"prefill_time":41.68000030517578,"decode_time":null}
  
  data:{"id":"endpoint_common_20","model_name":"Qwen1.5-14B-Chat","model_version":null,"text_output":"'m from","details":{"generated_tokens":2,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":43},"prefill_time":null,"decode_time":20.440000534057617}
  
  data:{"id":"endpoint_common_20","model_name":"Qwen1.5-14B-Chat","model_version":null,"text_output":"'m from France","details":{"generated_tokens":3,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":27},"prefill_time":null,"decode_time":12.175999641418457}
  
  data:{"id":"endpoint_common_20","model_name":"Qwen1.5-14B-Chat","model_version":null,"text_output":"'m from France.","details":{"generated_tokens":4,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":26},"prefill_time":null,"decode_time":12.128000259399414}
  
  data:{"id":"endpoint_common_20","model_name":"Qwen1.5-14B-Chat","model_version":null,"text_output":"'m from France. I","details":{"finish_reason":"length","generated_tokens":5,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":26},"prefill_time":null,"decode_time":12.458000183105469}

输出说明