数据集使用

数据集获取

• 可以从/usr/local/Ascend/llm_model/tests/modeltest/README_NEW.md获取如下数据集：
  • BoolQ
  • HumanEval
  • HumanEval_X
  • GSM8K
  • LongBench
  • MMLU
  • NeedleBench
  • TruthfulQA
  

  以上数据集获取路径必须通过镜像包安装MindIE，具体安装方式请参见《MindIE安装指南》的“拉取镜像方式”章节。

• OA数据集需要从官方链接获取，获取方式如下：
  1. 单击链接获取OA数据集。
  2. 转换为token id方式。

     使用tokenizer_model.encode进行加密。

     python脚本示例参考如下：

     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

     import csv from pathlib import Path import pyarrow.parquet as pq import glob, os from transformers import AutoTokenizer def read_oa(dataset_path, tokenizer_model): out_list = [] for file_path in glob.glob((Path(dataset_path) / "*.parquet").as_posix()): file_name = file_path.split("/")[-1].split("-")[0] data_dict = pq.read_table(file_path).to_pandas() data_dict = data_dict[data_dict['lang'] == 'zh'] ques_list = data_dict['text'].to_list() for ques in ques_list: tokens = tokenizer_model.encode(ques) if len(tokens) <= 2048: out_list.append(tokens) else: out_list.append(tokens[0:2048]) return out_list def save_csv(file_path, out_tokens_list): with open(file_path, 'w', newline='') as csvfile: csv_writer = csv.writer(csvfile) for row in out_tokens_list: csv_writer.writerow(row) if __name__ == '__main__': model_path = "/data/models/baichuan2-7b" oa_dir = "/home/xxx/oasst1" save_path = "oa_tokens.csv" tokenizer_model = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=True, local_files_only=True) tokens_lists = read_oa(oa_dir, tokenizer_model) save_csv(save_path, tokens_lists)

GSM8K数据集转tokenids

使用pandas read_json后，然后使用tokenizer直接转换，再用numpy保存到csv中。

python脚本示例如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import numpy as np
import pandas as pd
from transformers import AutoTokenizer

MODEL_PATH = "/home/weight/llama2-70b"
OUT_FILE = "token_gsm8k.csv"
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True, use_fast=True, local_files_only=True)

def gen_requests_from_trace(trace_file):
    len = 0
    with open(OUT_FILE, "w") as f:
        df = pd.read_json(trace_file, lines=True)
        for i, row in df.iterrows():
            ques = row["question"]
            token = tokenizer([ques], return_tensors="np")
            token: np.ndarray = token["input_ids"].astype(np.int64)
            np.savetxt(f, token, fmt="%d", delimiter=",")
            len+=token.shape[-1]
    print(len / 1319)

if __name__ == '__main__':
    gen_requests_from_trace("./GSM8K.jsonl")