开源模型应用落地-DeepSeek-R1-Distill-Qwen-7B-LoRA微调-LLaMA-Factor
- 人工智能
- 2025-08-30 10:00:03
一、前言
如今，大语言模型领域热闹非凡，各种模型不断涌现。DeepSeek-R1-Distill-Qwen-7B 模型凭借其出色的效果和性能，吸引了众多开发者的目光。而 LLaMa-Factory 作为强大的微调工具，能让模型更好地满足个性化需求。
在本篇中，将深入探讨如何运用 LLaMa-Factory 对 DeepSeek-R1-Distill-Qwen-7B 模型进行微调，探索如何通过微调，让模型更好地为我们所用。
二、术语介绍 2.1. LoRA微调
LoRA (Low-Rank Adaptation) 用于微调大型语言模型 (LLM)。 是一种有效的自适应策略，它不会引入额外的推理延迟，并在保持模型质量的同时显着减少下游任务的可训练参数数量。
2.2. 参数高效微调(PEFT)仅微调少量 (额外) 模型参数，同时冻结预训练 LLM 的大部分参数，从而大大降低了计算和存储成本。
2.3. LLaMA-Factory是一个与 LLaMA（Large Language Model Meta AI）相关的项目，旨在为用户提供一种简化和优化的方式来训练、微调和部署大型语言模型。该工具通常包括一系列功能，如数据处理、模型配置、训练监控等，以帮助研究人员和开发者更高效地使用 LLaMA 模型。
LLaMA-Factory支持的模型列表：
2.4. DeepSeek-R1-Distill-Qwen-7B是一个由DeepSeek开发的模型，它是通过蒸馏技术将Qwen-7B大型模型的一部分知识精华提取出来，以适应更小型的模型需求。
三、前置条件 3.1. 基础环境及前置条件
1. 操作系统：centos7
2. NVIDIA Tesla V100 32GB CUDA Version: 12.2
3. 提前下载好DeepSeek-R1-Distill-Qwen-7B模型
通过以下两个地址进行下载，优先推荐魔搭
huggingface：
huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/tree/main
ModelScope：
魔搭社区
按需选择SDK或者Git方式下载
使用git-lfs方式下载示例：
3.2. Anaconda安装 1、Update System sudo yum update -y sudo yum upgrade -y 2、Download Anaconda wget repo.anaconda /archive/Anaconda3-2022.10-Linux-x86_64.sh 3、Verify Data Integrity sha256sum Anaconda3-2022.10-Linux-x86_64.sh 4、Run Anaconda Installation Script bash Anaconda3-2022.10-Linux-x86_64.sh 安装目录：/opt/anaconda3 注：安装位置可以在执行安装脚本的时候直接指定，可以这样修改执行内容 bash Anaconda3-2022.10-Linux-x86_64.sh -p /opt/anaconda3 Do you wish the installer to initialize Anaconda3 by running conda init? yes 如果没有执行初始化，可以执行：/opt/anaconda3/bin/conda init 注：初始化时，anaconda将配置写入了~/.bashrc 文件，直接执行 source ~/.bashrc 5、Verify Installation conda --version 6、配置镜像源 conda config --add channels mirrors.tuna.tsinghua.edu /anaconda/pkgs/free/ conda config --add channels mirrors.tuna.tsinghua.edu /anaconda/pkgs/main/ conda config --set show_channel_urls yes 3.3.下载LLaMA-Factory方式一：直接下载
地址：GitHub - hiyouga/LLaMA-Factory: Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
方式二：使用git克隆项目
git clone --depth 1 github /hiyouga/LLaMA-Factory.git下载好的项目放置在/data/service目录下
3.4. 安装依赖 conda create --name llama_factory python=3.10 conda activate llama_factory cd /data/service/LLaMA-Factory pip install -e ".[torch,metrics]" -i pypi.tuna.tsinghua.edu /simplePS：软硬件要求
四、技术实现 4.1.数据准备
有两种格式选择，包括alpaca和sharegpt
alpaca示例格式：
[ { "instruction": "人类指令（必填）", "input": "人类输入（选填）", "output": "模型回答（必填）", "system": "系统提示词（选填）", "history": [ ["第一轮指令（选填）", "第一轮回答（选填）"], ["第二轮指令（选填）", "第二轮回答（选填）"] ] } ]对于上述格式的数据，dataset_info.json 中的数据集描述应为：
"数据集名称": { "file_name": "data.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } }sharegpt示例格式：
相比 alpaca 格式的数据集，sharegpt 格式支持更多的角色种类，例如 human、gpt、observation、function 等等。它们构成一个对象列表呈现在 conversations 列中。注意其中 human 和 observation 必须出现在奇数位置，gpt 和 function 必须出现在偶数位置。 [ { "conversations": [ { "from": "human", "value": "人类指令" }, { "from": "function_call", "value": "工具参数" }, { "from": "observation", "value": "工具结果" }, { "from": "gpt", "value": "模型回答" } ], "system": "系统提示词（选填）", "tools": "工具描述（选填）" } ]对于上述格式的数据，dataset_info.json 中的数据集描述应为：
"数据集名称": { "file_name": "data.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" } }以下采用sharegpt格式，在/data/service/LLaMA-Factory/data/目录上传测试数据，文件名为ds_zh_demo.json，测试数据示例如下：
[ { "conversations": [ { "from": "user", "value": "什么是深度学习，它与传统机器学习有什么区别？" }, { "from": "assistant", "value": "深度学习是机器学习的一个子领域，使用人工神经网络，尤其是深度神经网络，来进行数据建模。与传统机器学习方法相比，深度学习能够自动提取特征，无需手动特征工程，适合处理大规模数据集和复杂模式。" } ] }, { "conversations": [ { "from": "user", "value": "如何评估一个机器学习模型的性能？" }, { "from": "assistant", "value": "评估机器学习模型的性能通常使用多种指标，包括准确率、精确率、召回率、F1-score、ROC曲线和AUC值。选择合适的指标取决于具体任务的性质和目标。" } ] } ]修改数据集描述文件dataset_info.json
vi /data/service/LLaMA-Factory/data/dataset_info.json增加以下内容：
"ds_zh_demo": { "file_name": "ds_zh_demo.json", "formatting": "sharegpt", "columns": { "messages": "conversations" }, "tags": { "role_tag": "from", "content_tag": "value", "user_tag": "user", "assistant_tag": "assistant" } } 4.2.配置文件准备1) 备份原有的配置文件
cp /data/service/LLaMA-Factory/examples/train_lora/llama3_lora_sft.yaml /data/service/LLaMA-Factory/examples/train_lora/llama3_lora_sft.yaml.bak2) 创建新的配置文件
mv /data/service/LLaMA-Factory/examples/train_lora/llama3_lora_sft.yaml /data/service/LLaMA-Factory/examples/train_lora/ds_qwen7b_lora_sft.yaml3) 修改配置文件内容
vi /data/service/LLaMA-Factory/examples/train_lora/ds_qwen7b_lora_sft.yaml内容如下：
### model model_name_or_path: /data/model/DeepSeek-R1-Distill-Qwen-7B trust_remote_code: true ### method stage: sft do_train: true finetuning_type: lora lora_rank: 8 lora_target: all ### dataset dataset: ds_zh_demo template: deepseek3 cutoff_len: 4096 max_samples: 4019 overwrite_cache: true preprocessing_num_workers: 16 ### output output_dir: /data/model/sft/DeepSeek-R1-Distill-Qwen-7B logging_steps: 10 save_steps: 500 plot_loss: true overwrite_output_dir: true ### train per_device_train_batch_size: 1 gradient_accumulation_steps: 8 learning_rate: 1.0e-4 num_train_epochs: 1.0 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true ddp_timeout: 180000000 ### eval val_size: 0.1 per_device_eval_batch_size: 1 eval_strategy: steps eval_steps: 500需要关注以下参数
model_name_or_path：模型路径dataset：数据集名称，对应上面声明的qwen_zh_demotemplate：模版cutoff_len：控制输入序列的最大长度output_dir：微调后权重保存路径gradient_accumulation_steps：梯度累积的步数，GPU资源不足时需要减少该值num_train_epochs：训练的轮数 4.3.启动微调 conda activate llama_factory cd /data/service/LLaMA-Factory llamafactory-cli train /data/service/LLaMA-Factory/examples/train_lora/ds_qwen7b_lora_sft.yaml # 后台运行 nohup llamafactory-cli train /data/service/LLaMA-Factory/examples/train_lora/ds_qwen7b_lora_sft.yaml > output.log 2>&1 & 4.4.微调结果 [INFO|configuration_utils.py:1052] 2025-02-18 16:39:55,400 >> loading configuration file /data/model/DeepSeek-R1-Distill-Qwen-7B/generation_config.json [INFO|configuration_utils.py:1099] 2025-02-18 16:39:55,400 >> Generate config GenerationConfig { "bos_token_id": 151646, "do_sample": true, "eos_token_id": 151643, "temperature": 0.6, "top_p": 0.95 } [INFO|2025-02-18 16:39:55] llamafactory.model.model_utils.checkpointing:157 >> Gradient checkpointing enabled. [INFO|2025-02-18 16:39:55] llamafactory.model.model_utils.attention:157 >> Using torch SDPA for faster training and inference. [INFO|2025-02-18 16:39:55] llamafactory.model.adapter:157 >> Upcasting trainable params to float32. [INFO|2025-02-18 16:39:55] llamafactory.model.adapter:157 >> Fine-tuning method: LoRA [INFO|2025-02-18 16:39:55] llamafactory.model.model_utils.misc:157 >> Found linear modules: down_proj,o_proj,up_proj,k_proj,v_proj,q_proj,gate_proj [INFO|2025-02-18 16:39:55] llamafactory.model.loader:157 >> trainable params: 20,185,088 || all params: 7,635,801,600 || trainable%: 0.2643 Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. [INFO|trainer.py:667] 2025-02-18 16:39:55,807 >> Using auto half precision backend [INFO|trainer.py:2243] 2025-02-18 16:39:56,634 >> ***** Running training ***** [INFO|trainer.py:2244] 2025-02-18 16:39:56,634 >> Num examples = 3,617 [INFO|trainer.py:2245] 2025-02-18 16:39:56,634 >> Num Epochs = 1 [INFO|trainer.py:2246] 2025-02-18 16:39:56,634 >> Instantaneous batch size per device = 1 [INFO|trainer.py:2249] 2025-02-18 16:39:56,634 >> Total train batch size (w. parallel, distributed & accumulation) = 8 [INFO|trainer.py:2250] 2025-02-18 16:39:56,634 >> Gradient Accumulation steps = 8 [INFO|trainer.py:2251] 2025-02-18 16:39:56,634 >> Total optimization steps = 452 [INFO|trainer.py:2252] 2025-02-18 16:39:56,638 >> Number of trainable parameters = 20,185,088 0%| | 0/452 [00:00<?, ?it/s]/usr/local/miniconda3/envs/llama_factory/lib/python3.10/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead. with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined] 100%|██████████| 452/452 [4:06:28<00:00, 31.87s/it][INFO|trainer.py:3705] 2025-02-18 20:46:24,795 >> Saving model checkpoint to /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/checkpoint-452 [INFO|configuration_utils.py:670] 2025-02-18 20:46:24,819 >> loading configuration file /data/model/DeepSeek-R1-Distill-Qwen-7B/config.json [INFO|configuration_utils.py:739] 2025-02-18 20:46:24,820 >> Model config Qwen2Config { "architectures": [ "Qwen2ForCausalLM" ], "attention_dropout": 0.0, "bos_token_id": 151643, "eos_token_id": 151643, "hidden_act": "silu", "hidden_size": 3584, "initializer_range": 0.02, "intermediate_size": 18944, "max_position_embeddings": 131072, "max_window_layers": 28, "model_type": "qwen2", "num_attention_heads": 28, "num_hidden_layers": 28, "num_key_value_heads": 4, "rms_norm_eps": 1e-06, "rope_scaling": null, "rope_theta": 10000, "sliding_window": null, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.45.0", "use_cache": true, "use_mrope": false, "use_sliding_window": false, "vocab_size": 152064 } [INFO|tokenization_utils_base.py:2649] 2025-02-18 20:46:25,042 >> tokenizer config file saved in /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/checkpoint-452/tokenizer_config.json [INFO|tokenization_utils_base.py:2658] 2025-02-18 20:46:25,043 >> Special tokens file saved in /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/checkpoint-452/special_tokens_map.json [INFO|trainer.py:2505] 2025-02-18 20:46:25,377 >> Training completed. Do not forget to share your model on huggingface.co/models =) 100%|██████████| 452/452 [4:06:28<00:00, 32.72s/it] [INFO|trainer.py:3705] 2025-02-18 20:46:25,379 >> Saving model checkpoint to /data/model/sft/DeepSeek-R1-Distill-Qwen-7B [INFO|configuration_utils.py:670] 2025-02-18 20:46:25,401 >> loading configuration file /data/model/DeepSeek-R1-Distill-Qwen-7B/config.json [INFO|configuration_utils.py:739] 2025-02-18 20:46:25,401 >> Model config Qwen2Config { "architectures": [ "Qwen2ForCausalLM" ], "attention_dropout": 0.0, "bos_token_id": 151643, "eos_token_id": 151643, "hidden_act": "silu", "hidden_size": 3584, "initializer_range": 0.02, "intermediate_size": 18944, "max_position_embeddings": 131072, "max_window_layers": 28, "model_type": "qwen2", "num_attention_heads": 28, "num_hidden_layers": 28, "num_key_value_heads": 4, "rms_norm_eps": 1e-06, "rope_scaling": null, "rope_theta": 10000, "sliding_window": null, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.45.0", "use_cache": true, "use_mrope": false, "use_sliding_window": false, "vocab_size": 152064 } [INFO|tokenization_utils_base.py:2649] 2025-02-18 20:46:25,556 >> tokenizer config file saved in /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/tokenizer_config.json [INFO|tokenization_utils_base.py:2658] 2025-02-18 20:46:25,556 >> Special tokens file saved in /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/special_tokens_map.json {'loss': 3.6592, 'grad_norm': 0.38773563504219055, 'learning_rate': 2.173913043478261e-05, 'epoch': 0.02} {'loss': 3.667, 'grad_norm': 0.698821485042572, 'learning_rate': 4.347826086956522e-05, 'epoch': 0.04} {'loss': 3.4784, 'grad_norm': 0.41371676325798035, 'learning_rate': 6.521739130434783e-05, 'epoch': 0.07} {'loss': 3.2962, 'grad_norm': 0.4966348111629486, 'learning_rate': 8.695652173913044e-05, 'epoch': 0.09} {'loss': 3.0158, 'grad_norm': 0.333425909280777, 'learning_rate': 9.997605179330019e-05, 'epoch': 0.11} {'loss': 3.2221, 'grad_norm': 0.3786776065826416, 'learning_rate': 9.970689785771798e-05, 'epoch': 0.13} {'loss': 2.8439, 'grad_norm': 0.3683229386806488, 'learning_rate': 9.914027086842322e-05, 'epoch': 0.15} {'loss': 3.0528, 'grad_norm': 0.42745739221572876, 'learning_rate': 9.82795618288397e-05, 'epoch': 0.18} {'loss': 2.9092, 'grad_norm': 0.45462721586227417, 'learning_rate': 9.712992168898436e-05, 'epoch': 0.2} {'loss': 3.1055, 'grad_norm': 0.5547119379043579, 'learning_rate': 9.56982305193869e-05, 'epoch': 0.22} {'loss': 2.9412, 'grad_norm': 0.5830215811729431, 'learning_rate': 9.399305633701373e-05, 'epoch': 0.24} {'loss': 2.7873, 'grad_norm': 0.5862609148025513, 'learning_rate': 9.202460382960448e-05, 'epoch': 0.27} {'loss': 2.8255, 'grad_norm': 0.5828853845596313, 'learning_rate': 8.980465328528219e-05, 'epoch': 0.29} {'loss': 2.6266, 'grad_norm': 0.6733331084251404, 'learning_rate': 8.734649009291585e-05, 'epoch': 0.31} {'loss': 2.8745, 'grad_norm': 0.6904928684234619, 'learning_rate': 8.46648252351431e-05, 'epoch': 0.33} {'loss': 2.8139, 'grad_norm': 0.7874809503555298, 'learning_rate': 8.177570724986628e-05, 'epoch': 0.35} {'loss': 2.7818, 'grad_norm': 0.8345168232917786, 'learning_rate': 7.86964261870916e-05, 'epoch': 0.38} {'loss': 2.7198, 'grad_norm': 0.8806198239326477, 'learning_rate': 7.544541013588645e-05, 'epoch': 0.4} {'loss': 2.7231, 'grad_norm': 0.9481658935546875, 'learning_rate': 7.204211494069292e-05, 'epoch': 0.42} {'loss': 2.7371, 'grad_norm': 0.9718573093414307, 'learning_rate': 6.850690776699573e-05, 'epoch': 0.44} {'loss': 2.6862, 'grad_norm': 1.2056019306182861, 'learning_rate': 6.486094521315022e-05, 'epoch': 0.46} {'loss': 2.4661, 'grad_norm': 1.200085163116455, 'learning_rate': 6.112604669781572e-05, 'epoch': 0.49} {'loss': 2.4841, 'grad_norm': 1.1310691833496094, 'learning_rate': 5.732456388071247e-05, 'epoch': 0.51} {'loss': 2.3755, 'grad_norm': 1.1279083490371704, 'learning_rate': 5.3479246898159063e-05, 'epoch': 0.53} {'loss': 2.5552, 'grad_norm': 1.2654848098754883, 'learning_rate': 4.96131082139099e-05, 'epoch': 0.55} {'loss': 2.6197, 'grad_norm': 1.3887016773223877, 'learning_rate': 4.574928490008264e-05, 'epoch': 0.58} {'loss': 2.3773, 'grad_norm': 1.3009178638458252, 'learning_rate': 4.1910900172361764e-05, 'epoch': 0.6} {'loss': 2.3881, 'grad_norm': 1.346793532371521, 'learning_rate': 3.812092500812646e-05, 'epoch': 0.62} {'loss': 2.4821, 'grad_norm': 1.7273674011230469, 'learning_rate': 3.440204067565511e-05, 'epoch': 0.64} {'loss': 2.3563, 'grad_norm': 1.529177188873291, 'learning_rate': 3.077650299710653e-05, 'epoch': 0.66} {'loss': 2.1308, 'grad_norm': 1.5957469940185547, 'learning_rate': 2.7266009157601224e-05, 'epoch': 0.69} {'loss': 2.1709, 'grad_norm': 1.4444897174835205, 'learning_rate': 2.3891567857490372e-05, 'epoch': 0.71} {'loss': 2.275, 'grad_norm': 1.5686719417572021, 'learning_rate': 2.067337358489085e-05, 'epoch': 0.73} {'loss': 2.2075, 'grad_norm': 1.5931408405303955, 'learning_rate': 1.7630685760908622e-05, 'epoch': 0.75} {'loss': 2.1727, 'grad_norm': 1.7681787014007568, 'learning_rate': 1.4781713480810184e-05, 'epoch': 0.77} {'loss': 2.3562, 'grad_norm': 1.742925763130188, 'learning_rate': 1.2143506540914128e-05, 'epoch': 0.8} {'loss': 2.1187, 'grad_norm': 1.6716198921203613, 'learning_rate': 9.731853403356705e-06, 'epoch': 0.82} {'loss': 2.2564, 'grad_norm': 1.915489912033081, 'learning_rate': 7.561186709365653e-06, 'epoch': 0.84} {'loss': 2.261, 'grad_norm': 2.132519245147705, 'learning_rate': 5.644496906502233e-06, 'epoch': 0.86} {'loss': 2.1632, 'grad_norm': 1.591231107711792, 'learning_rate': 3.9932545067728366e-06, 'epoch': 0.88} {'loss': 2.1266, 'grad_norm': 1.584917664527893, 'learning_rate': 2.6173414408598827e-06, 'epoch': 0.91} {'loss': 2.2944, 'grad_norm': 1.5982666015625, 'learning_rate': 1.524991919285429e-06, 'epoch': 0.93} {'loss': 2.3799, 'grad_norm': 2.1475727558135986, 'learning_rate': 7.227431544266194e-07, 'epoch': 0.95} {'loss': 2.1196, 'grad_norm': 1.6714484691619873, 'learning_rate': 2.153962382888841e-07, 'epoch': 0.97} {'loss': 2.1427, 'grad_norm': 1.7334465980529785, 'learning_rate': 5.987410165758656e-09, 'epoch': 1.0} {'train_runtime': 14788.7396, 'train_samples_per_second': 0.245, 'train_steps_per_second': 0.031, 'train_loss': 2.6206856934370193, 'epoch': 1.0} ***** train metrics ***** epoch = 0.9997 total_flos = 100517734GF train_loss = 2.6207 train_runtime = 4:06:28.73 train_samples_per_second = 0.245 train_steps_per_second = 0.031 Figure saved at: /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/training_loss.png [WARNING|2025-02-18 20:46:25] llamafactory.extras.ploting:162 >> No metric eval_loss to plot. [WARNING|2025-02-18 20:46:25] llamafactory.extras.ploting:162 >> No metric eval_accuracy to plot. [INFO|trainer.py:4021] 2025-02-18 20:46:25,781 >> ***** Running Evaluation ***** [INFO|trainer.py:4023] 2025-02-18 20:46:25,781 >> Num examples = 402 [INFO|trainer.py:4026] 2025-02-18 20:46:25,781 >> Batch size = 1 100%|██████████| 402/402 [09:03<00:00, 1.35s/it]t] [INFO|modelcard.py:449] 2025-02-18 20:55:30,409 >> Dropping the following result as it does not have all the necessary fields: {'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}} ***** eval metrics ***** epoch = 0.9997 eval_loss = 2.2648 eval_runtime = 0:09:04.62 eval_samples_per_second = 0.738 eval_steps_per_second = 0.738生成的权重文件：
五、附带说明 5.1. dataset_info.json
包含了所有可用的数据集。如果您希望使用自定义数据集，请务必在 dataset_info.json 文件中添加数据集描述，并通过修改 dataset: 数据集名称 配置来使用数据集。
"数据集名称": { "hf_hub_url": "Hugging Face 的数据集仓库地址（若指定，则忽略 script_url 和 file_name）", "ms_hub_url": "ModelScope 的数据集仓库地址（若指定，则忽略 script_url 和 file_name）", "script_url": "包含数据加载脚本的本地文件夹名称（若指定，则忽略 file_name）", "file_name": "该目录下数据集文件夹或文件的名称（若上述参数未指定，则此项必需）", "formatting": "数据集格式（可选，默认：alpaca，可以为 alpaca 或 sharegpt）", "ranking": "是否为偏好数据集（可选，默认：False）", "subset": "数据集子集的名称（可选，默认：None）", "split": "所使用的数据集切分（可选，默认：train）", "folder": "Hugging Face 仓库的文件夹名称（可选，默认：None）", "num_samples": "该数据集所使用的样本数量。（可选，默认：None）", "columns（可选）": { "prompt": "数据集代表提示词的表头名称（默认：instruction）", "query": "数据集代表请求的表头名称（默认：input）", "response": "数据集代表回答的表头名称（默认：output）", "history": "数据集代表历史对话的表头名称（默认：None）", "messages": "数据集代表消息列表的表头名称（默认：conversations）", "system": "数据集代表系统提示的表头名称（默认：None）", "tools": "数据集代表工具描述的表头名称（默认：None）", "images": "数据集代表图像输入的表头名称（默认：None）", "videos": "数据集代表视频输入的表头名称（默认：None）", "audios": "数据集代表音频输入的表头名称（默认：None）", "chosen": "数据集代表更优回答的表头名称（默认：None）", "rejected": "数据集代表更差回答的表头名称（默认：None）", "kto_tag": "数据集代表 KTO 标签的表头名称（默认：None）" }, "tags（可选，用于 sharegpt 格式）": { "role_tag": "消息中代表发送者身份的键名（默认：from）", "content_tag": "消息中代表文本内容的键名（默认：value）", "user_tag": "消息中代表用户的 role_tag（默认：human）", "assistant_tag": "消息中代表助手的 role_tag（默认：gpt）", "observation_tag": "消息中代表工具返回结果的 role_tag（默认：observation）", "function_tag": "消息中代表工具调用的 role_tag（默认：function_call）", "system_tag": "消息中代表系统提示的 role_tag（默认：system，会覆盖 system column）" } } 5.2. 自定义对话模版在 template.py 中添加自己的对话模板。
github /hiyouga/LLaMA-Factory/blob/main/src/llamafactory/data/template.py
# Copyright 2025 the LlamaFactory team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http:// .apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from dataclasses import dataclass from typing import TYPE_CHECKING, Dict, List, Optional, Sequence, Tuple, Type, Union from typing_extensions import override from ..extras import logging from ..extras.misc import check_version from .data_utils import Role from .formatter import EmptyFormatter, FunctionFormatter, StringFormatter, ToolFormatter from .mm_plugin import get_mm_plugin if TYPE_CHECKING: from transformers import PreTrainedTokenizer from ..hparams import DataArguments from .formatter import SLOTS, Formatter from .mm_plugin import BasePlugin from .tool_utils import FunctionCall logger = logging.get_logger(__name__) @dataclass class Template: format_user: "Formatter" format_assistant: "Formatter" format_system: "Formatter" format_function: "Formatter" format_observation: "Formatter" format_tools: "Formatter" format_prefix: "Formatter" default_system: str stop_words: List[str] thought_words: Tuple[str, str] efficient_eos: bool replace_eos: bool replace_jinja_template: bool mm_plugin: "BasePlugin" def encode_oneturn( self, tokenizer: "PreTrainedTokenizer", messages: Sequence[Dict[str, str]], system: Optional[str] = None, tools: Optional[str] = None, ) -> Tuple[List[int], List[int]]: r""" Returns a single pair of token ids representing prompt and response respectively. """ encoded_messages = self._encode(tokenizer, messages, system, tools) prompt_ids = [] for encoded_ids in encoded_messages[:-1]: prompt_ids += encoded_ids response_ids = encoded_messages[-1] return prompt_ids, response_ids def encode_multiturn( self, tokenizer: "PreTrainedTokenizer", messages: Sequence[Dict[str, str]], system: Optional[str] = None, tools: Optional[str] = None, ) -> List[Tuple[List[int], List[int]]]: r""" Returns multiple pairs of token ids representing prompts and responses respectively. """ encoded_messages = self._encode(tokenizer, messages, system, tools) return [(encoded_messages[i], encoded_messages[i + 1]) for i in range(0, len(encoded_messages), 2)] def extract_tool(self, content: str) -> Union[str, List["FunctionCall"]]: r""" Extracts tool message. """ return self.format_tools.extract(content) def get_stop_token_ids(self, tokenizer: "PreTrainedTokenizer") -> List[int]: r""" Returns stop token ids. """ stop_token_ids = {tokenizer.eos_token_id} for token in self.stop_words: stop_token_ids.add(tokenizer.convert_tokens_to_ids(token)) return list(stop_token_ids) def _convert_elements_to_ids(self, tokenizer: "PreTrainedTokenizer", elements: "SLOTS") -> List[int]: r""" Converts elements to token ids. """ token_ids = [] for elem in elements: if isinstance(elem, str): if len(elem) != 0: token_ids += tokenizer.encode(elem, add_special_tokens=False) elif isinstance(elem, dict): token_ids += [tokenizer.convert_tokens_to_ids(elem.get("token"))] elif isinstance(elem, set): if "bos_token" in elem and tokenizer.bos_token_id is not None: token_ids += [tokenizer.bos_token_id] elif "eos_token" in elem and tokenizer.eos_token_id is not None: token_ids += [tokenizer.eos_token_id] else: raise ValueError(f"Input must be string, set[str] or dict[str, str], got {type(elem)}") return token_ids def _encode( self, tokenizer: "PreTrainedTokenizer", messages: Sequence[Dict[str, str]], system: Optional[str], tools: Optional[str], ) -> List[List[int]]: r""" Encodes formatted inputs to pairs of token ids. Turn 0: prefix + system + query resp Turn t: query resp """ system = system or self.default_system encoded_messages = [] for i, message in enumerate(messages): elements = [] if i == 0: elements += self.format_prefix.apply() if system or tools: tool_text = self.format_tools.apply(content=tools)[0] if tools else "" elements += self.format_system.apply(content=(system + tool_text)) if message["role"] == Role.USER.value: elements += self.format_user.apply(content=message["content"], idx=str(i // 2)) elif message["role"] == Role.ASSISTANT.value: elements += self.format_assistant.apply(content=message["content"]) elif message["role"] == Role.OBSERVATION.value: elements += self.format_observation.apply(content=message["content"]) elif message["role"] == Role.FUNCTION.value: elements += self.format_function.apply(content=message["content"]) else: raise NotImplementedError("Unexpected role: {}".format(message["role"])) encoded_messages.append(self._convert_elements_to_ids(tokenizer, elements)) return encoded_messages @staticmethod def _add_or_replace_eos_token(tokenizer: "PreTrainedTokenizer", eos_token: str) -> None: r""" Adds or replaces eos token to the tokenizer. """ is_added = tokenizer.eos_token_id is None num_added_tokens = tokenizer.add_special_tokens({"eos_token": eos_token}) if is_added: logger.info_rank0(f"Add eos token: {tokenizer.eos_token}.") else: logger.info_rank0(f"Replace eos token: {tokenizer.eos_token}.") if num_added_tokens > 0: logger.warning_rank0("New tokens have been added, make sure `resize_vocab` is True.") def fix_special_tokens(self, tokenizer: "PreTrainedTokenizer") -> None: r""" Adds eos token and pad token to the tokenizer. """ stop_words = self.stop_words if self.replace_eos: if not stop_words: raise ValueError("Stop words are required to replace the EOS token.") self._add_or_replace_eos_token(tokenizer, eos_token=stop_words[0]) stop_words = stop_words[1:] if tokenizer.eos_token_id is None: self._add_or_replace_eos_token(tokenizer, eos_token="<|endoftext|>") if tokenizer.pad_token_id is None: tokenizer.pad_token = tokenizer.eos_token logger.info_rank0(f"Add pad token: {tokenizer.pad_token}") if stop_words: num_added_tokens = tokenizer.add_special_tokens( dict(additional_special_tokens=stop_words), replace_additional_special_tokens=False ) logger.info_rank0("Add {} to stop words.".format(",".join(stop_words))) if num_added_tokens > 0: logger.warning_rank0("New tokens have been added, make sure `resize_vocab` is True.") @staticmethod def _jinja_escape(content: str) -> str: r""" Escape single quotes in content. """ return content.replace("'", r"\'") @staticmethod def _convert_slots_to_jinja(slots: "SLOTS", tokenizer: "PreTrainedTokenizer", placeholder: str = "content") -> str: r""" Converts slots to jinja template. """ slot_items = [] for slot in slots: if isinstance(slot, str): slot_pieces = slot.split("{{content}}") if slot_pieces[0]: slot_items.append("'" + Template._jinja_escape(slot_pieces[0]) + "'") if len(slot_pieces) > 1: slot_items.append(placeholder) if slot_pieces[1]: slot_items.append("'" + Template._jinja_escape(slot_pieces[1]) + "'") elif isinstance(slot, set): # do not use {{ eos_token }} since it may be replaced if "bos_token" in slot and tokenizer.bos_token_id is not None: slot_items.append("'" + tokenizer.bos_token + "'") elif "eos_token" in slot and tokenizer.eos_token_id is not None: slot_items.append("'" + tokenizer.eos_token + "'") elif isinstance(slot, dict): raise ValueError("Dict is not supported.") return " + ".join(slot_items) def _get_jinja_template(self, tokenizer: "PreTrainedTokenizer") -> str: r""" Returns the jinja template. """ prefix = self._convert_slots_to_jinja(self.format_prefix.apply(), tokenizer) system = self._convert_slots_to_jinja(self.format_system.apply(), tokenizer, placeholder="system_message") user = self._convert_slots_to_jinja(self.format_user.apply(), tokenizer) assistant = self._convert_slots_to_jinja(self.format_assistant.apply(), tokenizer) jinja_template = "" if prefix: jinja_template += "{{ " + prefix + " }}" if self.default_system: jinja_template += "{% set system_message = '" + self._jinja_escape(self.default_system) + "' %}" jinja_template += ( "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}" "{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}" "{% if system_message is defined %}{{ " + system + " }}{% endif %}" "{% for message in loop_messages %}" "{% set content = message['content'] %}" "{% if message['role'] == 'user' %}" "{{ " + user + " }}" "{% elif message['role'] == 'assistant' %}" "{{ " + assistant + " }}" "{% endif %}" "{% endfor %}" ) return jinja_template def fix_jinja_template(self, tokenizer: "PreTrainedTokenizer") -> None: r""" Replaces the jinja template in the tokenizer. """ if tokenizer.chat_template is None or self.replace_jinja_template: try: tokenizer.chat_template = self._get_jinja_template(tokenizer) except ValueError as e: logger.info_rank0(f"Cannot add this chat template to tokenizer: {e}.") @staticmethod def _convert_slots_to_ollama( slots: "SLOTS", tokenizer: "PreTrainedTokenizer", placeholder: str = "content" ) -> str: r""" Converts slots to ollama template. """ slot_items = [] for slot in slots: if isinstance(slot, str): slot_pieces = slot.split("{{content}}") if slot_pieces[0]: slot_items.append(slot_pieces[0]) if len(slot_pieces) > 1: slot_items.append("{{ " + placeholder + " }}") if slot_pieces[1]: slot_items.append(slot_pieces[1]) elif isinstance(slot, set): # do not use {{ eos_token }} since it may be replaced if "bos_token" in slot and tokenizer.bos_token_id is not None: slot_items.append(tokenizer.bos_token) elif "eos_token" in slot and tokenizer.eos_token_id is not None: slot_items.append(tokenizer.eos_token) elif isinstance(slot, dict): raise ValueError("Dict is not supported.") return "".join(slot_items) def _get_ollama_template(self, tokenizer: "PreTrainedTokenizer") -> str: r""" Returns the ollama template. """ prefix = self._convert_slots_to_ollama(self.format_prefix.apply(), tokenizer) system = self._convert_slots_to_ollama(self.format_system.apply(), tokenizer, placeholder=".System") user = self._convert_slots_to_ollama(self.format_user.apply(), tokenizer, placeholder=".Content") assistant = self._convert_slots_to_ollama(self.format_assistant.apply(), tokenizer, placeholder=".Content") return ( f"{prefix}{{{{ if .System }}}}{system}{{{{ end }}}}" f"""{{{{ range .Messages }}}}{{{{ if eq .Role "user" }}}}{user}""" f"""{{{{ else if eq .Role "assistant" }}}}{assistant}{{{{ end }}}}{{{{ end }}}}""" ) def get_ollama_modelfile(self, tokenizer: "PreTrainedTokenizer") -> str: r""" Returns the ollama modelfile. TODO: support function calling. """ modelfile = "# ollama modelfile auto-generated by llamafactory\n\n" modelfile += f'FROM .\n\nTEMPLATE """{self._get_ollama_template(tokenizer)}"""\n\n' if self.default_system: modelfile += f'SYSTEM """{self.default_system}"""\n\n' for stop_token_id in self.get_stop_token_ids(tokenizer): modelfile += f'PARAMETER stop "{tokenizer.convert_ids_to_tokens(stop_token_id)}"\n' modelfile += "PARAMETER num_ctx 4096\n" return modelfile @dataclass class Llama2Template(Template): @override def _encode( self, tokenizer: "PreTrainedTokenizer", messages: Sequence[Dict[str, str]], system: str, tools: str, ) -> List[List[int]]: system = system or self.default_system encoded_messages = [] for i, message in enumerate(messages): elements = [] system_text = "" if i == 0: elements += self.format_prefix.apply() if system or tools: tool_text = self.format_tools.apply(content=tools)[0] if tools else "" system_text = self.format_system.apply(content=(system + tool_text))[0] if message["role"] == Role.USER.value: elements += self.format_user.apply(content=system_text + message["content"]) elif message["role"] == Role.ASSISTANT.value: elements += self.format_assistant.apply(content=message["content"]) elif message["role"] == Role.OBSERVATION.value: elements += self.format_observation.apply(content=message["content"]) elif message["role"] == Role.FUNCTION.value: elements += self.format_function.apply(content=message["content"]) else: raise NotImplementedError("Unexpected role: {}".format(message["role"])) encoded_messages.append(self._convert_elements_to_ids(tokenizer, elements)) return encoded_messages def _get_jinja_template(self, tokenizer: "PreTrainedTokenizer") -> str: prefix = self._convert_slots_to_jinja(self.format_prefix.apply(), tokenizer) system_message = self._convert_slots_to_jinja( self.format_system.apply(), tokenizer, placeholder="system_message" ) user_message = self._convert_slots_to_jinja(self.format_user.apply(), tokenizer) assistant_message = self._convert_slots_to_jinja(self.format_assistant.apply(), tokenizer) jinja_template = "" if prefix: jinja_template += "{{ " + prefix + " }}" if self.default_system: jinja_template += "{% set system_message = '" + self._jinja_escape(self.default_system) + "' %}" jinja_template += ( "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}" "{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}" "{% for message in loop_messages %}" "{% if loop.index0 == 0 and system_message is defined %}" "{% set content = " + system_message + " + message['content'] %}" "{% else %}{% set content = message['content'] %}{% endif %}" "{% if message['role'] == 'user' %}" "{{ " + user_message + " }}" "{% elif message['role'] == 'assistant' %}" "{{ " + assistant_message + " }}" "{% endif %}" "{% endfor %}" ) return jinja_template TEMPLATES: Dict[str, "Template"] = {} def register_template( name: str, format_user: Optional["Formatter"] = None, format_assistant: Optional["Formatter"] = None, format_system: Optional["Formatter"] = None, format_function: Optional["Formatter"] = None, format_observation: Optional["Formatter"] = None, format_tools: Optional["Formatter"] = None, format_prefix: Optional["Formatter"] = None, default_system: str = "", stop_words: Optional[Sequence[str]] = None, thought_words: Optional[Tuple[str, str]] = None, efficient_eos: bool = False, replace_eos: bool = False, replace_jinja_template: bool = False, mm_plugin: "BasePlugin" = get_mm_plugin(name="base"), template_class: Type["Template"] = Template, ) -> None: r""" Registers a chat template. To add the following chat template: ``` <s><user>user prompt here <model>model response here</s> <user>user prompt here <model>model response here</s> ``` The corresponding code should be: ``` register_template( name="custom", format_user=StringFormatter(slots=["<user>{{content}}\n<model>"]), format_assistant=StringFormatter(slots=["{{content}}</s>\n"]), format_prefix=EmptyFormatter("<s>"), ) ``` """ if name in TEMPLATES: raise ValueError(f"Template {name} already exists.") default_slots = ["{{content}}"] if efficient_eos else ["{{content}}", {"eos_token"}] default_user_formatter = StringFormatter(slots=["{{content}}"]) default_assistant_formatter = StringFormatter(slots=default_slots) default_function_formatter = FunctionFormatter(slots=default_slots, tool_format="default") default_tool_formatter = ToolFormatter(tool_format="default") default_prefix_formatter = EmptyFormatter() TEMPLATES[name] = template_class( format_user=format_user or default_user_formatter, format_assistant=format_assistant or default_assistant_formatter, format_system=format_system or default_user_formatter, format_function=format_function or default_function_formatter, format_observation=format_observation or format_user or default_user_formatter, format_tools=format_tools or default_tool_formatter, format_prefix=format_prefix or default_prefix_formatter, default_system=default_system, stop_words=stop_words or [], thought_words=thought_words or ("<think>", "</think>"), efficient_eos=efficient_eos, replace_eos=replace_eos, replace_jinja_template=replace_jinja_template, mm_plugin=mm_plugin, ) def parse_template(tokenizer: "PreTrainedTokenizer") -> "Template": r""" Extracts a chat template from the tokenizer. """ def find_diff(short_str: str, long_str: str) -> str: i, j = 0, 0 diff = "" while i < len(short_str) and j < len(long_str): if short_str[i] == long_str[j]: i += 1 j += 1 else: diff += long_str[j] j += 1 return diff prefix = tokenizer.decode(tokenizer.encode("")) messages = [{"role": "system", "content": "{{content}}"}] system_slot = tokenizer.apply_chat_template(messages, add_generation_prompt=False, tokenize=False)[len(prefix) :] messages = [{"role": "system", "content": ""}, {"role": "user", "content": "{{content}}"}] user_slot_empty_system = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) user_slot_empty_system = user_slot_empty_system[len(prefix) :] messages = [{"role": "user", "content": "{{content}}"}] user_slot = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) user_slot = user_slot[len(prefix) :] messages = [{"role": "user", "content": "{{content}}"}, {"role": "assistant", "content": "{{content}}"}] assistant_slot = tokenizer.apply_chat_template(messages, add_generation_prompt=False, tokenize=False) assistant_slot = assistant_slot[len(prefix) + len(user_slot) :] if len(user_slot) > len(user_slot_empty_system): default_system = find_diff(user_slot_empty_system, user_slot) sole_system = system_slot.replace("{{content}}", default_system, 1) user_slot = user_slot[len(sole_system) :] else: # if defaut_system is empty, user_slot_empty_system will be longer than user_slot default_system = "" return Template( format_user=StringFormatter(slots=[user_slot]), format_assistant=StringFormatter(slots=[assistant_slot]), format_system=StringFormatter(slots=[system_slot]), format_function=FunctionFormatter(slots=[assistant_slot], tool_format="default"), format_observation=StringFormatter(slots=[user_slot]), format_tools=ToolFormatter(tool_format="default"), format_prefix=EmptyFormatter(slots=[prefix]) if prefix else EmptyFormatter(), default_system=default_system, stop_words=[], thought_words=("<think>", "</think>"), efficient_eos=False, replace_eos=False, replace_jinja_template=False, mm_plugin=get_mm_plugin(name="base"), ) def get_template_and_fix_tokenizer(tokenizer: "PreTrainedTokenizer", data_args: "DataArguments") -> "Template": r""" Gets chat template and fixes the tokenizer. """ if data_args.template is None: if isinstance(tokenizer.chat_template, str): logger.warning_rank0("`template` was not specified, try parsing the chat template from the tokenizer.") template = parse_template(tokenizer) else: logger.warning_rank0("`template` was not specified, use `empty` template.") template = TEMPLATES["empty"] # placeholder else: if data_args.template not in TEMPLATES: raise ValueError(f"Template {data_args.template} does not exist.") template = TEMPLATES[data_args.template] if template.mm_plugin.__class__.__name__ != "BasePlugin": check_version("transformers>=4.45.0") if data_args.train_on_prompt and template.efficient_eos: raise ValueError("Current template does not support `train_on_prompt`.") if data_args.tool_format is not None: logger.info_rank0(f"Using tool format: {data_args.tool_format}.") default_slots = ["{{content}}"] if template.efficient_eos else ["{{content}}", {"eos_token"}] template.format_function = FunctionFormatter(slots=default_slots, tool_format=data_args.tool_format) template.format_tools = ToolFormatter(tool_format=data_args.tool_format) template.fix_special_tokens(tokenizer) template.fix_jinja_template(tokenizer) return template register_template( name="alpaca", format_user=StringFormatter(slots=["### Instruction:\n{{content}}\n\n### Response:\n"]), format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}, "\n\n"]), default_system=( "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n" ), replace_jinja_template=True, ) register_template( name="aquila", format_user=StringFormatter(slots=["Human: {{content}}###Assistant:"]), format_assistant=StringFormatter(slots=["{{content}}###"]), format_system=StringFormatter(slots=["System: {{content}}###"]), default_system=( "A chat between a curious human and an artificial intelligence assistant. " "The assistant gives helpful, detailed, and polite answers to the human's questions." ), stop_words=["</s>"], ) register_template( name="atom", format_user=StringFormatter( slots=[{"bos_token"}, "Human: {{content}}\n", {"eos_token"}, {"bos_token"}, "Assistant:"] ), format_assistant=StringFormatter(slots=["{{content}}\n", {"eos_token"}]), ) register_template( name="baichuan", format_user=StringFormatter(slots=[{"token": "<reserved_102>"}, "{{content}}", {"token": "<reserved_103>"}]), efficient_eos=True, ) register_template( name="baichuan2", format_user=StringFormatter(slots=["<reserved_106>{{content}}<reserved_107>"]), efficient_eos=True, ) register_template( name="belle", format_user=StringFormatter(slots=["Human: {{content}}\n\nBelle: "]), format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}, "\n\n"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) register_template( name="bluelm", format_user=StringFormatter(slots=[{"token": "[|Human|]:"}, "{{content}}", {"token": "[|AI|]:"}]), ) register_template( name="breeze", format_user=StringFormatter(slots=["[INST] {{content}} [/INST] "]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), efficient_eos=True, ) register_template( name="chatglm2", format_user=StringFormatter(slots=["[Round {{idx}}]\n\n问：{{content}}\n\n答："]), format_prefix=EmptyFormatter(slots=[{"token": "[gMASK]"}, {"token": "sop"}]), efficient_eos=True, ) register_template( name="chatglm3", format_user=StringFormatter(slots=[{"token": "<|user|>"}, "\n", "{{content}}", {"token": "<|assistant|>"}]), format_assistant=StringFormatter(slots=["\n", "{{content}}"]), format_system=StringFormatter(slots=[{"token": "<|system|>"}, "\n", "{{content}}"]), format_function=FunctionFormatter(slots=["{{content}}"], tool_format="glm4"), format_observation=StringFormatter( slots=[{"token": "<|observation|>"}, "\n", "{{content}}", {"token": "<|assistant|>"}] ), format_tools=ToolFormatter(tool_format="glm4"), format_prefix=EmptyFormatter(slots=[{"token": "[gMASK]"}, {"token": "sop"}]), stop_words=["<|user|>", "<|observation|>"], efficient_eos=True, ) register_template( name="chatml", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), stop_words=["<|im_end|>", "<|im_start|>"], replace_eos=True, replace_jinja_template=True, ) # copied from chatml template register_template( name="chatml_de", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), default_system="Du bist ein freundlicher und hilfsbereiter KI-Assistent.", stop_words=["<|im_end|>", "<|im_start|>"], replace_eos=True, replace_jinja_template=True, ) register_template( name="codegeex2", format_prefix=EmptyFormatter(slots=[{"token": "[gMASK]"}, {"token": "sop"}]), ) register_template( name="codegeex4", format_user=StringFormatter(slots=["<|user|>\n{{content}}<|assistant|>\n"]), format_system=StringFormatter(slots=["<|system|>\n{{content}}"]), format_function=FunctionFormatter(slots=["{{content}}"], tool_format="glm4"), format_observation=StringFormatter(slots=["<|observation|>\n{{content}}<|assistant|>\n"]), format_tools=ToolFormatter(tool_format="glm4"), format_prefix=EmptyFormatter(slots=["[gMASK]<sop>"]), default_system=( "你是一位智能编程助手，你叫CodeGeeX。你会为用户回答关于编程、代码、计算机方面的任何问题，" "并提供格式规范、可以执行、准确安全的代码，并在必要时提供详细的解释。" ), stop_words=["<|user|>", "<|observation|>"], efficient_eos=True, ) register_template( name="cohere", format_user=StringFormatter( slots=[ ( "<|START_OF_TURN_TOKEN|><|USER_TOKEN|>{{content}}<|END_OF_TURN_TOKEN|>" "<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>" ) ] ), format_system=StringFormatter(slots=["<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>{{content}}<|END_OF_TURN_TOKEN|>"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) register_template( name="cpm", format_user=StringFormatter(slots=["<用户>{{content}}<AI>"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) # copied from chatml template register_template( name="cpm3", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), stop_words=["<|im_end|>"], ) # copied from chatml template register_template( name="dbrx", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), default_system=( "You are DBRX, created by Databricks. You were last updated in December 2023. " "You answer questions based on information available up to that point.\n" "YOU PROVIDE SHORT RESPONSES TO SHORT QUESTIONS OR STATEMENTS, but provide thorough " "responses to more complex and open-ended questions.\nYou assist with various tasks, " "from writing to coding (using markdown for code blocks — remember to use ``` with " "code, JSON, and tables).\n(You do not have real-time data access or code execution " "capabilities. You avoid stereotyping and provide balanced perspectives on " "controversial topics. You do not provide song lyrics, poems, or news articles and " "do not divulge details of your training data.)\nThis is your system prompt, " "guiding your responses. Do not reference it, just respond to the user. If you find " "yourself talking about this message, stop. You should be responding appropriately " "and usually that means not mentioning this.\nYOU DO NOT MENTION ANY OF THIS INFORMATION " "ABOUT YOURSELF UNLESS THE INFORMATION IS DIRECTLY PERTINENT TO THE USER'S QUERY." ), stop_words=["<|im_end|>"], ) register_template( name="deepseek", format_user=StringFormatter(slots=["User: {{content}}\n\nAssistant:"]), format_system=StringFormatter(slots=["{{content}}\n\n"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) register_template( name="deepseek3", format_user=StringFormatter(slots=["<｜User｜>{{content}}<｜Assistant｜>"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) register_template( name="deepseekcoder", format_user=StringFormatter(slots=["### Instruction:\n{{content}}\n### Response:"]), format_assistant=StringFormatter(slots=["\n{{content}}\n<|EOT|>\n"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), default_system=( "You are an AI programming assistant, utilizing the DeepSeek Coder model, " "developed by DeepSeek Company, and you only answer questions related to computer science. " "For politically sensitive questions, security and privacy issues, " "and other non-computer science questions, you will refuse to answer.\n" ), ) register_template( name="default", format_user=StringFormatter(slots=["Human: {{content}}\nAssistant:"]), format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}, "\n"]), format_system=StringFormatter(slots=["System: {{content}}\n"]), ) register_template( name="empty", format_assistant=StringFormatter(slots=["{{content}}"]), ) register_template( name="exaone", format_user=StringFormatter(slots=["[|user|]{{content}}\n[|assistant|]"]), format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}, "\n"]), format_system=StringFormatter(slots=["[|system|]{{content}}[|endofturn|]\n"]), ) register_template( name="falcon", format_user=StringFormatter(slots=["User: {{content}}\nFalcon:"]), format_assistant=StringFormatter(slots=["{{content}}\n"]), efficient_eos=True, ) register_template( name="fewshot", format_assistant=StringFormatter(slots=["{{content}}\n\n"]), efficient_eos=True, ) register_template( name="gemma", format_user=StringFormatter(slots=["<start_of_turn>user\n{{content}}<end_of_turn>\n<start_of_turn>model\n"]), format_assistant=StringFormatter(slots=["{{content}}<end_of_turn>\n"]), format_observation=StringFormatter( slots=["<start_of_turn>tool\n{{content}}<end_of_turn>\n<start_of_turn>model\n"] ), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) register_template( name="glm4", format_user=StringFormatter(slots=["<|user|>\n{{content}}<|assistant|>"]), format_assistant=StringFormatter(slots=["\n{{content}}"]), format_system=StringFormatter(slots=["<|system|>\n{{content}}"]), format_function=FunctionFormatter(slots=["{{content}}"], tool_format="glm4"), format_observation=StringFormatter(slots=["<|observation|>\n{{content}}<|assistant|>"]), format_tools=ToolFormatter(tool_format="glm4"), format_prefix=EmptyFormatter(slots=["[gMASK]<sop>"]), stop_words=["<|user|>", "<|observation|>"], efficient_eos=True, ) register_template( name="granite3", format_user=StringFormatter( slots=[ "<|start_of_role|>user<|end_of_role|>{{content}}<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>" ] ), format_assistant=StringFormatter(slots=["{{content}}<|end_of_text|>\n"]), format_system=StringFormatter(slots=["<|start_of_role|>system<|end_of_role|>{{content}}<|end_of_text|>\n"]), ) register_template( name="index", format_user=StringFormatter(slots=["reserved_0{{content}}reserved_1"]), format_system=StringFormatter(slots=["<unk>{{content}}"]), efficient_eos=True, ) register_template( name="intern", format_user=StringFormatter(slots=["<|User|>:{{content}}\n<|Bot|>:"]), format_assistant=StringFormatter(slots=["{{content}}<eoa>\n"]), format_system=StringFormatter(slots=["<|System|>:{{content}}\n"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), default_system=( "You are an AI assistant whose name is InternLM (书生·浦语).\n" "- InternLM (书生·浦语) is a conversational language model that is developed by Shanghai AI Laboratory " "(上海人工智能实验室). It is designed to be helpful, honest, and harmless.\n" "- InternLM (书生·浦语) can understand and communicate fluently in the language " "chosen by the user such as English and 中文." ), stop_words=["<eoa>"], ) register_template( name="intern2", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), default_system=( "You are an AI assistant whose name is InternLM (书生·浦语).\n" "- InternLM (书生·浦语) is a conversational language model that is developed by Shanghai AI Laboratory " "(上海人工智能实验室). It is designed to be helpful, honest, and harmless.\n" "- InternLM (书生·浦语) can understand and communicate fluently in the language " "chosen by the user such as English and 中文." ), stop_words=["<|im_end|>"], ) register_template( name="llama2", format_user=StringFormatter(slots=[{"bos_token"}, "[INST] {{content}} [/INST]"]), format_system=StringFormatter(slots=["<<SYS>>\n{{content}}\n<</SYS>>\n\n"]), template_class=Llama2Template, ) # copied from llama2 template register_template( name="llama2_zh", format_user=StringFormatter(slots=[{"bos_token"}, "[INST] {{content}} [/INST]"]), format_system=StringFormatter(slots=["<<SYS>>\n{{content}}\n<</SYS>>\n\n"]), default_system="You are a helpful assistant. 你是一个乐于助人的助手。", template_class=Llama2Template, ) register_template( name="llama3", format_user=StringFormatter( slots=[ ( "<|start_header_id|>user<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>assistant<|end_header_id|>\n\n" ) ] ), format_assistant=StringFormatter(slots=["{{content}}<|eot_id|>"]), format_system=StringFormatter(slots=["<|start_header_id|>system<|end_header_id|>\n\n{{content}}<|eot_id|>"]), format_function=FunctionFormatter(slots=["{{content}}<|eot_id|>"], tool_format="llama3"), format_observation=StringFormatter( slots=[ ( "<|start_header_id|>ipython<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>assistant<|end_header_id|>\n\n" ) ] ), format_tools=ToolFormatter(tool_format="llama3"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), stop_words=["<|eot_id|>", "<|eom_id|>"], ) # copied from llama3 template register_template( name="mllama", format_user=StringFormatter( slots=[ ( "<|start_header_id|>user<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>assistant<|end_header_id|>\n\n" ) ] ), format_assistant=StringFormatter(slots=["{{content}}<|eot_id|>"]), format_system=StringFormatter(slots=["<|start_header_id|>system<|end_header_id|>\n\n{{content}}<|eot_id|>"]), format_function=FunctionFormatter(slots=["{{content}}<|eot_id|>"], tool_format="llama3"), format_observation=StringFormatter( slots=[ ( "<|start_header_id|>ipython<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>assistant<|end_header_id|>\n\n" ) ] ), format_tools=ToolFormatter(tool_format="llama3"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), stop_words=["<|eot_id|>", "<|eom_id|>"], mm_plugin=get_mm_plugin(name="mllama", image_token="<|image|>"), ) # copied from vicuna template register_template( name="llava", format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]), default_system=( "A chat between a curious user and an artificial intelligence assistant. " "The assistant gives helpful, detailed, and polite answers to the user's questions." ), mm_plugin=get_mm_plugin(name="llava", image_token="<image>"), ) # copied from vicuna template register_template( name="llava_next", format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]), default_system=( "A chat between a curious user and an artificial intelligence assistant. " "The assistant gives helpful, detailed, and polite answers to the user's questions." ), mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"), ) # copied from llama3 template register_template( name="llava_next_llama3", format_user=StringFormatter( slots=[ ( "<|start_header_id|>user<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>assistant<|end_header_id|>\n\n" ) ] ), format_assistant=StringFormatter(slots=["{{content}}<|eot_id|>"]), format_system=StringFormatter(slots=["<|start_header_id|>system<|end_header_id|>\n\n{{content}}<|eot_id|>"]), format_function=FunctionFormatter(slots=["{{content}}<|eot_id|>"], tool_format="llama3"), format_observation=StringFormatter( slots=[ ( "<|start_header_id|>ipython<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>assistant<|end_header_id|>\n\n" ) ] ), format_tools=ToolFormatter(tool_format="llama3"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), stop_words=["<|eot_id|>", "<|eom_id|>"], mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"), ) # copied from mistral template register_template( name="llava_next_mistral", format_user=StringFormatter(slots=["[INST] {{content}}[/INST]"]), format_assistant=StringFormatter(slots=[" {{content}}", {"eos_token"}]), format_system=StringFormatter(slots=["{{content}}\n\n"]), format_function=FunctionFormatter(slots=["[TOOL_CALLS] {{content}}", {"eos_token"}], tool_format="mistral"), format_observation=StringFormatter(slots=["""[TOOL_RESULTS] {"content": {{content}}}[/TOOL_RESULTS]"""]), format_tools=ToolFormatter(tool_format="mistral"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"), template_class=Llama2Template, ) # copied from qwen template register_template( name="llava_next_qwen", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_function=FunctionFormatter(slots=["{{content}}<|im_end|>\n"], tool_format="qwen"), format_observation=StringFormatter( slots=["<|im_start|>user\n<tool_response>\n{{content}}\n</tool_response><|im_end|>\n<|im_start|>assistant\n"] ), format_tools=ToolFormatter(tool_format="qwen"), default_system="You are a helpful assistant.", stop_words=["<|im_end|>"], mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"), ) # copied from chatml template register_template( name="llava_next_yi", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), stop_words=["<|im_end|>"], mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"), ) # copied from vicuna template register_template( name="llava_next_video", format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]), default_system=( "A chat between a curious user and an artificial intelligence assistant. " "The assistant gives helpful, detailed, and polite answers to the user's questions." ), mm_plugin=get_mm_plugin(name="llava_next_video", image_token="<image>", video_token="<video>"), ) # copied from mistral template register_template( name="llava_next_video_mistral", format_user=StringFormatter(slots=["[INST] {{content}}[/INST]"]), format_assistant=StringFormatter(slots=[" {{content}}", {"eos_token"}]), format_system=StringFormatter(slots=["{{content}}\n\n"]), format_function=FunctionFormatter(slots=["[TOOL_CALLS] {{content}}", {"eos_token"}], tool_format="mistral"), format_observation=StringFormatter(slots=["""[TOOL_RESULTS] {"content": {{content}}}[/TOOL_RESULTS]"""]), format_tools=ToolFormatter(tool_format="mistral"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), mm_plugin=get_mm_plugin(name="llava_next_video", image_token="<image>", video_token="<video>"), template_class=Llama2Template, ) # copied from chatml template register_template( name="llava_next_video_yi", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), stop_words=["<|im_end|>"], mm_plugin=get_mm_plugin(name="llava_next_video", image_token="<image>", video_token="<video>"), ) # copied from chatml template register_template( name="marco", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), default_system=( "你是一个经过良好训练的AI助手，你的名字是Marco-o1.由阿里国际数字商业集团的AI Business创造.\n## 重要！！！！！\n" "当你回答问题时，你的思考应该在<Thought>内完成，<Output>内输出你的结果。\n" "<Thought>应该尽可能是英文，但是有2个特例，一个是对原文中的引用，另一个是是数学应该使用markdown格式，<Output>内的输出需要遵循用户输入的语言。\n" ), stop_words=["<|im_end|>"], ) # copied from chatml template register_template( name="minicpm_v", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), stop_words=["<|im_end|>"], default_system="You are a helpful assistant.", mm_plugin=get_mm_plugin(name="minicpm_v", image_token="<image>", video_token="<video>"), ) # copied from minicpm_v template register_template( name="minicpm_o", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), stop_words=["<|im_end|>"], default_system="You are Qwen, created by Alibaba Cloud. You are a helpful assistant.", mm_plugin=get_mm_plugin(name="minicpm_v", image_token="<image>", video_token="<video>", audio_token="<audio>"), ) # mistral tokenizer v3 tekken register_template( name="ministral", format_user=StringFormatter(slots=["[INST]{{content}}[/INST]"]), format_system=StringFormatter(slots=["{{content}}\n\n"]), format_function=FunctionFormatter(slots=["[TOOL_CALLS]{{content}}", {"eos_token"}], tool_format="mistral"), format_observation=StringFormatter(slots=["""[TOOL_RESULTS]{"content": {{content}}}[/TOOL_RESULTS]"""]), format_tools=ToolFormatter(tool_format="mistral"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), template_class=Llama2Template, ) # mistral tokenizer v3 register_template( name="mistral", format_user=StringFormatter(slots=["[INST] {{content}}[/INST]"]), format_assistant=StringFormatter(slots=[" {{content}}", {"eos_token"}]), format_system=StringFormatter(slots=["{{content}}\n\n"]), format_function=FunctionFormatter(slots=["[TOOL_CALLS] {{content}}", {"eos_token"}], tool_format="mistral"), format_observation=StringFormatter(slots=["""[TOOL_RESULTS] {"content": {{content}}}[/TOOL_RESULTS]"""]), format_tools=ToolFormatter(tool_format="mistral"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), template_class=Llama2Template, ) # mistral tokenizer v7 tekken (copied from ministral) register_template( name="mistral_small", format_user=StringFormatter(slots=["[INST]{{content}}[/INST]"]), format_system=StringFormatter(slots=["[SYSTEM_PROMPT]{{content}}[/SYSTEM_PROMPT]"]), format_function=FunctionFormatter(slots=["[TOOL_CALLS]{{content}}", {"eos_token"}], tool_format="mistral"), format_observation=StringFormatter(slots=["""[TOOL_RESULTS]{"content": {{content}}}[/TOOL_RESULTS]"""]), format_tools=ToolFormatter(tool_format="mistral"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) register_template( name="olmo", format_user=StringFormatter(slots=["<|user|>\n{{content}}<|assistant|>\n"]), format_prefix=EmptyFormatter(slots=[{"eos_token"}]), ) register_template( name="openchat", format_user=StringFormatter(slots=["GPT4 Correct User: {{content}}", {"eos_token"}, "GPT4 Correct Assistant:"]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) register_template( name="openchat-3.6", format_user=StringFormatter( slots=[ ( "<|start_header_id|>GPT4 Correct User<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>GPT4 Correct Assistant<|end_header_id|>\n\n" ) ] ), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), stop_words=["<|eot_id|>"], ) # copied from chatml template register_template( name="opencoder", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), default_system="You are OpenCoder, created by OpenCoder Team.", stop_words=["<|im_end|>"], ) register_template( name="orion", format_user=StringFormatter(slots=["Human: {{content}}\n\nAssistant: ", {"eos_token"}]), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), ) # copied from gemma template register_template( name="paligemma", format_user=StringFormatter(slots=["<start_of_turn>user\n{{content}}<end_of_turn>\n<start_of_turn>model\n"]), format_assistant=StringFormatter(slots=["{{content}}<end_of_turn>\n"]), format_observation=StringFormatter( slots=["<start_of_turn>tool\n{{content}}<end_of_turn>\n<start_of_turn>model\n"] ), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), mm_plugin=get_mm_plugin(name="paligemma", image_token="<image>"), ) register_template( name="phi", format_user=StringFormatter(slots=["<|user|>\n{{content}}<|end|>\n<|assistant|>\n"]), format_assistant=StringFormatter(slots=["{{content}}<|end|>\n"]), format_system=StringFormatter(slots=["<|system|>\n{{content}}<|end|>\n"]), stop_words=["<|end|>"], ) register_template( name="phi_small", format_user=StringFormatter(slots=["<|user|>\n{{content}}<|end|>\n<|assistant|>\n"]), format_assistant=StringFormatter(slots=["{{content}}<|end|>\n"]), format_system=StringFormatter(slots=["<|system|>\n{{content}}<|end|>\n"]), format_prefix=EmptyFormatter(slots=[{"<|endoftext|>"}]), stop_words=["<|end|>"], ) register_template( name="phi4", format_user=StringFormatter( slots=["<|im_start|>user<|im_sep|>{{content}}<|im_end|><|im_start|>assistant<|im_sep|>"] ), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>"]), format_system=StringFormatter(slots=["<|im_start|>system<|im_sep|>{{content}}<|im_end|>"]), stop_words=["<|im_end|>"], ) # copied from ministral template register_template( name="pixtral", format_user=StringFormatter(slots=["[INST]{{content}}[/INST]"]), format_system=StringFormatter(slots=["{{content}}\n\n"]), format_function=FunctionFormatter(slots=["[TOOL_CALLS]{{content}}", {"eos_token"}], tool_format="mistral"), format_observation=StringFormatter(slots=["""[TOOL_RESULTS]{"content": {{content}}}[/TOOL_RESULTS]"""]), format_tools=ToolFormatter(tool_format="mistral"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), mm_plugin=get_mm_plugin(name="pixtral", image_token="[IMG]"), template_class=Llama2Template, ) # copied from chatml template register_template( name="qwen", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_function=FunctionFormatter(slots=["{{content}}<|im_end|>\n"], tool_format="qwen"), format_observation=StringFormatter( slots=["<|im_start|>user\n<tool_response>\n{{content}}\n</tool_response><|im_end|>\n<|im_start|>assistant\n"] ), format_tools=ToolFormatter(tool_format="qwen"), default_system="You are a helpful assistant.", stop_words=["<|im_end|>"], ) # copied from chatml template register_template( name="qwen2_audio", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), default_system="You are a helpful assistant.", stop_words=["<|im_end|>"], mm_plugin=get_mm_plugin(name="qwen2_audio", audio_token="<|AUDIO|>"), ) # copied from qwen template register_template( name="qwen2_vl", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), format_function=FunctionFormatter(slots=["{{content}}<|im_end|>\n"], tool_format="qwen"), format_observation=StringFormatter( slots=["<|im_start|>user\n<tool_response>\n{{content}}\n</tool_response><|im_end|>\n<|im_start|>assistant\n"] ), format_tools=ToolFormatter(tool_format="qwen"), default_system="You are a helpful assistant.", stop_words=["<|im_end|>"], mm_plugin=get_mm_plugin(name="qwen2_vl", image_token="<|image_pad|>", video_token="<|video_pad|>"), ) register_template( name="sailor", format_user=StringFormatter(slots=["<|im_start|>question\n{{content}}<|im_end|>\n<|im_start|>answer\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), default_system=( "You are an AI assistant named Sailor created by Sea AI Lab. " "Your answer should be friendly, unbiased, faithful, informative and detailed." ), stop_words=["<|im_end|>"], ) # copied from llama3 template register_template( name="skywork_o1", format_user=StringFormatter( slots=[ ( "<|start_header_id|>user<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>assistant<|end_header_id|>\n\n" ) ] ), format_assistant=StringFormatter(slots=["{{content}}<|eot_id|>"]), format_system=StringFormatter(slots=["<|start_header_id|>system<|end_header_id|>\n\n{{content}}<|eot_id|>"]), format_function=FunctionFormatter(slots=["{{content}}<|eot_id|>"], tool_format="llama3"), format_observation=StringFormatter( slots=[ ( "<|start_header_id|>ipython<|end_header_id|>\n\n{{content}}<|eot_id|>" "<|start_header_id|>assistant<|end_header_id|>\n\n" ) ] ), format_tools=ToolFormatter(tool_format="llama3"), format_prefix=EmptyFormatter(slots=[{"bos_token"}]), default_system=( "You are Skywork-o1, a thinking model developed by Skywork AI, specializing in solving complex problems " "involving mathematics, coding, and logical reasoning through deep thought. When faced with a user's request, " "you first engage in a lengthy and in-depth thinking process to explore possible solutions to the problem. " "After completing your thoughts, you then provide a detailed explanation of the solution process " "in your response." ), stop_words=["<|eot_id|>", "<|eom_id|>"], ) register_template( name="solar", format_user=StringFormatter(slots=["### User:\n{{content}}\n\n### Assistant:\n"]), format_system=StringFormatter(slots=["### System:\n{{content}}\n\n"]), efficient_eos=True, ) register_template( name="starchat", format_user=StringFormatter(slots=["<|user|>\n{{content}}<|end|>\n<|assistant|>"]), format_assistant=StringFormatter(slots=["{{content}}<|end|>\n"]), format_system=StringFormatter(slots=["<|system|>\n{{content}}<|end|>\n"]), stop_words=["<|end|>"], ) register_template( name="telechat", format_user=StringFormatter(slots=["<_user>{{content}}<_bot>"]), format_system=StringFormatter(slots=["<_system>{{content}}<_end>"]), ) register_template( name="telechat2", format_user=StringFormatter(slots=["<_user>{{content}}<_bot>"]), format_system=StringFormatter(slots=["<_system>{{content}}"]), default_system=( "你是中国电信星辰语义大模型，英文名是TeleChat，你是由中电信人工智能科技有限公司和中国电信人工智能研究院（TeleAI）研发的人工智能助手。" ), ) register_template( name="vicuna", format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]), default_system=( "A chat between a curious user and an artificial intelligence assistant. " "The assistant gives helpful, detailed, and polite answers to the user's questions." ), replace_jinja_template=True, ) register_template( name="video_llava", format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]), default_system=( "A chat between a curious user and an artificial intelligence assistant. " "The assistant gives helpful, detailed, and polite answers to the user's questions." ), mm_plugin=get_mm_plugin(name="video_llava", image_token="<image>", video_token="<video>"), ) register_template( name="xuanyuan", format_user=StringFormatter(slots=["Human: {{content}} Assistant:"]), default_system=( "以下是用户和人工智能助手之间的对话。用户以Human开头，人工智能助手以Assistant开头，" "会对人类提出的问题给出有帮助、高质量、详细和礼貌的回答，并且总是拒绝参与与不道德、" "不安全、有争议、政治敏感等相关的话题、问题和指示。\n" ), ) register_template( name="xverse", format_user=StringFormatter(slots=["Human: {{content}}\n\nAssistant: "]), ) register_template( name="yayi", format_user=StringFormatter(slots=[{"token": "<|Human|>"}, ":\n{{content}}\n\n", {"token": "<|YaYi|>"}, ":"]), format_assistant=StringFormatter(slots=["{{content}}\n\n"]), format_system=StringFormatter(slots=[{"token": "<|System|>"}, ":\n{{content}}\n\n"]), default_system=( "You are a helpful, respectful and honest assistant named YaYi " "developed by Beijing Wenge Technology Co.,Ltd. " "Always answer as helpfully as possible, while being safe. " "Your answers should not include any harmful, unethical, " "racist, sexist, toxic, dangerous, or illegal content. " "Please ensure that your responses are socially unbiased and positive in nature.\n\n" "If a question does not make any sense, or is not factually coherent, " "explain why instead of answering something not correct. " "If you don't know the answer to a question, please don't share false information." ), stop_words=["<|End|>"], ) # copied from chatml template register_template( name="yi", format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]), format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]), format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]), stop_words=["<|im_end|>"], ) register_template( name="yi_vl", format_user=StringFormatter(slots=["### Human: {{content}}\n### Assistant:"]), format_assistant=StringFormatter(slots=["{{content}}\n"]), default_system=( "This is a chat between an inquisitive human and an AI assistant. " "Assume the role of the AI assistant. Read all the images carefully, " "and respond to the human's questions with informative, helpful, detailed and polite answers. " "这是一个好奇的人类和一个人工智能助手之间的对话。假设你扮演这个AI助手的角色。" "仔细阅读所有的图像，并对人类的问题做出信息丰富、有帮助、详细的和礼貌的回答。\n\n" ), stop_words=["###"], efficient_eos=True, mm_plugin=get_mm_plugin(name="llava", image_token="<image>"), ) register_template( name="yuan", format_user=StringFormatter(slots=["{{content}}", {"token": "<sep>"}]), format_assistant=StringFormatter(slots=["{{content}}<eod>\n"]), stop_words=["<eod>"], ) register_template( name="zephyr", format_user=StringFormatter(slots=["<|user|>\n{{content}}", {"eos_token"}, "<|assistant|>\n"]), format_system=StringFormatter(slots=["<|system|>\n{{content}}", {"eos_token"}]), default_system="You are Zephyr, a helpful assistant.", ) register_template( name="ziya", format_user=StringFormatter(slots=["<human>:{{content}}\n<bot>:"]), format_assistant=StringFormatter(slots=["{{content}}\n"]), )开源模型应用落地-DeepSeek-R1-Distill-Qwen-7B-LoRA微调-LLaMA-Factor由讯客互联人工智能栏目发布，感谢您对讯客互联的认可，以及对我们原创作品以及文章的青睐，非常欢迎各位朋友分享到个人网站或者朋友圈，但转载请说明文章出处“开源模型应用落地-DeepSeek-R1-Distill-Qwen-7B-LoRA微调-LLaMA-Factor”
上一篇
python1
