# Template-free prompt construction Template-free prompt construction with the **`input_output`** format ### Background #### Masking Inputs One of the most popular features of [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) is setting the following configuration value: ```bash train_on_inputs: false ``` If you declare a [dataset formats](https://github.com/OpenAccess-AI-Collective/axolotl?tab=readme-ov-file#dataset) such as **`alpaca`** or **`chatml`**, axolotl knows what is an input (i.e. human) vs. an output (i.e. the assistant) and masks the input labels so that your model can focus on predicting the outputs only. ### You may not want prompt templates However, there are many situations where you don’t want to use one of these formats or templates. This is because they can: * Add unnecessary boilerplate to your prompts. * Create artifacts like special delimiters `<|im_start|>` that can quickly become footguns if you don’t include them correctly at inference time. * Enforce a *chat* interface when you do not want one. Sometimes you just want to fine-tune a model to a very specific task and do NOT want multi-turn conversations, roles, etc. * Limit you to only certain roles that the template allows. #### The `input_output` format You can construct your prompts without a template by using the `input_output` format, by setting `type: input_output` in your configuration file like this: #### **config.yml** ```bash train_on_inputs: false # Mask segments of your data datasets: - path: output.jsonl type: input_output # use template free prompt construction ``` Unlike **`type: completion`**, which is also template-free, **`type: input_output`** allows you to mask segments of your text. More details on how this works are described below. ## Usage This is how you can use the **`input_output`** format: #### Prepare Data To use the `input_output` format, collect your data in the following format into a **jsonl file** (below is the first row from the file `output`.jsonl\` pretty printed): ```bash $ head -n1 output.jsonl | python -m json.tool {.cell-output .cell-output-stdout} { "segments": [ { "label": true, "text": "Hello\n" }, { "label": true, "text": "hi there!. " }, { "label": false, "text": "goodbye " }, { "label": true, "text": "farewell" } ] } ``` Set **`label:false`** when you want to mask a segment of text so that the model isn’t trained on it. Some things to keep in mind: {% hint style="warning" %} **EOS, BOS, spaces, newlines etc. are entirely up to you.** Axolotl concatenates all the segments as-is. The tokenizer doesn’t add anything additional. Notice how I added spaces, newlines, `` (BOS), and `` (EOS) myself. Make sure you check the materialised output to validate that the prompt is getting assembled how you like. {% endhint %} #### Use `type:`` ``input_output` Let’s materialise data with our `output.jsonl` file by setting `type: input_output` in our axolotl config: ```bash # training_config.yaml base_model: mistralai/Mistral-7B-v0.1 data_seed: 49 seed: 49 datasets: - path: output.jsonl type: input_output val_set_size: 0.1 sequence_len: 896 sample_packing: false micro_batch_size: 2 gradient_accumulation_steps: 3 eval_batch_size: 2 num_epochs: 1 learning_rate: 0.0002 train_on_inputs: false special_tokens: bos_token: "" eos_token: "" unk_token: "" ``` You can use the following command to materialise your data. The **`--debug`** flag will print the tokens, along with the labels so you can verify that the correct items are being ignored: {% code overflow="wrap" %} ```bash $ python -m axolotl.cli.preprocess training_config.yaml --debug [2024-03-05 23:36:46,969] [INFO] [axolotl.check_example_labels:35] [PID:607731] [RANK:0] (1, 1) Hello(22557, 22557) (13, 13) hi(12014, 12014) there(736, 736) !(28808, 28808) .(28723, 28723) (28705, 28705) good(-100, 1179) bye(-100, 17664) (-100, 28705) fare(19111, 19111) well(5458, 5458) (2, 2) ``` {% endcode %} The format is **`decoded_token`****(****`label`****,**** ****`token_id`****)**, for example, **`(1, 1)`** means that the token is **``**, the label is **`1`** and the token\_id is **`1`**. When the label is **`-100`** then that token is ignored for training. #### Check the prompts Here is another way to check the materialised output: ```bash from transformers import AutoTokenizer from datasets import load_from_disk import yaml directory = !ls last_run_prepared/ with open('training_config.yaml', 'r') as f: cfg = yaml.safe_load(f) model_id = cfg['base_model'] tok = AutoTokenizer.from_pretrained(model_id) ds = load_from_disk(f'last_run_prepared/{directory[0]}/') ``` ``` >>> row = ds[0] >>> print(tok.decode(row['input_ids'])) Hello hi there!. goodbye farewell ``` We can check that the right tokens are ignored by comparing the labels to each token: ```bash import pandas as pd pd.DataFrame([{'token': tok.decode(i), 'label': l, 'id':i} for i,l in zip(row['input_ids'], row['labels'])]) ```
tokenlabelid
0<s>1
1Hello22557
213
3hi12014
4there736
5!28808
6.28723
728705
8good-100
9bye-100
10-100
11fare19111
12well5458
13</s>2
If we look at the input data, the above table seems correct! The jsonl version is repeated below for reference: ```bash $ head -n1 output.jsonl | python -m json.tool {.cell-output .cell-output-stdout} { "segments": [ { "label": true, "text": "Hello\n" }, { "label": true, "text": "hi there!. " }, { "label": false, "text": "goodbye " }, { "label": true, "text": "farewell" } ] } ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://axolotl.continuumlabs.pro/download-the-dataset/template-free-prompt-construction.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.