CSV to chat-format JSONL converter for fine-tuning datasets
A Python script that turns a spreadsheet export of prompt and ideal-answer pairs into chat-format JSONL (one messages array per line) for fine-tuning, with column mapping, a shared or per-row system message, dedupe, and atomic output.
File in Data by AISkillHub Library
What it does
Fine-tuning data usually starts in a spreadsheet: subject-matter experts write a question in one column and the ideal answer in another. Training tools want JSONL in the chat format, one example per line. This script does that conversion and refuses to produce a broken file.
**Output format**
```json {"messages": [{"role": "system", "content": "You answer billing questions for Acme."}, {"role": "user", "content": "Can I change my billing date?"}, {"role": "assistant", "content": "Yes. Go to Settings > Billing..."}]} ```
This is the chat fine-tuning format documented by OpenAI and accepted by many other training tools. Check your provider's documentation for any additional requirements such as minimum example counts.
**Features**
- Map any column names with `--user-col` and `--assistant-col`. - One system message for all rows (`--system "..."`) or a per-row column (`--system-col`). - Handles quoted multi-line cells, Excel's UTF-8 BOM (default encoding `utf-8-sig`), and other delimiters (`--delimiter ";"`). - Strips whitespace from cells by default (`--no-strip` to keep it). - `--dedupe` drops exact repeated examples, which otherwise get extra weight in training. - Rows with an empty user or assistant cell fail the run with row numbers, or are skipped with `--skip-invalid`. - Writes to a temp file and renames it at the end, so a failed run never leaves a half-written dataset.
**Example**
``` $ python3 scripts/csv_to_jsonl.py faq.csv train.jsonl --user-col question --assistant-col answer --system "You answer billing questions for Acme." --dedupe wrote 412 example(s) to train.jsonl (skipped 0 invalid, 3 duplicate) ```
**What it will not do**
- Build multi-turn conversations. Each row becomes one user message and one assistant reply. - Judge the quality of answers or split train and validation sets. - Remove personal data. Run a redaction step first if the sheet came from real tickets.
Exit codes: 0 written, 1 invalid rows or no valid rows, 2 usage error (missing file or column).
How to use it
1. Export your sheet as CSV (UTF-8). Make sure the first row has column headers. 2. Copy `scripts/csv_to_jsonl.py` into your project. 3. Convert: ``` python3 scripts/csv_to_jsonl.py data.csv train.jsonl --user-col question --assistant-col answer python3 scripts/csv_to_jsonl.py data.csv train.jsonl --user-col prompt --assistant-col ideal --system "Short system message" python3 scripts/csv_to_jsonl.py data.csv train.jsonl --user-col q --assistant-col a --system-col persona --dedupe --skip-invalid ``` 4. If the script reports invalid rows, fix them in the sheet and re-export rather than skipping them, unless you are sure they are junk. 5. Validate the result before uploading (the chat JSONL validator listing in this library does that).
Python 3.8+, no packages, no environment variables.
Files
Save each file at the path shown, relative to your project root.