Files

Chat JSONL dataset validator for fine-tuning and eval uploads

A Python script that checks a chat-format JSONL file line by line for structural errors (roles, ordering, empty content), then warns about duplicates, over-long examples and 'as an AI' boilerplate before you upload it.

File in Data by AISkillHub Library

What it does

Upload a malformed training file and you find out minutes or hours later from a terse error, or worse, the job runs and quietly learns from bad examples. This validator checks the file locally in seconds and tells you exactly which lines to fix.

**Errors (fail the file)**

- Line is not valid JSON, or has no non-empty `messages` list - A role outside the allowed set (default `system`, `user`, `assistant`; change with `--allow-roles`) - `system` anywhere but the first position - Empty content, or content that is neither a string nor a list of typed parts - No user message, no assistant message, or a conversation that does not end with the assistant - Two consecutive user or assistant messages - Fewer than `--min-examples` valid examples (default 10)

**Warnings (reported, do not fail)**

- Exact duplicate examples - Examples longer than `--max-tokens` by a characters-per-token estimate (approximate; check your provider's real limit) - Assistant replies containing "As an AI..." boilerplate you probably do not want to train into the model

**Example output**

``` ERROR line 18: messages[2] repeats role 'user' from the previous message ERROR line 18: last message must be from the assistant WARN line 240: duplicate of line 97

lines=500, valid_examples=499, errors=2, warnings=1, approx_tokens_total=183402, approx_tokens_max=2210, approx_tokens_mean=367.5 ```

`--json-report report.json` writes the full list for dashboards or CI artifacts.

**What it will not do**

It checks structure, not content quality or provider-specific rules such as tool-call fields or weight keys. If your provider accepts extra roles or fields, widen `--allow-roles` and review its documentation for the rest.

Exit codes: 0 valid, 1 errors found, 2 file not found.

How to use it

1. Copy `scripts/validate_chat_jsonl.py` into your project. 2. Run it on your dataset: ``` python3 scripts/validate_chat_jsonl.py train.jsonl python3 scripts/validate_chat_jsonl.py val.jsonl --min-examples 1 --max-tokens 8000 python3 scripts/validate_chat_jsonl.py train.jsonl --allow-roles system,user,assistant,tool --json-report report.json ``` 3. Fix every error and decide on each warning before uploading. 4. Optional CI step: `python3 scripts/validate_chat_jsonl.py data/train.jsonl` in any workflow that touches the dataset.

Python 3.8+, no packages, no environment variables.

Files

Save each file at the path shown, relative to your project root.

  • scripts/validate_chat_jsonl.py7.3 KB