GitHub Actions check that JSON, JSONL and YAML files parse
A CI workflow and a small Python script that fail a pull request when any JSON, JSONL or YAML file in the repo is malformed or a JSON object repeats a key. Built for repos full of prompt configs, eval sets and agent settings.
Automation in Engineering by AISkillHub Library
What it does
AI projects accumulate config files that nothing parses until runtime: eval sets in JSONL, tool schemas in JSON, agent and pipeline settings in YAML. A stray comma in line 4,000 of a dataset or a duplicated key in a schema usually shows up as a confusing failure much later. This listing catches those on the pull request.
**What it does**
- Walks the repository (skipping `.git`, `node_modules`, virtualenvs, `dist` and `build`) and checks every `.json`, `.jsonl`, `.yml` and `.yaml` file. - JSON: must parse, and **no object may repeat a key**. Python's `json` module silently keeps the last duplicate, which hides real mistakes in schemas and settings. - JSONL: every non-blank line must parse on its own, with the line number reported. - YAML: every document in the file must load with `yaml.safe_load_all`. - Inside GitHub Actions, errors are also printed as annotations, so they appear on the changed line in the PR.
**What it will not do**
- It does not validate files against a JSON Schema. It only checks that they parse. - It does not detect duplicate keys in YAML; PyYAML accepts them. - YAML that uses custom tags (CloudFormation `!Ref`, Ansible `!vault`) fails `safe_load`. Exclude those paths with `--exclude`. - Helm templates and other files with templating syntax are not YAML until rendered; exclude them too.
**Example output**
``` FAIL evals/support.jsonl:212: Expecting ',' delimiter (column 87) FAIL schemas/tool.json: duplicate key 'description' checked 41 file(s), 2 failed ```
Exit codes: 0 all valid, 1 at least one file failed, 2 usage error or PyYAML missing when YAML files exist.
How to use it
1. Copy `scripts/validate_config.py` and `.github/workflows/validate-config.yml` into your repository at the same paths. 2. Commit and open a pull request. The workflow runs on PRs and pushes to `main` that touch JSON, JSONL or YAML files. 3. If your default branch is not `main`, change `branches: [main]` in the workflow. 4. Exclude generated or templated files by editing the last step, for example: `python scripts/validate_config.py --exclude "charts/*/templates/**" --exclude "fixtures/invalid/**"` 5. Run it locally before pushing: ``` pip install pyyaml python3 scripts/validate_config.py # whole repo python3 scripts/validate_config.py evals/ # one folder ``` No secrets or environment variables are needed.
Files
Save each file at the path shown, relative to your project root.