Auto-label pull requests that change prompts, evals or agent config
An actions/labeler v5 configuration and workflow that labels PRs touching prompts, eval sets, training data, or AI assistant config files such as CLAUDE.md, AGENTS.md, .mcp.json and .cursorrules.
Automation in Engineering by AISkillHub Library
What it does
Changes to a prompt or to an agent's instruction file can change product behavior as much as a code change, but they are easy to miss in a large PR. Labels make them visible in the PR list, help the right reviewer spot them, and let other workflows trigger only on relevant PRs (for example, running evals only when the `prompts` or `evals` label is present).
**Labels this config applies**
| Label | Applied when the PR changes | |---|---| | `prompts` | `prompts/**`, `*.prompt.md`, `*.prompt.txt` | | `evals` | `evals/**`, `*.eval.jsonl`, `*.eval.yaml` | | `ai-agent-config` | `CLAUDE.md`, `AGENTS.md`, `.claude/**`, `.mcp.json`, `.cursorrules`, `.cursor/**`, `.github/copilot-instructions.md`, `.github/instructions/**` | | `training-data` | `data/**/*.jsonl`, `datasets/**`, `fine-tune/**` |
`sync-labels: true` removes a label again if a later push no longer touches matching files.
**Security note**
The workflow uses `pull_request_target` so it can label PRs from forks. That event runs with write permissions in the context of your base branch, which is safe here only because the workflow never checks out or runs code from the PR. Do not add a checkout step or scripts to this workflow.
**What it will not do**
- It does not review the content of the change; it only reads the list of changed paths. - It does not set label colors or descriptions. Create the labels in the repository first (Issues > Labels) if you want to choose those. - It uses the v5 configuration format. The older v4 format (plain lists of globs under each label) does not work with `actions/labeler@v5`.
How to use it
1. Copy `.github/labeler.yml` and `.github/workflows/label-ai-changes.yml` into your repository. 2. Optional: create the labels `prompts`, `evals`, `ai-agent-config` and `training-data` under Issues > Labels to set colors and descriptions. 3. Adjust the globs in `.github/labeler.yml` to match your folder names. Each top-level key is a label name; add or remove labels freely. 4. Merge to the default branch. `pull_request_target` workflows run from the default branch's copy of the workflow, so the labeler starts working on PRs opened after the merge. 5. To run another workflow only for labeled PRs, add a condition such as `if: contains(github.event.pull_request.labels.*.name, 'prompts')` to that workflow's job.
No secrets are needed; the workflow uses the built-in `GITHUB_TOKEN` with `pull-requests: write`.
Files
Save each file at the path shown, relative to your project root.