GitHub Actions LLM Setup: A Workflow That Reviews PRs

A model call in GitHub Actions is an HTTP request with a bearer token, which is why the first version usually works within twenty minutes and the tenth version is still being tuned six weeks later. The request is trivial. What takes the time is the surrounding machinery: where the key lives, what happens when the provider is slow, and what stops the job from running on every commit to every branch forever.

This guide is the mechanical version — the YAML, the secrets, the knobs. The policy questions of what a model step should be allowed to do live in running LLMs in CI, and the fork security cliff described there is worth reading before you ship this anywhere public.

Repository secrets, and what they do not protect

The key goes in a repository secret, set under Settings, Secrets and variables, Actions. Not in the workflow file, not in a repository variable, not in a committed .env. Repository variables are readable by anyone who can read the repository; secrets are not, and they are redacted from logs.

Redaction is the part people over-trust. Actions masks the exact string of a secret in log output. It does not mask a base64-encoded version, a JSON-escaped version, or one that got split across a line break by a wrapping tool. Never echo the value, never pass it as a command-line argument where it lands in a process listing, and export it into the environment of the one step that needs it rather than the whole job.

If you run several repositories, an organisation secret scoped to the repositories that need it saves you from rotating in fifteen places. Combine that with one credential shared across tools — a single key across your tooling — and rotation becomes an afternoon rather than a project.

A minimal workflow that runs on pull requests

The smallest useful shape: trigger on pull requests, compute the diff, send it, write the answer to a file. Everything else in this article is a modification of this.

name: llm-review
on:
  pull_request:
    paths:
      - "src/**/*.ts"
      - "src/**/*.py"

permissions:
  contents: read
  pull-requests: write

concurrency:
  group: llm-review-${{ github.ref }}
  cancel-in-progress: true

jobs:
  review:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Build request
        run: |
          set -euo pipefail
          git diff --unified=0 "origin/$GITHUB_BASE_REF"...HEAD > diff.txt
          test -s diff.txt || { echo "empty diff"; exit 0; }
          jq -n --rawfile d diff.txt '{
            model: "kimi-k3",
            max_tokens: 800,
            messages: [
              {role: "system", content: "Review this diff. Findings only, no praise."},
              {role: "user", content: $d}
            ]
          }' > req.json

      - name: Call model
        env:
          LLM_KEY: ${{ secrets.LLM_API_KEY }}
        run: |
          set -euo pipefail
          curl -sS --fail-with-body --max-time 90 \
            -H "Authorization: Bearer $LLM_KEY" \
            -H "Content-Type: application/json" \
            -d @req.json \
            https://api.example.com/v1/chat/completions > resp.json
          jq -r '.choices[0].message.content' resp.json > review.md

Note --fail-with-body. Plain curl exits zero on an HTTP 429 or 500, so without it the job cheerfully proceeds to parse an error object and posts an empty comment. That silent-success pattern is the single most common bug in CI model steps.

Keep the output as a comment, not a status

Post the result as a pull request comment and let a human decide what it is worth. A comment is advisory by construction: it cannot block a merge, it does not need to be deterministic, and nobody has to override it.

      - name: Comment
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: gh pr comment "$PR_NUMBER" --body-file review.md

Use --body-file rather than interpolating the text into a command. The content was generated from a diff someone else wrote, so treat it as untrusted input: it can contain quotes, backslashes, shell metacharacters and anything else the diff author felt like including.

One comment per run accumulates into noise on a long-lived branch. Either find and update a previous comment identified by a marker string in its body, or delete the old one before posting. A pull request with nineteen stale review comments gets ignored entirely, which wastes the whole feature. The shape of a good summary is covered in building a PR summariser.

Two clocks: the job timeout and the request timeout

Set both. timeout-minutes on the job is your backstop against a hung runner; without it, GitHub applies a default measured in hours and you pay for a runner sitting on a dead socket. Five minutes is generous for a review step.

The job timeout alone is not enough, because a job that gets cancelled at the deadline gives you no diagnostic and may leave a request still running upstream. Set a client timeout too — --max-time for curl, an explicit timeout argument for an SDK. SDK defaults are frequently much longer than you want in CI and often include automatic retries, so one slow call can consume the whole job budget before you see any output. Tuning timeouts works through picking the number.

Make the step non-fatal while you are at it. continue-on-error: true on the model step means a provider outage produces a missing comment rather than a red pull request, which is the correct severity for advisory output.

Rate limits and what a matrix does to them

Rate limits arrive suddenly in CI because the traffic shape is bursty. Nothing happens for an hour, then someone merges a stack of twelve pull requests and a monorepo matrix fans out into forty concurrent jobs, all authenticating with the same key against the same quota.

The mitigations, in order of effectiveness. Reduce the fan-out first: run the model step once per pull request rather than once per matrix leg, because a language matrix does not need four independent reviews of the same diff. Then serialise what remains with a concurrency group. Then, and only then, add retry with backoff for the requests that still collide — and read the Retry-After header rather than guessing, as rate limits and retries explains.

Retries in CI deserve suspicion. A retry loop with no ceiling turns one upstream blip into hundreds of billable requests across a fan-out, and the bill arrives long after the incident. Cap attempts at two or three and let the step fail softly.

Path filters and concurrency are the cost controls

Two lines of YAML do most of the cost work. The paths filter on the trigger stops the workflow running at all for documentation-only changes, lockfile bumps and generated files. Mirror your lint ignores here; if a path is not worth linting it is not worth a model call.

The concurrency group with cancel-in-progress kills superseded runs. Someone who pushes four times in ten minutes while iterating should generate one review, not four, and without this you pay for three abandoned analyses of code that no longer exists.

Beyond that, cap the input. A pull request touching 900 files is exactly the one where a summary is least useful and most expensive; above a threshold, post a note saying the diff was too large and exit cleanly. Track the resulting spend as cost per merged pull request rather than as a monthly total, because that is the number that tells you whether the step is proportionate.

Pin the model and log what ran

Put an explicit model identifier in the workflow rather than a floating alias. When output quality changes, you want a commit to blame, and a pinned identifier makes an upgrade a reviewable pull request instead of a mystery.

Write the model name, the prompt version and the token counts from the response into the job summary. It costs one jq invocation and it is the only way to answer why the comments got worse last Tuesday, or which workflow is responsible for a spend spike.

If you also run this on self-hosted runners, remember they persist state between jobs. A key exported into a shell profile or a cached credential file on a self-hosted runner is readable by every subsequent job on that machine, including ones from other repositories.

The takeaway

Secret in repository secrets and exported to one step only. --fail-with-body so HTTP errors actually fail. Both clocks set, with the step marked non-fatal. Path filter on the trigger, concurrency group with cancellation, hard cap on diff size. Output as an updated comment rather than a status check, and a pinned model with token counts in the job summary. The same list transfers almost unchanged to GitLab CI and Jenkins, with different names for each knob.