CyphrQ
Menu

CI/CD for LLM Applications: How to Ship AI Features Safely

How to ship LLM features safely: version prompts and model IDs, gate releases on evals and red-team tests, secure the model supply chain, and roll back.

By CyphrQ.ai

Written with AI assistance and human editorial review. Every figure linked to its source.

11 min read

CI/CD for LLM applications means treating every prompt, model ID and model file as a release artifact. Each one is versioned in Git, tested by automated evaluations and red-team checks before deploy, pulled from a verified supply chain using short-lived credentials, and easy to roll back. The tests differ from ordinary software because the same input can produce different output.

What is CI/CD for LLM applications?

CI/CD (continuous integration and continuous delivery) is the automated pipeline that builds, tests and deploys every change. In an LLM application, the "code" includes things a normal pipeline never checks: the prompt text, the model you call, its settings, and sometimes model weights you download.

A one-word prompt edit can shift accuracy, tone or safety as much as a code change. So each one should pass the same gate: review, test, approve, deploy, and a way back.

This post covers the release pipeline. For risks once the feature is live and reading untrusted content, see prompt injection explained and AI agent guardrails. Related posts sit in the AI Security pillar.

Why do LLM features break ordinary release testing?

A unit test expects the same output for the same input. An LLM doesn't guarantee that, and a provider can change the model behind a name.

So the test question changes. Instead of "is the output exactly X?", you ask "does the output meet the criteria, often enough?" LangChain's LangSmith documentation puts the distinction plainly: "Testing asserts correctness. A system can only be deployed if it passes all tests," and "Evaluation metrics can be converted into tests." (LangSmith docs)

Without a pipeline, someone edits a prompt in a dashboard and nobody can say which version answered a customer last Tuesday.

How do you version prompts and model IDs?

Treat the prompt as source code. Keep it in the repository, change it through a pull request, and review the diff.

Then pin everything that shapes the answer into one versioned release:

Part of the release Pin it as Why
Prompt templates A file in Git, changed by pull request You can diff, review and revert it
Model The most specific version ID your provider offers, not a moving alias An alias can point to a different model later
Parameters Temperature, token limits, enabled tools, in config They change behaviour as much as the prompt
Evaluation dataset A named version A score change then means the app changed, not the test
Downloaded weights A file hash and a fixed repository revision The file you tested is the file you ship

The model row matters most. An alias is a short name that follows the latest release. With the versioned ID in config, a model upgrade becomes a pull request with evaluation results attached, not something that happens to you.

Log the release version with every response. When a complaint arrives, you can name the exact prompt, model and settings behind it.

How do automated LLM evaluations work in a pipeline?

An evaluation runs your application against a fixed set of test cases and scores the answers. In a pipeline, a score below your threshold fails the build, like a failing unit test. Scores come from code checks (is the output valid JSON?), from a second model acting as judge, or from human reviewers grading a sample.

Three tools document this. They appear alphabetically, described from their own documentation as checked on 25 October 2026, not ranked.

LangSmith (LangChain) separates offline evaluation, on "curated test cases with reference outputs" for uses including regression testing, from online evaluation on "real production traces without reference outputs". Evaluations "can be written using standard testing tools like pytest or Vitest/Jest", and "regression tests can assert that new versions must outperform baseline versions on relevant metrics." (LangSmith docs)

MLflow's GenAI evaluation guide calls mlflow.genai.evaluate with a dataset, a prediction function and a list of scorers. It offers built-in LLM-as-a-judge scorers such as Correctness and custom judges such as a Guidelines check. Evaluation Datasets serve as "a centralized repository for managing test cases." (MLflow docs)

promptfoo is a command-line tool. Its CI/CD guide runs npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json, writes JSON, HTML or JUnit XML results, and offers --fail-on-error, which fails "on any test failures." Its examples also compute a pass rate in shell and fail the job below 95%. (promptfoo docs)

Whichever you use, turn every bug report into a test case, and set the pass threshold from a baseline run. LLM judges make mistakes too, so spot-check their grades by hand.

Should red-team tests block an LLM release?

For anything customer-facing, or anything that can take actions, yes. Red-teaming here means automated probes that try to make the application leak data, ignore its instructions or produce harmful content. The goal is to catch a release that is worse than the last one, not to prove it is safe.

promptfoo's guide shows one approach: a GitHub Actions step using promptfoo/promptfoo-action@v1 with type: 'redteam', or the promptfoo redteam run command. Its example selects plugins such as harmful, pii and contracts, and runs on a daily schedule. (promptfoo docs)

A workable split: a fast, fixed suite on every pull request that blocks the merge if results drop below the baseline; a larger scheduled scan, triaged by a person, whose confirmed findings become fixed tests; and the full suite before any model upgrade, because a new model can fail differently on the same prompt. Passing scores don't replace design limits on what the feature can read and do.

How do you secure the model supply chain for an LLM app?

If you only call a hosted model through an API, your supply chain is the provider plus your libraries. If you download weights, the model file itself is software you run.

File format matters. Hugging Face's security documentation says pickle is "the default format for PyTorch model weights", and that "de-serializing pickle means that code can be executed." (Hugging Face docs) Loading an untrusted pickle-based model is like running an untrusted program. The same page lists safetensors, a "simple serialization format for weights", among the alternatives.

Scan before you load. Hugging Face scans every file pushed to the Hub with ClamAV and a pickle import scan, but warns this "is not 100% foolproof. It is your responsibility as a user to check if something is safe or not." (Hugging Face docs) Run your own scan in the pipeline as well.

Pin what you approved. Record each approved file's hash and repository revision, and verify both at build time. A branch name such as main can move after you tested it. A hash can't.

Write it down in an ML-BOM. A software bill of materials (SBOM) lists the components inside software. CycloneDX, standardised as ECMA-424, extends this to machine learning: its ML-BOM "represents datasets, models, and configurations for AI and machine learning systems." (CycloneDX) When an advisory names a model or dataset, the ML-BOM tells you in minutes whether you use it.

How should AI pipelines handle secrets and tokens?

AI pipelines often hold more valuable keys than ordinary builds: a paid model API key, a cloud key, a write token for a model hub. Two GitHub Actions features cut what a leaked pipeline gives away.

OIDC instead of stored keys. With OpenID Connect, the job proves its identity to your cloud provider and receives a short-lived token. GitHub's docs: "You won't need to duplicate your cloud credentials as long-lived GitHub secrets," and the token "is only valid for a single job, and then automatically expires." (GitHub docs) For Hugging Face's equivalent, Trusted Publishers, see our Hugging Face Spaces case file.

Environments with protection rules. A GitHub environment, such as production, holds its own secrets and can require approval before a job receives them. "Secrets stored in an environment are only available to workflow jobs that reference the environment," and a job awaiting approval can't read them until a reviewer approves. (GitHub docs)

Put the production model key and deploy credentials in that protected environment. Pull-request checks and evaluation runs get a separate, lower-privilege key with a spending limit.

How do you roll back an LLM feature?

If the release is one versioned bundle of prompt, model ID, settings and evaluation dataset, rollback means deploying the previous bundle. If those parts live in different places, rollback becomes a hunt.

  • Keep the previous release deployable, including access to its model version, until the new one has run cleanly in production.
  • Put new prompts or models behind a feature flag or percentage rollout, so you can switch back without a deploy.
  • Agree in advance which evaluation or feedback signal triggers a rollback.
  • Ask your provider when your pinned model version will be retired, and plan the upgrade before that date.

Going deeper: for security and IT teams

Threat model. Three attacker positions matter: a poisoned artifact (a model file or dependency that runs code when loaded), a compromised pipeline component (a third-party action that reads the runner's secrets), and a stolen long-lived credential (a model API key or hub write token usable until rotated). Prompts and evaluation datasets need the same branch protection and code owners as source, because write access to them changes production behaviour without touching code.

In practice: malicious models on a public hub. JFrog reported on 27 February 2024 that it had "pinpointed around 100 instances" of Hugging Face models with real harmful payloads, excluding scanner false positives. "PyTorch models exhibit the highest prevalence, followed closely by Tensorflow Keras models." The model it analysed in detail opened a reverse shell to an external IP address when loaded through pickle deserialisation, and JFrog noted that Hugging Face's scanning marked such models unsafe rather than blocking them (JFrog). Control: admit only weight-only formats such as safetensors into production builds by policy, scan anything else in an isolated job with no secrets and no outbound network, and fail the build on any unexpected pickle import.

In practice: a compromised GitHub Action. CISA's alert of 18 March 2025, revised 26 March, says the third-party action tj-actions/changed-files (CVE-2025-30066) was compromised, allowing "information disclosure of secrets including, but not limited to, valid access keys", plus GitHub personal access tokens, npm tokens and private RSA keys. It was potentially enabled by a compromise of reviewdog/action-setup@v1 (CVE-2025-30154). CISA advised auditing workflows run between 2025-03-12 00:00 UTC and 2025-03-15 12:00 UTC and said to "Rotate all identified secrets immediately." Both CVEs went into the KEV catalog, and the fix was v46.0.1 (CISA). Reference third-party actions by full commit SHA rather than a version tag, which is a movable pointer, and keep high-value keys out of jobs that run third-party code.

OIDC trust conditions. GitHub's example token subject is repo:octo-org/octo-repo:environment:prod. Bind your cloud role's trust policy to the environment, not only the repository, so only jobs that passed the environment's protection rules can assume the production role. GitHub notes that repositories created after 15 July 2026 use "an immutable default subject format that includes owner and repository IDs", so check that trust policies match the format your repositories actually emit (GitHub docs).

Environment settings to verify. Per GitHub's reference: up to six required reviewers, with one approval enough; "prevent self-review", so whoever triggers a deploy can't approve it; a wait timer of 1 to 43,200 minutes; deployment branch and tag restrictions; at most six protection rules per environment; and administrator bypass, allowed by default and switchable off per environment. On GitHub Free, Pro or Team, required reviewers and wait timers work only in public repositories, and environment secrets in private repositories need Pro, Team or Enterprise (GitHub docs).

Eval tooling hygiene. promptfoo's example uses npx promptfoo@latest. In a release gate, pin the tool version like any dependency, so a tool update can't move your pass rate. The guide documents PROMPTFOO_STRIP_RESPONSE_OUTPUT=true and PROMPTFOO_STRIP_TEST_VARS=true to keep outputs and test variables out of shared results (promptfoo docs).

Evidence an auditor can test. For any production release, produce the commit holding prompt, model ID and parameters; the evaluation and red-team results for that commit; its ML-BOM; and the deployment record showing who approved it. If one is missing, the pipeline isn't yet a control.

Checklist: before an LLM feature ships

  • Prompts live in Git and change only through reviewed pull requests.
  • Config names a versioned model ID, not a moving alias, with parameters pinned alongside.
  • An evaluation suite runs on every change and fails the build below a chosen threshold.
  • A fixed red-team suite runs on every pull request, and a larger scan runs on a schedule.
  • Downloaded models use a weight-only format or are scanned, pinned by hash, and listed in an ML-BOM.
  • Production keys sit in a protected environment, and CI uses OIDC or a low-privilege, spend-capped key.
  • The previous release can be redeployed in minutes.

Key takeaways

  • Prompts, model IDs, parameters and model files are release artifacts. Version them together.
  • Pickle-based model files can run code when loaded, so treat a downloaded model as software you execute.
  • Short-lived OIDC tokens and protected environments mean a compromised pipeline step can't walk away with production keys.

Sources