By Marvin Tutt, Chief Executive Officer, Caia Tech

The Monday demo was slick. By Friday, a customer asked why the AI denied their claim last quarter—and no one could say exactly which prompt, model version, or policy the system used that day. Logs showed what happened, not why. That gap is the audit problem in AI. The fix is simpler than most “MLOps” decks suggest: use Git as the source of truth for anything that changes AI behavior, and anchor runtime decisions to that history.

Thesis in one line

AI behavior changes outside of code (prompts, configs, models, data). Git’s cryptographic, distributed history—paired with a few conventions—gives you a tamper‑evident, reproducible audit trail without buying another platform.

The problem no one is auditing

Behavior shifts daily without a code change: prompt edits, routing rules, temperature tweaks, RAG corpus updates, tool/plugin versions, provider-side model updates.

Traditional logs answer “what happened,” not “why the system decided that then.”

Regulators, customers, and incident postmortems need: who changed what, when, with whose approval; what state the system ran; and the ability to replay or at least explain the decision.

Why Git fits the audit job

Cryptographic history: Git’s Merkle DAG and content-addressed objects make tampering detectable. Rewrite history and the hashes betray you.

Distributed trust: everyone has a full copy; there’s no single database to quietly edit after the fact.

Governance built in: signed commits and tags, protected branches, CODEOWNERS, required checks, review workflows.

Cheap and ubiquitous: it’s already in your deployment pipeline and team skill set; it plays well with CI and object storage.

Reference design you can copy

System state in Git: prompts, configs, routing, model references, RAG corpus manifests, index build params, tool schemas, evals, risk policies, CI workflows, audit runbooks.

Bulky/PII outside Git: store models, vector indexes, and decision logs in immutable object storage (for example, S3 with Object Lock). In Git, keep small signed manifests that reference those objects by hash.

CI as gatekeeper: any change to prompts/configs/routing/models runs a golden‑set evaluation. If metrics or cost regress beyond thresholds, the PR fails. On merge, tag and sign a release; emit a provenance attestation.

Runtime traces: every decision logs the exact Git commit and content hashes for the prompt/config/policy, model identifier/digest, RAG snapshot and retrieved doc hashes, tool call hashes, decoding params, input fingerprint, and output hash. Write to append‑only storage with retention.

Repository layout (skeletal)

prompts/: human-readable prompt files grouped by domain.
configs/: model routing, temperatures, tool schemas, policy rules (JSON/YAML).
model_refs/: pointers to exact model artifacts or provider versions; include digests of weights if local.
data_manifests/: RAG corpus and index manifests (doc IDs, content hashes, source URIs, licenses, ingestion timestamps, embedding model version, index build params).
evals/: golden sets, thresholds, last‑run reports tied to commits.
policies/: redaction rules, safety filters, risk rubrics, commit policies.
audits/: daily manifest files that list immutable decision-log object keys and their digests (not the raw logs).
ci/: pre‑commit hooks, validation scripts, CI workflows.
docs/: runbooks for incident response, audit replay instructions, compliance mapping.

Version everything that changes behavior

Prompts and prompt templates.
Routing and decoding parameters (temperature, top_p, max_tokens, seeds).
Tool schemas and versions the agent can call.
Model identifiers and digests (or provider/model/version for hosted models).
RAG data: corpus manifest, embedding model and version, index build params.
Environment: container image digest, dependency lockfiles, CUDA/BLAS versions for local inference.

Decision trace schema (append‑only, PII‑aware)

Minimum fields per request:

ts, request_id, tenant/service actor.
git_commit and git_tag.
prompt_sha, config_sha, policy_sha.
model_id and model_digest (or provider/model/version).
decoding params: temperature, top_p, max_tokens, seed.
RAG: index_id, embedding_model_digest, k, retrieved doc IDs with content hashes and scores.
tool calls: tool name, version, schema_hash, args_hash, result_hash.
input_fingerprint (salted hash) and redaction_policy.
output_hash, decision/label, confidence, human_reviewed flag and reviewer pseudonym.

Abridged NDJSON example (single line)

{
  "ts":"2025-08-21T14:35:12Z",
  "request_id":"req_ab12cd34",
  "git_commit":"8f2a1d9",
  "git_tag":"release-2025-08-21",
  "prompt_sha":"sha256:3b7f...",
  "config_sha":"sha256:9ac2...",
  "policy_sha":"sha256:77aa...",
  "model_id":"llama3-8b-instruct",
  "model_digest":"sha256:1f94...",
  "temperature":0.2,
  "seed":424242,
  "rag":{
    "index_id":"cust_support_v23",
    "embed_model":"bge-small-en-v1",
    "k":4,
    "docs":[
      ["doc_981","sha256:7a2c...",0.84],
      ["doc_774","sha256:19cd...",0.81]
    ]
  },
  "tools":[
    ["sql_query","v1.3","sha256:5d1a...","sha256:3e9b...","sha256:77aa..."]
  ],
  "input_fp":"fp:salted:sha256:0b5e...",
  "output_hash":"sha256:f6a1...",
  "decision":"deny_claim",
  "confidence":0.88,
  "human_reviewed":false
}

Minimal implementation sketch (Python-ish)

At request time: read Git commit, compute file SHAs, fingerprint input, call model, hash output, write trace to object storage with Object Lock, append object key to a daily manifest committed to Git and timestamped.

import hashlib, json, os, subprocess
from datetime import datetime

def sha256_bytes(b): 
    return hashlib.sha256(b).hexdigest()

def file_sha(path): 
    return sha256_bytes(open(path,'rb').read())

def git_commit():
    return subprocess.check_output(["git","rev-parse","--short","HEAD"]).decode().strip()

def git_tag():
    try: 
        return subprocess.check_output(["git","describe","--tags","--abbrev=0"]).decode().strip()
    except: 
        return None

def fingerprint(text, salt=os.environ.get("AUDIT_SALT","")):
    return "fp:salted:sha256:" + sha256_bytes((salt+text).encode())

def log_to_object_store(bucket, key, blob_bytes):
    # Replace with your SDK and enable object lock/retention on the bucket
    pass

def log_decision(record, bucket):
    key = f"audits/{datetime.utcnow().date()}/{record['request_id']}.json"
    blob = json.dumps(record).encode()
    log_to_object_store(bucket, key, blob)
    
    with open("audits/daily_manifest.txt","a") as f:
        f.write(key+" "+sha256_bytes(blob)+"\n")
    
    subprocess.check_call(["git","add","audits/daily_manifest.txt"])
    subprocess.check_call(["git","commit","-m",f"Audit manifest {datetime.utcnow().date()} adds {record['request_id']}"])
    
    return key

Reproducibility that matters

Bitwise reproducibility: identical outputs bit‑for‑bit. Feasible for your own models when decoding deterministically (for example, temperature=0) on pinned containers/hardware, though low‑level non‑determinism in some GPU kernels can still make this a challenge. For strict replay, prefer CPU inference or deterministic modes and pinned libraries.

Behavioral reproducibility: same decision/label or materially equivalent output. Usually sufficient for audits and customer trust.

Record and pin: decoding params and seed, container image digest, dependency lockfiles, model digest or provider/model/version, and the RAG snapshot. For closed APIs, run daily shadow tests on a golden set to detect provider drift and alert on deltas.

Governance and integrity

Require signed commits and tags on protected branches; reject unsigned merges.
CODEOWNERS for prompts, policies, and model_refs; require approvals from owners.
Required status checks in CI: schema validation, secrets scan, golden‑set eval, cost budget checks.
Disallow force‑push and direct commits to main; enforce linear history.
External timestamps: anchor daily manifests and release tags using OpenTimestamps or RFC 3161 to prove when things existed.
PII discipline: redact at the edge before calling models; log fingerprints instead of raw inputs; encrypt sensitive trace fields in storage; segregate keys; never push PII into Git history.

Operational discipline (no out‑of‑band changes)

Rule one: all AI behavior changes—prompts, configs, routing, model refs—flow through Git with signed, reviewed PRs. No out‑of‑band edits.

Break‑glass path for emergencies:

Who: 2–3 on‑call owners with a time‑bound role.
How: perform the minimum change; log an EmergencyChange record (who/why/when) to immutable storage; reproduce via PR within 24 hours linking the EmergencyChange ID.
Detect: run a drift detector that hashes live config/runtime params in prod and compares to the deployed signed tag; alert on mismatch.

Manage commit noise:

Squash‑merge iterative prompt PRs so main reflects approved changes.
Bundle small prompt edits into weekly tagged releases unless a hotfix is needed.
Use conventional commits (prompt:, policy:, config:, model:) and auto‑generate a human‑readable CHANGELOG.
Track two org metrics: break‑glass count (target near zero) and mean audit retrieval time (from request_id to report).

RAG and tool auditing specifics

Corpus manifest: per document ID, content hash, source URI, license, ingestion timestamp. Keep index build params and embedding model version/digest.

Per‑request retrieval: store retrieved doc IDs, hashes, and scores. This explains “why the model said that.”

Tool calls: capture tool version and schema hash, argument and result hashes. For high‑risk tools (payments, PII lookups), store redacted results under object lock and reference them.

CI/CD that turns Git into the gate

Pre‑commit: prompt linting, schema validation, secrets scanning, policy-as-code checks (for example, OPA/Conftest) to block risky phrases, missing placeholders, or unsafe tool grants.

PR template: change rationale, risk level, expected impact, links to eval results.

Golden‑set evaluation: run on every PR touching prompts/configs/routing/models. Fail if accuracy drops or cost spikes beyond thresholds. Attach a diff report artifact.

Release: on merge to main, tag and sign a release; publish a provenance attestation; update a human‑readable CHANGELOG.

Rollback: git revert the offending commit, re‑run eval; if green, retag and redeploy.

GitOps tip: only sync from signed tags (not branches) and verify signatures in the controller.

Surface the audit (for non‑technical stakeholders)

Git and object storage are the tamper‑evident backend; auditors need a human‑readable front end. Provide a one‑click “Audit Bundle” that, given a request_id or date range, verifies signatures and timestamps, resolves the exact prompt/config/model/policy from Git, lists retrieved docs and tool calls (by hash and link), and renders a timeline and decision summary as signed HTML/PDF. Aim for an SLO like “audit bundle in under 15 minutes.”

Example CLI

audit report --request-id req_ab12cd34 --format html --out reports/req_ab12cd34.html

Cost and scaling

Keep Git small: don’t store models or raw logs in Git. Use object storage and reference by digest.
Logs: NDJSON in object storage, partitioned by date/tenant; query with DuckDB or your warehouse.
Storage hygiene: retention policies, legal holds, access logging, and lifecycle rules (tier older logs to colder classes).

Compared to “all‑in‑one” MLOps suites, this pattern covers 80–90% of audit and compliance value at a fraction of the cost and lock‑in. Add platforms later for experiment UX if you truly need them.

”Oh no” scenarios and exact steps

Why was this claim denied six months ago?

git checkout the release tag for that date.
Find the request_id in your case system; fetch the trace in object storage.
Verify signatures and timestamps; open the prompt/config/policy at that commit.
Inspect retrieved docs and tool call hashes; replay with saved seed and RAG snapshot if needed.

Prove we didn’t use a biased prompt in Q2.

Show prompt and policy diffs over the quarter with signed commits and reviewers.
Produce golden‑set eval trend reports tied to commits.
Provide daily manifest timestamps as independent anchors.

Roll back a bad behavior change.

Identify the prompt/config commit that introduced the regression via git bisect + eval.
git revert, re‑run eval; if green, merge and redeploy a signed tag.

Common objections (and blunt answers)

Git can’t handle big artifacts. Correct—don’t put them in Git. Use object storage and reference by digest. Git is your state machine, not your blob store.

Closed APIs change under us. True—log provider/model/version, run shadow tests daily, alert on drift, and push vendors for version pinning on enterprise plans.

Isn’t this re‑implementing MLOps? We’re implementing audit, provenance, and gating—the 20% that gives 80% of compliance value. Start here; add layers if warranted.

What about privacy? Never commit PII. Redact before inference, fingerprint inputs, encrypt sensitive trace fields, and lock your buckets.

Your implementation checklist (day one)

Stand up a repo with prompts/, configs/, model_refs/, data_manifests/, evals/, policies/, audits/, ci/, docs/.
Enable branch protection, signed commits and tags, CODEOWNERS for prompts/policies, and required status checks.
Add pre‑commit hooks for prompt lint, schema validation, and secrets scanning; add policy-as-code checks.
Wire CI to run golden‑set evals on PRs; fail on metric/cost regressions; attach diff artifacts.
On merge, sign and tag releases; publish a provenance attestation; update a CHANGELOG.
Implement runtime trace logging with the fields above; write to object storage with Object Lock and retention.
Commit a daily manifest of decision log object keys and hashes; timestamp and sign it.
Add a drift detector that compares live configs to the current signed tag and alerts on mismatch.
Ship a simple “audit bundle” CLI that produces a human‑readable report from a request_id or time range.
Document your reproducibility policy (bitwise vs behavioral), retention policy, privacy redaction rules, and the break‑glass workflow.

Closing

Every day, teams ship AI that can change behavior without a code change—and then scramble when a regulator, customer, or incident demands an explanation. Git already solved the core of this problem in 2005. Treat your prompts, configs, models, and data manifests like code. Make CI the gate. Log decisions with pointers back to that exact state in immutable storage, and be disciplined about out‑of‑band changes. You’ll get a tamper‑evident, legally defensible audit trail that’s cheaper, simpler, and more reliable than yet another platform—and you can start this week.