DEV Community: DevOps Daily

How to resolve merge conflicts in Git without guessing: markers, --ours/--theirs and the rebase flip

DevOps Daily — Thu, 08 Oct 2026 10:15:49 +0000

You are rebasing your branch onto main. Git stops on a conflict in limits.py. You want to keep your version, so you type git checkout --ours limits.py, stage it, continue and push. Your change is gone. During a rebase, "ours" is main, and "theirs" is the commit you wrote.

That one swap is why so many people learn how to resolve merge conflicts in Git by trial and error: keep the top half, run the tests, try the bottom half. Most tutorials stop at "delete the markers and commit". This one covers the parts that bite later, including the rebase flip and the clean merge that still breaks the build.

The hands-on part is the Git Merge Conflict Simulator: six lessons and 27 steps. Five open on a repository that is already mid-conflict; the sixth starts just before a clean merge that breaks the code. You type commands into a terminal, edit files in an editor, and watch the "Index stages" and "Ours and theirs" panels change. It does not run real Git: it models the commands each lesson needs, with output based on real Git's. Disclosure: I help build it; it is free, runs in the browser, no signup.

A conflict is three versions, not two

Lesson 1, Read the conflict, opens after git merge feature/retry has stopped. config.yaml started with timeout: 30. Main raised it to 45, the feature branch to 60. The markers show two of the three versions:

service: api
<<<<<<< HEAD
timeout: 45
=======
timeout: 60
>>>>>>> feature/retry
retries: 3

The third version, the common ancestor, is the most useful one and the one the markers hide. While a path is unmerged, Git keeps up to three copies of it in the index: stage 1 is the base, stage 2 is ours, stage 3 is theirs. The lesson has you run git show :1:config.yaml before you touch anything. Once you know the base was 30, you know both sides raised the timeout, so the question is "which increase wins", not "who broke it".

If you want the base inline every time, run git config --global merge.conflictStyle zdiff3 (added in Git 2.35). Conflicts then show a ||||||| section with the base between the two sides.

Resolving means writing the file you want

Lesson 2, Resolve by editing: main added redis==5.0.8 to requirements.txt, and a metrics branch added prometheus-client==0.20.0 at the same spot. The project needs both. Neither --ours nor --theirs gives you that, so you open the editor, delete the three marker lines, keep both packages and save. A save that still has markers or drops a package does not complete the step.

Then git add requirements.txt. That is how you tell Git a conflict is resolved: it replaces the three index stages with the one version you staged. Git does not read the content, so it will stage a file full of markers without complaint. In real Git, git diff --cached --check reports "leftover conflict marker" lines before you commit.

Take a whole side when nobody can merge it by hand

Lesson 3, Take a whole side: assets/logo.png is binary, so Git writes no markers at all. The working tree keeps main's logo and the index holds all three versions. The merge is a rebrand, so you want the incoming file: git checkout --theirs assets/logo.png, then git add and git commit --no-edit. During a merge, --ours is the branch you are on and --theirs is the branch you are merging in. Taking a side copies the whole file, so main's sharper outline is lost.

The rebase flip

Lesson 4, Rebase flips ours and theirs, is the opening story. A rebase checks out main's tip (HEAD is detached there) and replays your commits on top of it, one at a time. So HEAD and --ours mean main plus whatever of yours has been replayed so far, and --theirs is your commit being replayed. The simulator's side panel shows both rules:

During a	`--ours` / `HEAD`	`--theirs`
merge	the branch you are on	the branch you are merging in
rebase	the rebased result so far	your commit being replayed

The git checkout man page says the same: during git rebase and git pull --rebase, ours and theirs "may appear swapped". In the lesson the team agreed your stricter limit of 30 requests per minute beats main's 200, so the right command is git checkout --theirs limits.py, then git add and git rebase --continue. If you already typed --ours and have not staged it, git checkout -m limits.py recreates the conflict.

The escape hatch

Lesson 5, The escape hatch: you merged release/2025.12 when you meant release/2026.09, and six files are conflicted. None of them is worth resolving. git merge --abort tries to put the index and working tree back and removes MERGE_HEAD; git rebase --abort does the same for a rebase. The caveat the lesson spells out: if you had uncommitted changes when you started the merge, --abort may not be able to restore them. Commit or stash first.

Clean merge, broken code

Lesson 6, Clean merge, broken code, has no markers at all. Main renamed settings.get_timeout() to get_request_timeout() and fixed every caller it had. A week-old branch adds a new caller of the old name in a different file. git merge feature/worker succeeds without asking anything, make test fails, and you fix one line in worker.py. Each branch passed its own tests; only the combination is broken. Git merges text, not behaviour, which is why CI should run on the merge result, or a merge queue should test the combined code before it lands.

Try the flip in a real repo

To see lesson 4 on your own machine, build the same situation in a scratch directory:

mkdir conflict-lab && cd conflict-lab
git init -b main
echo "limit = 100" > limits.py
git add limits.py && git commit -m "Add rate limiting"

git switch -c feature/rate-limit
echo "limit = 30" > limits.py
git commit -am "Tighten the default rate limit"

git switch main
echo "limit = 200" > limits.py
git commit -am "Allow 200 requests per minute"

git switch feature/rate-limit
git rebase main
cat limits.py

With the default conflict style on Git 2.39, the rebase stops and cat prints this (your hash will differ, and with zdiff3 set you also get a ||||||| base section):

<<<<<<< HEAD
limit = 200
=======
limit = 30
>>>>>>> f9ce8c6 (Tighten the default rate limit)

The half labelled HEAD is main's line, even though you started on your feature branch. git show :2:limits.py prints limit = 200 and git show :3:limits.py prints limit = 30. Run git checkout --theirs limits.py, git add limits.py and git rebase --continue (save the commit message when the editor opens), and your 30 survives.

Where to go next

Do the six lessons in order: the first three build the model that makes the fourth obvious. The simulator is one of 50+ free DevOps games and simulators. Official references: the "How conflicts are presented" section of git-merge and the --ours/--theirs note in git-checkout.

OpenTofu 1.13 Re-Encodes base64gzip and Drops WinRM. We Upgraded the Same State to See What Breaks

DevOps Daily — Wed, 07 Oct 2026 09:00:00 +0000

OpenTofu 1.13.0 was published on September 30 (the release announcement is dated September 29), and 1.13.1 followed on October 1 with two fixes for ephemeral values. The headline features are new functions that let module authors describe values OpenTofu cannot know until apply, plus two experiments: built-in linting and symbol libraries. The upgrade notes are what you will notice first. base64gzip now returns different bytes for the same input, WinRM provisioners are gone, and 1.13 is the last series with official 32-bit builds.

Upgrade notes tell you what changed. They do not show you your next plan, or the step at which a removed feature fails. So we tested it. We applied configurations with OpenTofu 1.12.7, ran 1.13.1 against the same state, and recorded what each command printed. This post goes through the results and ends with a checklist you can run before you change the version in CI.

TLDR

base64gzip in 1.13.1 returns a different string for the same input. OpenTofu moved to Go 1.27, and Go changed its DEFLATE encoder. Both outputs decompress to identical bytes, but providers compare the string. With no config change, our plan was 1 to add, 1 to change, 1 to destroy.
On aws_instance, a changed user_data_base64 means a stop/start, or a replacement if you set user_data_replace_on_change. On azurerm_linux_virtual_machine, a changed custom_data forces a new VM.
ignore_changes hides the diff. In our test it also hid a real edit to the cloud-init file. Gzip done inside the cloudinit provider did not change at all.
A winrm provisioner passes tofu validate and tofu plan on 1.13.1, with the old "will be removed in a future version" warning. It fails at apply, after the resource exists, and leaves the resource tainted.
assumenotnull fixed a classic Invalid count argument error. assumestringprefix moved a mis-wired module input from a mid-apply failure to a plan error. Older versions reject these functions, and since 1.12 a required_version in a .tf file does not stop them.
-lint=all is a useful experiment, but it only adds warnings. The exit code stays 0.

Prerequisites

An OpenTofu 1.12.x codebase, or a Terraform codebase that you plan to move to OpenTofu
A CI job or shell that can run tofu plan with read access to your real state
jq, for the plan JSON recipe
The 1.13.1 zip for your platform from the GitHub release page, checked against its SHA256SUMS file

How we tested

We ran everything on a Raspberry Pi with a 64-bit (arm64) OS: OpenTofu 1.12.7 and 1.13.1 for linux_arm64, the 1.13.1 linux_arm (32-bit) build, and Terraform 1.16.5 for comparison. Every zip matched its published SHA256 sum. In the outputs below, tofu-1.12.7 and tofu-1.13.1 are the two release binaries side by side.

We did not use a cloud account. The built-in terraform_data resource stands in for provider attributes. Its input argument updates in place when the value changes, and its triggers_replace argument forces a replacement. Those are the two ways real providers treat user data, which we checked in the provider docs. Every output in this post comes from these runs. Where we cut lines from an output, the block shows ... or the grep/tail we used, and long base64 strings are shortened with ....

base64gzip: same input, different string

base64gzip compresses a string with gzip and base64-encodes the result. Its most common job is cloud-init user data, because EC2 limits user data to 16 KB and compression gives you more room. Here is the same call on both versions, plus Terraform 1.16.5, and then a check of what a real payload (a 1,208-byte cloud-init file that installs nginx and writes a systemd unit) decompresses to:

base64gzip across versions

# same input, three binaries
$ echo 'base64gzip("hello")' | ./tofu-1.12.7 console
"H4sIAAAAAAAA/8pIzcnJBwAAAP//AQAA//+GphA2BQAAAA=="
$ echo 'base64gzip("hello")' | ./tofu-1.13.1 console
"H4sIAAAAAAAA/wAFAPr/aGVsbG8AAAD//wMAhqYQNgUAAAA="
$ echo 'base64gzip("hello")' | ./terraform-1.16.5 console
"H4sIAAAAAAAA/8pIzcnJBwAAAP//AQAA//+GphA2BQAAAA=="
# decompress the cloud-init payload from each version
$ echo 'base64gzip(file("cloud-init.yaml"))' | ./tofu-1.12.7 console | tr -d '"' | base64 -d | gunzip | sha256sum
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1 -
$ echo 'base64gzip(file("cloud-init.yaml"))' | ./tofu-1.13.1 console | tr -d '"' | base64 -d | gunzip | sha256sum
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1 -
$ sha256sum < cloud-init.yaml
3a62ee85006428492dc1aeccc789867735496026ada3bfaef3d0cf00dcc1bcd1 -

The encoded strings differ. The content does not: both outputs decompress to the exact bytes of the source file.

The cause is upstream. OpenTofu 1.12.7 is built with Go 1.26.6 and 1.13.1 with Go 1.27.1 (each binary records its Go version). The Go 1.27 release notes say that "the exact encoded output from Writer may be different from Go 1.26 as a result of the encoder implementation change", and that this carries through to compress/gzip. The OpenTofu 1.13 changelog calls the new output "equivalent to but not equal to" the output of earlier releases. The new output is stable: three runs of 1.13.1 gave the same string.

Terraform 1.16.5 is built with Go 1.26.8 and produced the same bytes as OpenTofu 1.12.7. So you also get this diff when you move from Terraform 1.16 to OpenTofu 1.13, not only when you upgrade OpenTofu.

What the plan shows

cloud-init.yaml unchanged
base64gzip() Go 1.27 DEFLATE
New string same bytes after gunzip
Provider diff string != state
Plan update or replace

This is the config we applied with 1.12.7:

locals {
  user_data = base64gzip(file("${path.module}/cloud-init.yaml"))
}

# Stands in for an attribute that a provider updates in place,
# such as user_data_base64 on aws_instance.
resource "terraform_data" "web_in_place" {
  input = local.user_data
}

# Stands in for an attribute that forces replacement,
# such as custom_data on azurerm_linux_virtual_machine.
resource "terraform_data" "web_replace" {
  triggers_replace = local.user_data
}

After tofu-1.12.7 apply, a 1.12.7 plan reported no changes. Then we planned with 1.13.1 against the same state and the same config:

$ tofu-1.13.1 plan
terraform_data.web_replace: Refreshing state... [id=4437d213-ef2e-bd8b-9c81-b5831d1b46c8]
terraform_data.web_in_place: Refreshing state... [id=b0ec5936-31c0-4653-0fb6-e353122675fa]

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place (current -> planned)
-/+ destroy and then create replacement

OpenTofu will perform the following actions:

  # terraform_data.web_in_place will be updated in-place
  ~ resource "terraform_data" "web_in_place" {
        id = "b0ec5936-31c0-4653-0fb6-e353122675fa"
      ~ input = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> "H4sIAAAAAAAA/5RTXW/jNhB89..."
      ~ output = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> (known after apply)
    }

  # terraform_data.web_replace must be replaced
-/+ resource "terraform_data" "web_replace" {
      ~ id = "4437d213-ef2e-bd8b-9c81-b5831d1b46c8" -> (known after apply)
      ~ triggers_replace = "H4sIAAAAAAAA/5RU3U7zRhC9z..." -> "H4sIAAAAAAAA/5RTXW/jNhB89..."
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Two resources, no edits, one update and one replacement. On real resources, the result depends on the provider:

aws_instance: for user_data_base64, where gzip output belongs, "Updates to this field will trigger a stop/start of the EC2 instance by default. If the user_data_replace_on_change is set then updates to this field will trigger a destroy and recreate of the EC2 instance."
azurerm_linux_virtual_machine: for custom_data, "Changing this forces a new resource to be created."
aws_launch_template: a changed user_data creates a new template version. Instances get it the next time your Auto Scaling group launches or refreshes from that version.

None of this is a disaster if you plan for it. But a stop/start of a single production box at 2 p.m. is the kind of surprise you want to catch in plan review, not after someone approves an apply because "it is only a version bump".

Find it before the upgrade

Start with a search. Run it after tofu init, so that it also covers registry and git modules in .terraform/modules:

grep -rnE --include='*.tf' --include='*.tofu' 'base64gzip\(|"winrm"' .

On our test folder it found both problems this post covers:

main.tf:2: user_data = base64gzip(file("${path.module}/cloud-init.yaml"))
winrm/main.tf:6: type = "winrm"

A search only finds literal calls. A plan is the reliable check. Run a 1.13.1 plan in a throwaway job, save it, and list each changed attribute with this jq filter (save it as changed-attrs.jq):

.resource_changes[]
| select(.change.actions != ["no-op"])
| . as $r
| [($r.change.before // {}) | keys[]
    | select($r.change.before[.] != $r.change.after[.]
             and ($r.change.after_unknown[.] | not)) ]
| "\($r.change.actions | join("+")) \($r.address) changed: \(join(", "))"


$ tofu-1.13.1 plan -lock=false -out=upgrade.tfplan > /dev/null
$ tofu-1.13.1 show -json upgrade.tfplan | jq -r -f changed-attrs.jq
update terraform_data.web_in_place changed: input
delete+create terraform_data.web_replace changed: triggers_replace

A plan does not write state, and -lock=false stops a dry-run job from blocking a real apply. Remove the flag if you prefer to wait for the lock. You want a list in which every changed attribute is user data. Anything else in the list is real drift or a different upgrade change, so examine it separately.

Three ways to handle it

Handling the base64gzip diff

Accept it

# Plan with 1.13.1, review, and apply in a window you choose.
# The content is identical, so the only effect is the
# stop/start or replacement itself.
tofu plan -out=upgrade.tfplan
tofu show -json upgrade.tfplan | jq -r -f changed-attrs.jq
tofu apply upgrade.tfplan

Hide it (temporary)

resource "aws_instance" "web" {
  # ...
  user_data_base64 = base64gzip(file("${path.module}/cloud-init.yaml"))

  lifecycle {
    # TEMPORARY: OpenTofu 1.13 re-encodes base64gzip output.
    # This also hides real cloud-init edits. Remove it on the next change.
    ignore_changes = [user_data_base64]
  }
}

Gzip in the provider

data "cloudinit_config" "web" {
  gzip = true
  base64_encode = true

  part {
    content_type = "text/cloud-config"
    content = file("${path.module}/cloud-init.yaml")
  }
}

resource "aws_instance" "web" {
  # ...
  user_data_base64 = data.cloudinit_config.web.rendered
}

Accept it. This is usually the correct choice. Do the upgrade apply in a maintenance window, one environment at a time, and use the jq list as the change record.

Hide it. The release notes suggest ignore_changes as a temporary fix, and it works: with ignore_changes on both test resources, the 1.13.1 plan said No changes. Then we added a line to cloud-init.yaml and planned again. It still said No changes.

Warning

ignore_changes cannot tell the encoder change from a real edit. While it is in place, changes to your cloud-init file do not reach your instances, and the plan does not tell you. If you use it, open a ticket to remove it, and remove it on the next intentional user data change.

Move the gzip into the provider. With cloudinit_config and gzip = true, the compression runs in the provider binary, not in OpenTofu. We rendered the same file with hashicorp/cloudinit v2.4.1 under both 1.12.7 and 1.13.1, and the SHA256 of rendered was identical. This does not make you immune. It moves the dependency: v2.4.1 is built with Go 1.26.8, and a future provider release built with Go 1.27 can cause the same one-time diff. So pin the provider version and read its changelog. The switch itself also changes your user data once, because cloudinit_config wraps parts in a MIME multi-part document. Do it in the same window as the upgrade.

WinRM: passes validate and plan, fails at apply

OpenTofu 1.12 deprecated the winrm connection type, and 1.13 removed it (#4012) because "some of the upstream libraries OpenTofu was using to implement these features are no longer maintained". We expected tofu validate to report it. It does not. This is the test config (the host is a closed local port with a short timeout, so that the run fails fast):

resource "terraform_data" "bootstrap" {
  provisioner "remote-exec" {
    inline = ["powershell -Command Install-WindowsFeature Web-Server"]

    connection {
      type = "winrm"
      host = "127.0.0.1"
      user = "Administrator"
      password = "example-only"
      https = true
      timeout = "10s"
    }
  }
}


$ tofu-1.13.1 validate
Warning: WinRM connection type is deprecated

  on main.tf line 6, in resource "terraform_data" "bootstrap":
   5: connection {
   6: type = "winrm"

The winrm connection type is deprecated and will be removed in a future
version of OpenTofu.
...
Success! The configuration is valid, but there were some validation warnings
as shown above.

$ tofu-1.13.1 plan | grep 'Plan:'
Plan: 1 to add, 0 to change, 0 to destroy.

$ time tofu-1.13.1 apply -auto-approve
...
Error: remote-exec provisioner error

  with terraform_data.bootstrap,
  on main.tf line 2, in resource "terraform_data" "bootstrap":
   2: provisioner "remote-exec" {

'winrm' connections are not supported in OpenTofu v1.13 or later

Error: Provisioners no longer support WinRM
...
real 0m0.153s

$ tofu-1.13.1 show | head -2
# terraform_data.bootstrap: (tainted)
resource "terraform_data" "bootstrap" {

1.13.1 still prints the 1.12 deprecation warning at validate and plan. It says the feature "will be removed in a future version", but the feature is already gone in this version. The error comes in under a second at apply. For comparison, 1.12.7 tried to connect to port 5986 and failed at the 10-second timeout, as it should.

The order matters. Provisioners run after the resource is created, and the OpenTofu docs say "if a creation-time provisioner fails, the resource is marked as tainted" and "will be planned for destruction and recreation upon the next tofu apply". With a real Windows VM, the VM is created and billed, then marked for replacement, and each apply after that recreates it and fails again until you remove the provisioner. Existing VMs whose provisioners ran long ago are not affected until something replaces them. Be careful with that last point: if a Windows VM uses gzipped custom_data, the base64gzip change above is exactly the kind of thing that replaces it, and then its WinRM provisioner runs again and fails.

To fix it, move to SSH or remove the provisioner:

Windows Server 2019 and later can run OpenSSH Server. Set type = "ssh" and target_platform = "windows" in the connection block. If the SSH default shell is PowerShell, the connection docs also tell you to set script_path to a .ps1 path.
Better, if you can: bake the configuration into the image, or run it from custom_data or user_data, so that no provisioner has to connect at all.

The grep above finds literal "winrm" strings. It does not find type = var.connection_type, so also search for connection blocks whose type comes from a variable.

32-bit builds: last series

The 1.13 changelog says this is "the final release series that will include official builds for 32-bit CPU architectures" (*_386 and *_arm). The Pi's 64-bit kernel can run 32-bit ARM binaries, so we ran the linux_arm build:

$ uname -m
aarch64
$ tofu-1.13.1-linux-arm version
OpenTofu v1.13.1
on linux_arm
$ tofu-1.13.1-linux-arm init

Warning: Support for 32-bit CPU architectures is ending soon

OpenTofu v1.13 is the last release series that will include official release
packages for 32-bit CPU architectures.

We recommend planning to migrate to a 64-bit CPU architecture instead.
Alternatively, you could build OpenTofu for linux_arm from source code
yourself, ...

Look at the second line of tofu version on each runner. If it says on linux_arm or on linux_386, that runner has to move. Common cases are 32-bit Raspberry Pi OS runners, old ARMv7 boards, and i386 container images. As the test shows, a 64-bit kernel does not help if the image or the binary you install is 32-bit. You have time: per the changelogs, the 1.13 series is supported until August 1, 2027, and 1.12 until February 1, 2027. The 1.11 series lost support on August 1, 2026.

The new functions: hints for unknown values

This is the main feature of the release. During a plan, any value that an API assigns at creation time is unknown, and OpenTofu cannot use an unknown value to decide how many instances to create. Here is the classic case: a network module creates a VPC, and a second module creates flow logs only when it gets a VPC ID.

# modules/network/main.tf
# terraform_data stands in for aws_vpc: its output is unknown until apply,
# the same way a VPC id is decided by the AWS API at creation time.
resource "terraform_data" "vpc" {
  input = "vpc-0a1b2c3d4e5f60718"
}

output "vpc_id" {
  value = terraform_data.vpc.output
}

# modules/flow_logs/main.tf
variable "vpc_id" {
  type = string
  default = null
}

resource "terraform_data" "flow_log" {
  count = var.vpc_id != null ? 1 : 0
  input = var.vpc_id
}

On a first plan, 1.12.7 and 1.13.1 fail the same way:

Error: Invalid count argument

  on modules/flow_logs/main.tf line 7, in resource "terraform_data" "flow_log":
   7: count = var.vpc_id != null ? 1 : 0

The "count" value depends on resource attributes that cannot be determined
until apply, so OpenTofu cannot predict how many instances will be created.
...

The assume... functions let the module author state what is true about the value even before it exists. Our first attempt failed:

Error: Invalid function argument

  on modules/network/main.tf line 8, in output "vpc_id":
   8: value = assumenotnull(terraform_data.vpc.output)

Invalid value for "value" parameter: given value must have a known type;
consider using the \"convert\" function to specify a type to assume.

terraform_data.output takes on the type of input, so its type is also unknown during the plan. Most provider attributes, such as aws_vpc.id, are typed strings and do not have this problem. For a value like this, the docs tell you to combine the hint with the new convert function:

output "vpc_id" {
  value = assumenotnull(convert(terraform_data.vpc.output, string))
}


$ tofu-1.13.1 plan | grep -E 'will be created|Plan:'
  # module.flow_logs.terraform_data.flow_log[0] will be created
  # module.network.terraform_data.vpc will be created
Plan: 2 to add, 0 to change, 0 to destroy.

$ tofu-1.13.1 apply -auto-approve | tail -1
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The usual workaround for this error is a two-step apply with -exclude (the error message itself suggests it). Here, one plan is enough.

A hint is a promise, and OpenTofu checks it. The docs say that if the value turns out to be null, "the function raises an error", so the apply fails. Use a hint only where the provider really guarantees it, for example that an ID is never null after create.

Catch wiring mistakes at plan time

assumestringprefix is useful together with variable validation. Here the caller connects the subnet output to an input that expects a VPC ID, a mistake that is easy to miss in review:

# main.tf
module "flow_logs" {
  source = "./modules/flow_logs"
  vpc_id = module.network.subnet_id # wrong output wired in
}

# modules/flow_logs/main.tf
variable "vpc_id" {
  type = string

  validation {
    condition = startswith(var.vpc_id, "vpc-")
    error_message = "vpc_id must be a VPC id (vpc-...)."
  }
}

Without hints, the plan passes because the value is unknown, and validation waits for apply. The apply created the VPC and the subnet, and then failed. Both outputs are filtered to the key lines, and the IDs are shortened:

$ tofu-1.13.1 plan
Plan: 3 to add, 0 to change, 0 to destroy.
$ tofu-1.13.1 apply -auto-approve
module.network.terraform_data.subnet: Creation complete after 0s [id=...]
module.network.terraform_data.vpc: Creation complete after 0s [id=...]
Error: Invalid value for variable
vpc_id must be a VPC id (vpc-...).
$ tofu-1.13.1 state list
module.network.terraform_data.subnet
module.network.terraform_data.vpc

Then we added prefix hints to the network module outputs:

output "vpc_id" {
  value = assumenotnull(assumestringprefix(convert(terraform_data.vpc.output, string), "vpc-"))
}

output "subnet_id" {
  value = assumenotnull(assumestringprefix(convert(terraform_data.subnet.output, string), "subnet-"))
}

Now the same mistake fails the plan, before anything is created:

$ tofu-1.13.1 plan
Error: Invalid value for variable

  on main.tf line 7, in module "flow_logs":
   7: vpc_id = module.network.subnet_id # wrong output wired in
    ├────────────────
    │ var.vpc_id is a string

vpc_id must be a VPC id (vpc-...).

This was checked by the validation rule at modules/flow_logs/main.tf:4,3-13.

If you maintain shared modules, the outputs of your network, IAM and DNS modules are the best places for these hints. Callers get the benefit without changing their own code. assumeequal goes further: the docs show it with the AWS provider's arn_build function to make an IAM role ARN fully known at plan time, so that policy checks can see the real policy document.

Guard the module version correctly

A module that uses these functions does not work on older versions, and the errors are not clear. On 1.12.7, assumenotnull(...) alone gives Call to unknown function. With convert(..., string), the error is Invalid reference, because 1.12 reads string as a resource address. Terraform 1.16.5 also gives Call to unknown function.

You would usually add required_version = ">= 1.13.0" to a terraform block. That does not work here. Since 1.12, OpenTofu ignores required_version in .tf files and honors it only in .tofu files (see the RFC tracking issue and the settings docs). We tested this with a constraint that no version can meet:

Constraint and file	1.12.7	1.13.1
`required_version = ">= 99.0"` in `main.tf`	ignored, validate passes	ignored, validate passes
`required_version = ">= 99.0"` in `versions.tofu`	`Incompatible module`	`Incompatible module`
`language { compatible_with { opentofu = ">= 1.13" } }` in `versions.tofu`	`Incompatible module`	plan passes

So put the guard in a .tofu file:

# versions.tofu (the language block needs OpenTofu 1.12 or later)
language {
  compatible_with {
    opentofu = ">= 1.13"
  }
}

On 1.12.7 this gives This module is not compatible with OpenTofu v1.12.7, which is the clear message you want. If a module must also work with Terraform, use the .tofu precedence rule: when outputs.tf and outputs.tofu are both present, OpenTofu loads only the .tofu file. We put a hinted output in outputs.tofu and a plain one in outputs.tf. tofu validate (1.13.1) and terraform validate (1.16.5) both passed.

The lint experiment

validate, plan, apply and refresh accept a new -lint flag. 1.13 has four rules, all for the root module only: core:no-type-variable, core:unused-variable, core:unused-local, and core:count-instead-enabled. The last one suggests the enabled lifecycle argument instead of count = cond ? 1 : 0. We ran the experiment on a small file that breaks all four rules:

$ tofu-1.13.1 validate -lint=all -json | jq -r '.diagnostics[] | select(.severity == "warning") | .summary'
Experimental linting enabled
Input variable not used (core:unused-variable)
Local value not used (core:unused-local)
Could use enabled instead of count (core:count-instead-enabled)
Variable with no type (core:no-type-variable)

The exit code was 0. Lint results are warnings, and OpenTofu always adds an "Experimental linting enabled" warning. You can turn off one rule with !, for example -lint='all,!core:unused-variable'. The linting docs say that rules can change even in minor releases. For now, make it a non-blocking CI step and read the output. Do not gate merges on it yet.

Smaller changes worth a line

Plan text. After No changes. Your infrastructure matches the configuration., 1.12.7 printed one more paragraph ("OpenTofu has compared your real infrastructure ... no changes are needed."). 1.13.1 does not. If a script greps for that paragraph, change it to use tofu plan -detailed-exitcode, which returns 0 for no changes, 1 for errors, and 2 for changes. Our upgrade plan returned 2.
Platforms. Windows on ARM64 is now an official platform, and macOS builds require macOS 13 Ventura or later.
State encryption. The aws_kms key provider accepts an encryption_context. gcp_kms accepts additional_authenticated_data, and openbao accepts associated_data.
Crash recovery. A Go panic now writes a partial errored.tfstate to help you recover.
Saved plans include the provider schemas, so tofu show on a plan file usually does not need to start providers.
If you stay on 1.12 for now , take 1.12.7. It fixes a deadlock that an attacker-controlled SSH server could cause in remote-exec and file provisioners (CVE-2026-78662).

Upgrade checklist

After tofu init, search your code and .terraform/modules for base64gzip( and "winrm", and look for connection blocks whose type comes from a variable.
Remove WinRM provisioners before you upgrade. Validate and plan will not stop you, and a failure at apply taints the resource.
Run a 1.13.1 plan against real state in a throwaway job, save it, and run the jq filter. Every changed attribute should be user data.
For each user data change, decide: accept the stop/start or replacement in a window, or use a temporary ignore_changes with a ticket to remove it.
Check tofu version on every runner and workstation image, and move anything on linux_arm or linux_386 to 64-bit before August 1, 2027.
Change scripts that read plan text to use -detailed-exitcode.
In shared modules, add assume... hints to outputs whose IDs callers use in count, for_each or validation. Put a language block in a .tofu file to guard them.
Add -lint=all to CI as a non-blocking step and see what it finds.

The upgrade is not hard, but two of these changes are not visible until a specific step: the base64gzip diff shows only in a plan against real state, and the WinRM removal shows only at apply. If you run steps 1 to 3 first, you see both before they affect a real server.

Originally published at devops-daily.com.

p99, Load Balancers and Autoscaling: Latency Intuition You Can Play With

DevOps Daily — Tue, 06 Oct 2026 18:43:17 +0000

Averages hide latency problems. Every SRE learns this, usually the hard way: the dashboard says 80ms average while support tickets say the app is slow, because the average smooths over a tail where one request in a hundred takes four seconds, and your heaviest users, the ones making the most requests, hit that tail most often.

Percentiles are how you see the tail, and the vocabulary around them (p50, p95, p99, tail latency) is quick to memorize and slow to internalize. Intuition, the ability to look at a p99 spike and have sensible suspects, usually comes from incidents. Three free browser simulators let you build a chunk of it without the incidents. Disclosure: I help build them; free, browser-based, no signup.

Part 1: watch a percentile move

The Latency Percentiles Simulator generates request-latency distributions and renders them as a histogram with P50, P90, P95 and P99 markers that move as you change the controls. The five scenarios are synthetic distributions shaped like five kinds of trouble:

Healthy API: the baseline. A tight distribution; the percentiles sit close together. This is what "fine" looks like, and knowing its shape is what lets you recognize not-fine.
Cold Starts: a small fraction of requests pay a big startup cost. P50 barely notices; P99 jumps. This is the canonical "the average is fine, the tail is not" case, and it is the shape serverless platforms and JIT warmup produce.
Cache Churn: hits are fast, misses are slow, and the mix moves the middle percentiles around. Different shape than cold starts: the distribution goes wide rather than growing a spike.
Noisy Neighbor: shared infrastructure steals capacity intermittently, smearing the whole distribution outward. The tail grows, but unlike cold starts, so does everything above the median.
Rolling Deploy: a distribution shaped like a fleet mid-deploy, two populations blended, which is the shape to recognize in real telemetry when half your instances are new.

These are stylized shapes, not simulations of caches or neighbors, and that is fine for the purpose: the transferable skill is that similar p99 numbers can sit on very different distributions. Percentiles alone do not diagnose a cause, but knowing the classic shapes gives you better suspects to check when a tail spike arrives.

The other lesson hiding here is why tails matter more than their percentage suggests: if a page fans out to dozens of roughly independent backend calls and waits for all of them, the chance that at least one hits the p99 grows fast, so the "1% case" ends up in far more than 1% of page loads. Google's SRE material makes the same point: a backend's p99 can effectively become the frontend's median.

Part 2: the balancer decides who eats the tail

The Load Balancer Simulator animates a three-server fleet behind a balancer you configure live: the algorithm (round robin, least connections, IP hash, random), the traffic rate up to a burst mode, a server crash probability (up to a brutal 30%), retries on or off, and, the good part, you can click a server to knock it offline mid-run.

Two experiments teach the most. First, crank the failure rate with retries on and watch what a crash actually costs: the request does not just fail, it goes around again, adding load precisely when the fleet is struggling. Retries are not free; the counter of crashed-then-retried requests makes that concrete. Second, knock a server offline under each algorithm and watch the traffic redistribute; then try it with IP hash, where the same clients always land on the same server. Stickiness is what session-state architectures want, and the redistribution behavior when a sticky target dies is the price nobody mentions in the algorithm's one-line description.

Round robin versus least connections rounds it out: equal turns versus routing by current load. With identical healthy servers they look similar; under crashes and bursts, watching which algorithm piles work where is the point.

Part 3: capacity as a moving target

The Scaling Simulator hands you autoscaling controls (thresholds, instance counts, vertical versus horizontal choices) and four traffic scenarios to survive: Gradual Growth, Sudden Spike, Black Friday, Variable Load.

Gradual growth is the tutorial level; almost any policy works. The hard lessons are in the other three:

Sudden Spike punishes reactive scaling: by the time the threshold trips and instances come up, the spike already hurt users. The lesson is that autoscaling has a reaction time, and traffic that moves faster than it needs headroom, not thresholds.
Black Friday is the predictable-surge case: adding capacity before the surge you know is coming beats any reactive policy. The simulator lets you provision ahead by hand, which is the whole trick, minus the calendar automation real platforms add.
Variable Load is oscillating traffic, the pattern that punishes twitchy autoscaling with flapping in real systems. The simulator's scale-down cooldown is deliberately long, so what you actually observe is the defense working: capacity ratchets up and holds through the dips instead of thrashing. Cooldowns and stabilization windows are exactly how production autoscalers (Kubernetes' HPA included) buy calm at the cost of some idle capacity.

The through-line with parts 1 and 2: capacity decisions surface as latency and errors. The scaling simulator tracks response time and failures rather than full percentile distributions, but the connection stands: under-provisioned fleets are where the ugly histograms of part 1 come from, and the autoscaler's job is to buy the healthy shape with as few idle instances as possible.

Using the trio

A sequence that works: learn the five distribution shapes in part 1, run the crash-and-retry and kill-a-server experiments in part 2, then survive Sudden Spike and Black Friday in part 3. After that, three things in production read differently: a p99 chart makes you ask what the distribution under it looks like, balancer choice reads as a failure-behavior decision, and autoscaler settings read as a bet about how your traffic moves.

All three live with the rest of our free DevOps games and simulators. For the deep end afterwards, Google's SRE book chapter on monitoring distributed systems and Gil Tene's classic How NOT to Measure Latency are the standard references, easier to absorb once you have watched a histogram misbehave yourself.

A Poisoned .git/config Runs Code on git status. We Tested Which Commands and Copies Carry It

DevOps Daily — Tue, 06 Oct 2026 09:00:00 +0000

On October 2, GitLab's Threat Research Group published ConfigPoisoning (CVE-2026-102437), a command execution bug in DeepSeek-Reasonix Studio, a git client built for working next to AI coding assistants. The tool was careful. It neutralized core.fsmonitor on every git call and added --no-ext-diff --no-textconv to every diff. It still ran attacker code when you opened a diff, because a clean filter named in .gitattributes runs anyway. GitLab says "the same pattern is present in several other widely used agent tools, currently under coordinated disclosure."

It is the second disclosure of this kind in five weeks. On September 1, Manifold Security published GitSpawn: eight findings across seven coding agents (Claude Code, Codex, Cursor, goose, Qwen Code, Grok Build and Hermes Agent) where repository-controlled git settings ran commands outside the agent's sandbox. Most of the detailed cases used a core.fsmonitor line in the repo's own .git/config; in Claude Code's case, the start-up git status ran it while the workspace-trust prompt was still waiting for an answer.

Both reports are about agents, but the mechanism is plain git, and git also runs in your CI jobs, bots and editor plugins. So we tested git itself: which everyday commands run programs that a repo's .git/config or hooks point at, whether the usual hardening flags stop them, and which ways of moving a repo between machines carry that config along.

TLDR

A bare git status ran repo-supplied programs: core.fsmonitor, a post-index-change hook from .git/hooks or core.hooksPath, and on git 2.55 a hook defined in config.
git diff --no-ext-diff --no-textconv still ran the clean filter (and a filter.<name>.process helper). Overrides only stopped a filter when we named its driver.
On git 2.55, -c core.hooksPath=/dev/null did not stop a hook defined in config (hook.<name>.command, new in git 2.54). Adding -c hook.<event>.enabled=false did, in our fixture.
git clone and cloning from a bundle never brought the config or hooks. A tar of the working copy, which is what a workspace cache or artifact is, always did.
git's ownership check (safe.directory) refused a copy owned by another user on our Raspberry Pi. On GitHub-hosted runners it never fired, because the images set safe.directory = *.
The scripts, the recorded results and a read-only audit are in the repo below.

Prerequisites

git and bash, to run the scripts
A machine where you can create throwaway repos: every "payload" in the tests only appends a line to a log file

Why a repo can make git run programs at all

Some git settings are commands, not values. core.fsmonitor names a helper that tells git which files changed, so big repos do not stat every file. A filter driver (filter.<name>.clean, smudge or the long-running process) rewrites file content on the way into and out of the index; that is how Git LFS works. diff.external and diff.<name>.textconv replace or preprocess diffs. Hooks are executables in .git/hooks or in the directory core.hooksPath points to, and since git 2.54 a hook can also be a command defined in config (hook.<name>.command plus hook.<name>.event, see git-hook).

Git reads all of these from the repository's own .git/config and hooks directory. That is documented, intended behaviour for a repo you created. The trouble starts when .git came from someone else.

A normal clone protects you: git does not transfer .git/config or hooks over a fetch. .gitattributes does travel, because it is a tracked file, but a filter named there does nothing unless config defines it. So the question for a DevOps team is not "can a repo do this", it is "where do we copy a .git directory instead of cloning it". GitLab's report lists the answers: "an archive, a synced folder, a CI cache, or a devcontainer build", plus a compromised agent that writes the config into a repo you cloned normally.

The test

The repo builds small throwaway repositories. Each has one committed file and the same file with a line appended, so there is always a change to show, and a .git/config (or hooks directory) that points one setting at a script whose only job is to log that it ran and pass content through:

The-DevOps-Daily/git-config-exec-check on GitHub

We tested eleven settings: core.fsmonitor; clean, smudge and process filters; a textconv driver; diff.external; hooks in .git/hooks; hooks via core.hooksPath; a hook defined in config; core.sshCommand; and core.pager. For each one, scripts/matrix.sh copies the repo fresh, runs 15 commands that tools and people run all the time, and records the exit code and what ran.

We ran it on a Raspberry Pi with git 2.39.5, and on GitHub-hosted ubuntu-latest and macos-latest runners with git 2.55.0. Ubuntu and macOS were identical. Git 2.39.5 ran the same programs except for the config-hook column, which it does not support. This is the git 2.55.0 table:

Command	fsmonitor	clean	process	smudge	textconv	diff.external	hooks dir	config hook	sshCommand
`git status`	runs						post-index-change	runs
`git status --porcelain`	runs						post-index-change	runs
`git diff`	runs	runs	runs		runs	runs
`git diff --no-ext-diff --no-textconv`	runs	runs	runs
`git log -p -1`					runs
`git log --oneline -1`
`git show HEAD`					runs
`git blame notes.txt`	runs	runs	runs		runs
`git ls-files`	runs
`git rev-parse HEAD`
`git add -A`	runs	runs	runs				post-index-change	runs
`git commit -qam wip`	runs	runs	runs				post-commit, post-index-change, pre-commit, reference-transaction	runs
`git stash`	runs	runs	runs	runs			post-index-change, reference-transaction	runs
`git checkout -- notes.txt`	runs		runs	runs			post-checkout, post-index-change	runs
`git fetch origin (fails)`	runs								runs

How to read it:

A blank cell means "not observed with this fixture", not "can never run". Because our modified file is longer than the committed one, git can see the change from the file size without filtering it. With a same-size edit or a touched file, git status may need to compare content, and then filters can run too.
hooks dir was identical for .git/hooks and core.hooksPath, so they share a column. core.pager never ran, because none of the commands had a terminal; agents and CI jobs usually do not either.
Our process helper does not speak git's filter protocol. Git started it either way; git 2.39.5 then exited with 128, and git 2.55.0 carried on and exited 0.
Only the sshCommand fixture has a remote, and its fetch fails on purpose. A successful fetch can fire more hooks than this row shows.

Three rows deserve a second look.

git status is not read-only. It refreshes the index, and refreshing the index is when git asks the fsmonitor helper what changed. When the refresh writes the index back, git fires the post-index-change hook, from the hooks directory and from config. All of that happened on a repo with one modified file.

git fetch ran fsmonitor before it contacted the remote. In the fsmonitor fixture there is no remote at all. scripts/trace-fetch.sh shows git starting the helper twice (asking for protocol version 2, then falling back to version 1 after our helper exited non-zero) and only then failing to find origin:

git 2.39.5 on Linux aarch64
run-command.c:655 trace: run_command: cd <tmp>/r; '<tmp>/hook.sh fsmonitor' 2 1791269076974045040
run-command.c:655 trace: run_command: cd <tmp>/r; '<tmp>/hook.sh fsmonitor' 1 1791269076974045040
run-command.c:655 trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 'git-upload-pack '\''origin'\'''
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

The diff flags that hardened tools add are aimed at the wrong half. --no-ext-diff and --no-textconv stop the programs that render a diff. The clean filter runs earlier, when git turns the working-tree file into a blob to compare. That is the GitLab finding, and plain git 2.55 behaves the same way. Diffs between two commits (git log -p, git show) never ran a filter, because no working-tree file is involved.

Do the usual flags help?

scripts/overrides.sh builds one repo with fsmonitor, a hook in .git/hooks, a hook defined in config, diff.external, a textconv driver and a clean filter. Each row starts from a fresh copy, runs git status and git diff, and adds more protection. On git 2.55.0:

git 2.55.0 on Linux x86_64
plain exit 0/0 ran: diff-external,filter-clean,fsmonitor,hook:config,hook:dir
diff --no-ext-diff --no-textconv exit 0/0 ran: filter-clean,fsmonitor,hook:config,hook:dir
+ -c core.fsmonitor=false -c core.hooksPath=/dev/null exit 0/0 ran: filter-clean,hook:config
+ -c filter.lab.clean= (needs the driver name) exit 0/0 ran: hook:config
+ -c hook.post-index-change.enabled=false (per event) exit 0/0 ran:

Read it row by row:

With no protection, five programs ran: fsmonitor, the hooks-directory hook, the config hook, the clean filter and diff.external. The textconv driver did not, because diff.external replaces the whole diff; the matrix above is where textconv shows up, and where --no-textconv stops it.
The diff flags stopped diff.external, and nothing else.
core.fsmonitor=false and core.hooksPath=/dev/null are what a careful wrapper adds. They stopped fsmonitor and the hooks-directory hook, but not the hook defined in config, which is not looked up in a hooks directory. The clean filter also still ran.
Clearing the clean filter worked only because we knew the driver was called lab. An attacker picks the name, and .gitattributes can name a different driver for each file pattern. The same goes for smudge and process, which a wrapper has to clear too.
hook.<event>.enabled=false, which the git 2.55 git-hook documentation describes as a switch for every hook of one event, stopped the config hook in this fixture without knowing its name. We only tested it together with core.hooksPath=/dev/null, so keep both. It is also per event, so a wrapper has to list every event its commands can fire.

On git 2.39.5 the same script gave the same first four rows, minus the config hook, which that version does not know about.

Which copies bring the config along

scripts/delivery.sh builds one repo with core.fsmonitor, a clean filter and a post-index-change hook. All three call a helper stored inside .git by relative path, so the payload travels with any copy that includes .git. It copies the repo four ways, records anything that ran during the copy, and then runs git status and git diff in each copy. On the Pi:

git 2.39.5 on Linux aarch64
safe.directory already set on this machine: no
original repo copy ran: - then ran: filter-clean,fsmonitor,hook:post-index-change exit 0/0
git clone copy ran: nothing then ran: nothing exit 0/0
git clone from a bundle copy ran: nothing then ran: nothing exit 0/0
tar of the working copy copy ran: nothing then ran: filter-clean,fsmonitor,hook:post-index-change exit 0/0
same tar, owned by another user copy ran: - then ran: nothing exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.S42grY0m9P/other-owner'
  ...with no system or global config copy ran: - then ran: nothing exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.S42grY0m9P/other-owner'
  ...with -c safe.directory=* copy ran: - then ran: filter-clean,fsmonitor,hook:post-index-change exit 0/0

The clone and the bundle were clean, both during the copy and afterwards. The tar was not, and a tar of the working copy is what most "cache the workspace" setups produce: an actions/cache or GitLab cache: entry that includes .git, a build artifact someone zipped from the job directory, a devcontainer volume, a folder synced between machines. Once that copy lands, the next tool to run git status in it runs the repo's programs.

Here is that as a terminal session, from scripts/demo-restored-workspace.sh: a workspace restored from a tarball, two git commands, and our audit script at the end. The helper sits inside .git, so it arrived with the tarball, and the script empties ran.log before each git command.

restored workspace, git 2.39.5

# a workspace restored from a cache tarball, not cloned
$ git status --short
 M notes.txt
$ cat ../ran.log
ran: fsmonitor
ran: fsmonitor
$ git diff --no-ext-diff --no-textconv --stat
 notes.txt | 1 +
 1 file changed, 1 insertion(+)
$ cat ../ran.log
ran: fsmonitor
ran: fsmonitor
ran: filter-clean
ran: filter-clean
$ audit.sh .; echo "exit $?"
Repo config in . can run programs:
file:.git/config core.fsmonitor .git/lab-hook.sh fsmonitor
file:.git/config filter.lab.clean .git/lab-hook.sh filter-clean
exit 1

The ownership check, and the runners that turn it off

Since git 2.35.2 (the fix for CVE-2022-24765), git refuses to work in a repository owned by another user unless that path is allowed by safe.directory. On the Pi, that check stopped the poisoned tar as soon as another user owned it.

On GitHub-hosted runners, the same step ran everything. The script prints where safe.directory comes from, and on the Ubuntu runner (image ubuntu24 20260927.320.1) it said:

git 2.55.0 on Linux x86_64
safe.directory already set on this machine: system file:/etc/gitconfig *
original repo copy ran: - then ran: filter-clean,fsmonitor,hook:post-index-change exit 0/0
git clone copy ran: nothing then ran: nothing exit 0/0
git clone from a bundle copy ran: nothing then ran: nothing exit 0/0
tar of the working copy copy ran: nothing then ran: filter-clean,fsmonitor,hook:post-index-change exit 0/0
same tar, owned by another user copy ran: - then ran: filter-clean,fsmonitor,hook:post-index-change exit 0/0
  ...with no system or global config copy ran: - then ran: nothing exit 128/129 fatal: detected dubious ownership in repository at '/tmp/tmp.Y1kIjtRPwM/other-owner'
  ...with -c safe.directory=* copy ran: - then ran: filter-clean,fsmonitor,hook:post-index-change exit 0/0

The Ubuntu image writes directory = * into /etc/gitconfig, and the macOS image (macos26 20260907.0351.1 in our run) adds it to the global config; you can see both in the Ubuntu and macOS image scripts. The Ubuntu script's comment says why: git 2.35.2 "introduces security fix that breaks action\checkout". With system and global config switched off, git 2.55 refused the copy exactly like git 2.39 on the Pi, so the behaviour is git's and the wildcard is the image's.

Two caveats keep this in proportion. The ownership check only helps when the files belong to a different user; a cache that the job restores as itself passes the check with or without a wildcard. And a refusal stops git, it does not make the repository safe. Still, on those runners the ownership check is not part of your defence, so it comes down to whether you restore .git at all.

To see what your own runners and build containers do, print every entry with its scope and file. Git only honours safe.directory from system, global and command-line config, so a repo cannot allow itself. A * in any of those means the check is off for all paths, unless a later empty entry resets the list, and removing a global * does nothing if the system config still has one:

git --no-pager config --show-scope --show-origin --get-all safe.directory

What to do

Do not restore .git from a cache or artifact

Cache what is expensive to rebuild, not the repository. .git rarely needs caching: actions/checkout fetches a single commit by default, and a fetch never carries config or hooks. If you need full history for speed, cache a bundle (git bundle create) and clone from it; in our test, a bundle clone ran nothing.

This closes one route, not cache poisoning in general. A restored node_modules or provider directory can also contain code that runs, and GitHub's own cache security guidance says to treat restored caches as untrusted input. Keep caches separated by trust level, so a pull request job cannot write what the main branch job restores.

Treat artifacts the same way. If a later job downloads an artifact that contains a .git directory and runs any git command inside it, that job trusts whoever produced the artifact.

Audit before the first git command

When you cannot avoid a copied .git (a synced folder, a devcontainer volume, a repo an agent has had write access to), read its config before you run git in it. git --no-pager config --list reads config files without running any of the programs above; keep --no-pager, because a configured pager can run when the output goes to a terminal.

Our scripts/audit.sh wraps that. It lists repo-level keys whose value is a command, a hooks directory or an include (including hook.<name>.command, process filters, shell aliases and pager.<cmd>), plus executable hooks in the repo's hooks directory, and exits 0 (none), 1 (found) or 2 (it could not inspect the repo, for example because git refused it). scripts/audit-selftest.sh checks it against every fixture plus odd layouts, such as a driver name containing =, a symlinked hook, a linked worktree and an include outside .git, and confirms that auditing ran nothing. It is a detector for the settings it knows, not a safety certificate.

Run a copy you trust, kept outside the directory you are checking; a poisoned workspace can replace any script inside it. In CI, that means fetching the audit at a pinned commit before you restore anything:

# before restoring the cache: fetch the audit from a pinned commit, outside the workspace
curl -fsSL -o "$RUNNER_TEMP/audit.sh" \
  "https://raw.githubusercontent.com/The-DevOps-Daily/git-config-exec-check/<commit-sha>/scripts/audit.sh" || exit 1
# after restoring it
bash "$RUNNER_TEMP/audit.sh" "$GITHUB_WORKSPACE" || { echo "restored repo can run programs"; exit 1; }

Expect some findings on developer machines. git lfs install --local writes a filter driver into the repo config, and many teams use hooks on purpose. That is fine: the point is to see them before git runs them, in a place where you did not put them yourself.

If you write tools that shell out to git

GitLab's advice is to override every relevant key on every call, or avoid git's filter and textconv machinery. From our results, for the commands a tool runs to read state (status, diff, blame, log), that means:

Pass -c core.fsmonitor=false -c core.hooksPath=/dev/null on every call, including git status.
On git 2.55, also pass -c hook.<event>.enabled=false for each event your commands can fire. Our table saw pre-commit, post-commit, post-checkout, post-index-change and reference-transaction.
Add --no-ext-diff --no-textconv to diffs, and --no-pager to anything that might reach a terminal.
For filters, read the config first and clear clean, smudge and process for every driver it defines, or use commit-to-commit diffs when they are enough.
For fetch and push, the remote URL and core.sshCommand both come from the copied config. Set the transport yourself (for example -c core.sshCommand=ssh and an explicit URL) or do not let the tool talk to remotes from that copy.
When the repo came from outside, run git as a user that does not own it, with no safe.directory wildcard in system or global config.

These cover what we tested. They are not a complete list of every setting git can turn into a command, which is why the audit and the "do not copy .git" advice come first.

If you use one of the agents named in the two reports, update it. The DeepSeek-Reasonix fix is Studio 2.21.0 and npm 1.39.3. Manifold's post has a table of fixed versions; four of its eight findings were still unpatched when it was published.

What we did not test

Windows , credential helpers and editors. Each of those needs a different trigger.
A fetch that succeeds , and a process filter that speaks the protocol. Both can run more than our rows show.
Any specific agent. These scripts test git. How a given agent calls git decides which rows of the table apply to it, and the two reports above are the place to look for that.
Every command and every state. We picked 15 common commands and one kind of change. git rev-parse and git log --oneline ran nothing here; that is not a promise about other commands or other repo states, so run matrix.sh with the commands your tools use.

If you have read our posts on pre-commit hook security or the MCP design flaw, this is the same lesson from another side. The dangerous input is not always code you run on purpose. Sometimes it is the configuration of the tool you run, read from the directory you are standing in.

Originally published at devops-daily.com.

How Shopify Survives Black Friday: The Flash-Sale Playbook

DevOps Daily — Tue, 06 Oct 2026 09:00:00 +0000

Shopify merchants sold $14.6 billion over Black Friday Cyber Monday 2025, and the busiest minute, 12:01 p.m. EST on Black Friday, ran at $5.1 million in sales. A minute like that is not a surprise. It is on the calendar a year ahead, so the engineering problem is not reacting to a spike. It is rehearsing for one.

Shopify's engineering blog describes the rehearsal in unusual detail: load tests at 150% of last year's peak, capacity planned with the cloud providers months ahead, a throttle at the edge that queues buyers when checkout is full, and inventory reserved during payment so two buyers cannot claim the last unit. This post walks through each from Shopify's own write-ups, then rebuilds the checkout half on a Raspberry Pi with k6 to measure what the last two steps buy.

The short version: checking stock before payment oversold 41 to 47 units of 500 in three runs, and reserving the unit first sold exactly 500. When buyers arrived twice as fast as payment could take them, a plain queue charged more shoppers after they had given up than it confirmed purchases for. A simple waiting room kept checkout under 400 ms at p95 instead, at the cost of turning the overflow into shoppers who gave up waiting.

TL;DR

Shopify rehearses all year: its load generator, Genghis, runs checkout flows in production with flash-sale bursts on top, at 150% of last year's load, and capacity is planned months ahead.
When checkout is full, an edge throttle queues buyers. The first version was a lottery that left some waiting 40 minutes; a signed first-attempt timestamp made it fair.
Inventory is reserved when payment starts and claimed when it succeeds.
In our runs the oversell from check-then-pay was roughly arrival rate times payment time: lower traffic shrank it but did not remove the race. At twice the payment capacity, the question was not how many orders got through but who got them.
Our first batch of runs quietly skipped up to a fifth of its shoppers. Fail arrival-rate load tests on dropped_iterations.

Prerequisites

Comfort with HTTP, SQL transactions and the idea of a load test
To run the demo: Node.js 22.13 or newer (for the built-in node:sqlite) and the k6 binary; no Docker, no database server
About 25 minutes of machine time for the full set of recorded runs

A flash sale is an overload you can schedule

Bart de Water, who worked on Shopify Payments, defined the term in his QCon talk on Shopify's flash-sale architecture: "A flash sale is a sale for a limited amount of time, often with limited stock. It's over in a flash because the product can sell out in seconds, even if there are thousands of items in inventory." He also draws the line that matters here: "Storefront is mostly about read traffic, while our checkout does most of the writing and has to interact with external systems as well."

Reads cache well. Writes that call a payment provider do not, and that is where Shopify got hurt. The first of two posts on its checkout throttle describes Kylie Cosmetics running sales that sold out quickly, roughly every week, and one in February 2016 that "took down not just her store, but all others on the database shard where her store was allocated." What followed was a set of habits Shopify still writes about, and de Water's talk has the one-line reason they keep paying off: "today's flash sale will be tomorrow's base load."

Those habits answer three questions. Will the platform hold at the peak? What happens to buyers who arrive when checkout is full? Does the one write that matters stay correct under contention?

Load testing as a discipline, not a launch task

Shopify's 2025 BFCM readiness post opens with "Bimonthly fire drills all year, simulating 150% of last year's BFCM load." The tool is in-house:

Our load testing tool Genghis runs scripted workflows that mimic user behavior like browsing, cart adds, and checkout flows. We gradually ramp traffic to find breaking points. Tests run on production infrastructure simultaneously from three GCP regions (us-central, us-east, and europe-west4) to simulate global traffic patterns. We inject flash sale bursts on top of baseline load to test peak capacity.

Five major scale tests ran from April to October 2025. The fourth reached 146 million requests per minute and more than 80,000 checkouts per minute; the last went to the p99 forecast of 200 million requests per minute. Early tests found that "core operations threw errors and checkout queues backed up", and adding authenticated checkout "exposed rate-limit paths that anonymous browsing never touches."

Three details worth copying at any scale:

Production-shaped targets. De Water's talk describes Genghis hitting benchmark stores in production, at least one per pod, "at least weekly", paying through a benchmark gateway that "can respond with both successful and failed payments with a realistic distribution of response time latencies that we see in production." The demo copies that idea.
The burst, not just the volume. A 2023 post lists a flash-sale flow in which "simulated users purchase a single product", plus an "abort switch" that stops every test at once.
Written-down failure. Toxiproxy, Shopify's open source tool for simulating network conditions, injects failures during load tests, and findings go into a Resiliency Matrix of failure scenarios, recovery objectives and runbooks.

Pick the load model before the tool

What decides whether a load test can see a flash sale at all is the workload model. k6's page on open and closed models states the trap: "When the target system is stressed and starts to respond more slowly, a closed model load test will wait, resulting in increased iteration durations and a tapering off of the arrival rate of new VU iterations." A closed test slows down exactly when the system does and reports a calm result. Buyers at a drop do not wait for the previous buyer's page to load, so you need an open model, where arrivals follow a schedule whatever the server is doing.

All three common open source tools can drive an open model and fail a CI job; k6 and Gatling also have explicit closed-model options:

	k6	Gatling	Artillery
Scripts	JavaScript	Java, JavaScript, Kotlin, Scala	YAML, or JavaScript and TypeScript
Open model	`constant-arrival-rate`, `ramping-arrival-rate`	`constantUsersPerSec`, `rampUsersPerSec`	phases with `arrivalRate` and `rampTo`
Closed model	VU-based executors such as `constant-vus`	`constantConcurrentUsers`, `rampConcurrentUsers`	arrival-based phases; `maxVusers` only caps concurrency
Pass or fail	thresholds, non-zero exit	assertions: "If at least one assertion fails, the simulation fails"	`ensure` plugin, non-zero exit

On merit: k6 is a single Go binary whose thresholds work on custom metrics, which made "fail if any unit was sold twice" a one-line rule in the demo. Gatling separates open and closed injection in its API, so the model is explicit in every script, and suits JVM teams. Artillery's YAML scenarios are the quickest to write for plain HTTP, and its Playwright engine drives real browsers when the checkout page's JavaScript matters. Any of them could run the demo.

Pre-scale for the peak you know is coming

Shopify's 2025 preparation started in March with capacity planning, including "submitting our estimates to our cloud providers so they don't run out of cloud." In the uncertain year of 2020, Capacity Planning at Scale records the choice: "We decided to scale to our more aggressive growth scenarios to ensure our platform is stable regardless of what happens." Before scale tests, components are brought up to a "BFCM profile" ahead of time.

Change is managed the same way. A 2018 post describes a feature freeze that "starts several weeks before BFCM" and a code freeze a few days before. The 2025 post puts it as a rule: "We don't use BFCM as a release deadline." Architectural changes and migrations land months earlier.

Why not let autoscaling handle it? Shopify's posts do not say they turn it off, and we found no primary source that does. But in the demo below the stock is gone about six seconds into the sale, while a Kubernetes Horizontal Pod Autoscaler checks metrics every 15 seconds by default, before a new pod is even scheduled. Autoscaling suits the slow tides of a long weekend; for the first minute of a drop, the capacity has to exist already. Our Kubernetes HPA lab and horizontal vs vertical scaling simulator let you feel that lag safely.

Some capacity cannot be pre-scaled at all because it belongs to someone else, like a payment provider. The second experiment is about what to do when arrivals exceed it.

Put the queue at the edge, not inside checkout

After the Kylie outage, Shopify had a week before the next sale. Part I explains why a plain rate limit would not do: "For customers anything that looked like the website crashing would be interpreted as such." The team built a throttle into its Nginx and OpenResty edge tier:

The solution we landed on was to throttle users using a leaky bucket algorithm built into our edge tier. [...] The number of requests served would be reset every period (in our case, 5 seconds), and it was up to us to inform the client when to retry a rejected request.

Buyers over the limit saw a queue page that polled /checkout; those who got through received a signed cookie to skip the throttle for the session. The platform held, and then buyers complained of waiting up to 40 minutes for a 40-minute sale. The post admits that "in reality they were randomly polling the throttle", a lottery.

Part II fixed fairness without adding state to the edge. A buyer's first checkout attempt is stamped with a timestamp in a signed cookie, and each load balancer computes a threshold, "the virtual version of the 'Now Serving: 42' counters at delis", that lets the earliest timestamps through to the leaky bucket. A feedback controller moves the threshold; after simulations, the team kept only its proportional term.

Buyer clicks checkout
Edge throttle Nginx + Lua
Checkout writes + payment

Outcomes:

under capacity: signed cookie, straight through
over capacity: queue page, poll, earliest timestamp first

Five years later, de Water's 10 tips for resilient payment systems (2022) still describes "scriptable load balancers to throttle the amount of checkouts happening at any given time", with a waiting queue when demand exceeds capacity. It explains why with Little's Law and a sentence worth pinning above any capacity plan: "your application can't out scale the world." At the edge, a waiting buyer costs a cached page and a poll, not a worker, a database connection and a payment slot.

Protect the one write that must not go wrong

The 2025 post calls checkout, payment processing, order creation and fulfillment "critical journeys". Older parts of the architecture exist to keep them alive:

Pods. A pod "consists of a set of shops that live on a fully isolated set of datastores", which limits how far one shop's sale can reach. De Water adds that some extra-large merchants get a pod to themselves, and other systems stop a big merchant's flash sale from trying to "monopolize all the capacity" of a shared pod.
Semian. Shopify's circuit breaker and bulkhead library for Ruby starts from the fact that slow resources fail slowly, and while threads wait on one, "the slow resource has caused a cascading failure by occupying workers and therefore losing capacity."
Inventory reservations , the part the demo rebuilds.

Shopify's May 2026 post, We replaced Redis with MySQL for inventory reservations, says oversell protection works "by reserving inventory during payment processing", which it calls "a short hold that prevents two concurrent checkouts from claiming the same unit." Reserve holds items when payment starts; claim deducts them from the ledger when payment succeeds. The reservation used to be a Redis DECR, but then the claim step had to update the MySQL ledger and clean up Redis, two operations that could not be wrapped in one atomic step. In MySQL, the obvious schema failed: "A single row with a quantity column couldn't handle the contention." The shipped design uses one row per sellable unit, taken with SELECT ... FOR UPDATE SKIP LOCKED so concurrent checkouts take different rows, from a pool capped at 1,000 rows per item and location that a replenishment process refills. The post also ties this article's two halves together: "Slow reservations trigger throttling and a worse buyer experience." Remember the single-row detail; the demo's fix is the single-row version.

Rebuilding the checkout half on a Raspberry Pi

The demo is a small checkout service and one k6 scenario, built to answer two questions a skeptical reader can check: does reserving before payment matter at modest traffic, and what does a queue in front of checkout change when arrivals exceed what payment can process?

The-DevOps-Daily/flash-sale-playbook on GitHub

Everything ran on a 4-core Raspberry Pi 4 that was also doing other work, with k6 2.3.0 and the service on the same machine, so absolute numbers are small. Every variant ran the same scenario on the same machine, interleaved with the others, and each run records the load average before and after it.

The service is one Node.js 24 process, one product, and an SQLite database in memory-backed storage. One process stands in for a fleet: the await between the read and the write lets other requests run in between, as requests on separate servers would.
The payment provider is a stand-in that takes 100 to 400 ms (uniform, seeded per run) and declines 5%. In the second experiment it also has a fixed number of slots.
The scenario is a k6 ramping-arrival-rate executor with every VU created before the sale. Each iteration is one shopper. After the sale, teardown() asks the server what it sold and reports it as metrics, so thresholds fail the run on correctness, not just latency.

// k6/flash-sale.js, trimmed
export const options = {
  scenarios: {
    flash_sale: {
      executor: 'ramping-arrival-rate',
      startRate: 0,
      timeUnit: '1s',
      preAllocatedVUs: profile.vus,
      maxVUs: profile.vus,
      stages: profile.stages, // drop: 0 -> 300/s over 10s, hold 20s, down over 5s
    },
  },
  thresholds: {
    dropped_iterations: ['count==0'], // k6 kept its own schedule
    oversold_units: ['count==0'], // reported by teardown() from the server
    orders_to_clients_that_left: ['count==0'],
    'http_req_duration{name:checkout}': ['p(95)<2000'],
  },
};

Before the recorded runs, scripts/predict.mjs simulated every variant 400 times with an idealized copy of k6's arrival schedule (it has one extra arrival at the very end) and the same payment model, ignoring server CPU time. Its output from the start of the first batch, kept with the discarded runs, is identical to the one used here. Every recorded order count below fell inside the simulated 5th to 95th percentile range; a few latency figures landed just outside it, which is unsurprising for a model with no CPU time.

Experiment 1: check, pay, then write

The naive checkout is the order most people write first (simplified from server.mjs):

async function checkoutNaive(shopper) {
  const { stock } = q.readStock.get(SKU); // SELECT stock ...
  if (stock <= 0) return { status: 409 }; // sold out
  const payment = await authorizePayment(); // 100-400 ms; other requests run here
  if (payment === 'declined') return { status: 402 };
  q.decrement.run(SKU); // UPDATE inventory SET stock = stock - 1
  recordOrder(shopper);
  return { status: 200 };
}

The fix moves a conditional decrement in front of payment, so the database decides who gets each unit:

async function checkoutReserve(shopper) {
  // UPDATE inventory SET stock = stock - 1 WHERE sku = ? AND stock > 0
  const { changes } = q.reserve.run(SKU);
  if (changes === 0) return { status: 409 }; // sold out, no payment call
  const payment = await authorizePayment();
  if (payment !== 'paid') {
    q.release.run(SKU); // give the unit back
    return { status: 402 };
  }
  recordOrder(shopper);
  return { status: 200 };
}

The drop profile ramps from zero to 300 new shoppers a second over 10 seconds, holds for 20 and ramps down over 5: 8,249 shoppers for 500 units. Each mode ran three times, interleaved to spread background load across both:

Run	Checkout	Orders	Oversold	Stock counter after	Checkout p95
drop, seed 1	naive	544	44	-44	190 ms
drop, seed 2	naive	541	41	-41	174 ms
drop, seed 3	naive	547	47	-47	186 ms
drop, seed 1	reserve	500	0	0	257 ms
drop, seed 2	reserve	500	0	0	165 ms
drop, seed 3	reserve	500	0	0	168 ms
low, seed 1	naive	508	8	-8	369 ms
low, seed 1	reserve	500	0	0	372 ms

Every run started all its scheduled shoppers and used the same code, stamped by hash in its run.json. The p95 covers every checkout request, including the quick 409 sold-out answers most shoppers got.

Units sold beyond the 500 in stock

	Value	predicted mean	Series
Drop, naive, seed 1	44 units	42.5 units	Naive
Drop, naive, seed 2	41 units	42.5 units	Naive
Drop, naive, seed 3	47 units	42.5 units	Naive
Drop, reserve, seeds 1-3	0 units	0 units	Reserve
Low, naive, seed 1	8 units	6.6 units	Naive
Low, reserve, seed 1	0 units	0 units	Reserve

Recorded k6 runs on a Raspberry Pi 4. Tick marks show the mean of 400 simulated runs from scripts/predict.mjs. Reserve sold exactly 500 every time.

flash-sale-playbook

# two of the recorded runs, as scripts/run-all.sh started them
$ scripts/run.sh drop-naive-1 drop MODE=naive SEED=1
drop-naive-1: k6 exit 99, results in results/drop-naive-1
$ scripts/run.sh drop-reserve-1 drop MODE=reserve SEED=1
drop-reserve-1: k6 exit 0, results in results/drop-reserve-1
# what the server says it sold
$ jq -c '{mode: .config.mode, stockInitial, orders, oversold, stockNow}' results/drop-naive-1/server-stats.json
{"mode":"naive","stockInitial":500,"orders":544,"oversold":44,"stockNow":-44}
$ jq -c '{mode: .config.mode, stockInitial, orders, oversold, stockNow}' results/drop-reserve-1/server-stats.json
{"mode":"reserve","stockInitial":500,"orders":500,"oversold":0,"stockNow":0}
# why the naive run exited 99
$ grep -A14 THRESHOLDS results/drop-naive-1/k6-output.txt
  █ THRESHOLDS 

    dropped_iterations
    ✓ 'count==0' count=0

    http_req_duration{name:checkout}
    ✓ 'p(95)<2000' p(95)=189.78ms

    orders_to_clients_that_left
    ✓ 'count==0' count=0

    oversold_units
    ✗ 'count==0' count=44

Every naive drop run sold units that did not exist, 8 to 9% more than the stock, and left the counter negative, so nothing downstream would have stopped those orders. Every reservation run sold exactly 500. Latency could not tell them apart (most shoppers arrived after the sellout and got a quick 409), so a load test that only checked p95 and errors would have passed both.

The size of the oversell is predictable. Every shopper who passes the stock check before the counter reaches zero gets an order, and the counter only moves when a payment completes. So at the moment it hits zero, about (arrival rate × average payment time × share of payments approved) shoppers are still in flight. The 500th naive order landed about 6.2 seconds after the sale opened, with arrivals near 185 a second: 185 × 0.25 s × 0.95 ≈ 44. The simulation predicted a mean of 42.5, with 5th to 95th percentiles of 37 and 48; the runs landed at 44, 41 and 47.

Lower traffic shrinks it but does not remove the race: at a tenth of the arrival rate, the prediction was 4 to 9 and the run oversold 8 (1.6% of the stock). A slower payment step makes it larger. The bug is about how many payments are in flight when the last unit goes, not about Black Friday traffic.

Experiment 2: more buyers than payment can take

The second scenario is the BFCM shape: plenty of stock, arrivals beyond what checkout can process. Payment has 8 slots and averages 250 ms, so it completes about 32 checkouts a second (Little's Law: 8 / 0.25 s). The peak profile ramps to 64 shoppers a second, twice that, holds for 30 seconds and ramps down over 10: 2,559 shoppers. Each will spend at most 15 seconds trying to buy, in one slow request or several tries.

Three variants, one change each, on the reservation code:

Queue : no protection. Requests wait for a payment slot as long as it takes.
Deadline : the baseline a hostile reader would ask for. k6 sends the time the shopper gives up, and the server skips the charge (504) when less than the stand-in's maximum payment time, 400 ms, is left.
Waiting room : at most 8 checkouts in progress; everyone else gets 503 with Retry-After: 1 and retries after a jittered half to one and a half seconds, giving up when the next wait would take them past 15 seconds. This is Shopify's random-polling first version, the lottery, not the timestamp fix.

Three runs per variant, seeds 1 to 3	Queue	Deadline	Waiting room
Shopper saw a confirmation	1,028 to 1,071	1,794 to 1,834	1,603 to 1,642
Shopper saw a timeout	1,437 to 1,481	0	14 to 18
Shopper was refused at the deadline	0	636 to 670	0
Shopper gave up after "please wait"	0	0	826 to 847
Shopper saw a decline	50 to 61	89 to 96	77 to 91
Server charged a shopper who had left	1,375 to 1,408	0	10 to 14
Successful checkout request, p95	14.2 to 14.3 s	14.9 s	387 to 395 ms
Time to purchase, median	6.4 to 6.7 s	12.5 to 13.1 s	4.3 to 5.0 s
Most checkouts in progress at once	1,122 to 1,142	948 to 949	8
Waiting-room responses served	none	none	21,892 to 22,188
Predicted confirmations, mean	1,048	1,809	1,626
Predicted charges after leaving, mean	1,385	0	16

The first five rows add up to all 2,559 shoppers in every run. The charges in the sixth row overlap with the timeouts: in the queue variant, most shoppers who saw a timeout were charged anyway.

2,559 shoppers, payment capacity about 32 checkouts a second

	Value	predicted mean	Series
Queue: confirmed purchases	1049 shoppers	1047.9 shoppers	Confirmed purchase
Queue: charged after giving up	1389 shoppers	1384.8 shoppers	Charged after giving up
Deadline: confirmed purchases	1810 shoppers	1809.5 shoppers	Confirmed purchase
Deadline: charged after giving up	0 shoppers	0 shoppers	Charged after giving up
Waiting room: confirmed purchases	1622 shoppers	1626.1 shoppers	Confirmed purchase
Waiting room: charged after giving up	12 shoppers	16.2 shoppers	Charged after giving up

Mean of three recorded k6 runs per variant on a Raspberry Pi 4. Tick marks show the mean of 400 simulated runs from scripts/predict.mjs.

Through the 45 seconds of overload, every variant completed 29 to 31 orders a second, by the server's own timestamps. Payment was the limit and none of the three changed it. What changed was who got those orders, and how long everyone waited.

The queue turned the overload into charges nobody saw. The line for payment grew past 1,100 requests; once the wait passed 15 seconds, nearly every request was served after its shopper had given up. Of about 2,440 orders per run, roughly 1,390 went to shoppers whose browser had shown a timeout, more than the 1,050 or so who saw a confirmation. In a real shop, each is a card charged for a purchase the buyer thinks failed.

The deadline removed those charges and sold the most, about 1,810. The cost is time: the queue still settles near the patience limit, so the median buyer waited 12.5 to 13.1 seconds, with up to 949 requests open at once. In a thread-per-request server each would also hold a worker, the failure Semian exists to prevent.

The waiting room kept checkout fast: 8 in progress at most, under 400 ms at p95, and a median time to purchase of 4.3 to 5.0 seconds including the waiting. Between 826 and 847 shoppers per run gave up after close to 15 seconds of "please wait", and the server answered about 22,000 cheap 503s, around 8.6 per shopper. That is the trade Shopify made by moving the wait to a cached page at the edge.

It confirmed about 10% fewer purchases than the deadline queue, and the timestamps show where most of that gap built up. Both sold about 30 a second through the overload. After arrivals stopped, 50 seconds into the sale, the deadline queue kept serving its newest arrivals, who still had patience left, until about 62 seconds; the waiting room ran dry at about 58. Server-recorded orders after the 50-second mark account for about 154 of the roughly 175-order gap. The cause is this particular polling policy rather than waiting rooms in general: admission is random, so shoppers who had waited longest gave up first; a shopper gives up as soon as the next random wait would cross the 15-second limit, up to 1.5 seconds early; and a freed slot sits idle until someone's retry arrives. It is Shopify's Part I lesson in miniature: random retries spend the patience of the people who came first. The 10 to 14 late charges are shoppers admitted with less patience left than their payment took. Adding the deadline check to the waiting room should remove most of them; we did not run that combination.

The bug in our own load test

Our first full batch of runs, kept under results/discarded/, started with 50 pre-allocated VUs and let k6 create more on demand. In the peak runs every waiting shopper holds a VU; k6 could not create them fast enough and skipped the shoppers it could not start on time: 541, 542 and 440 of 2,559. The runs completed, the summaries looked plausible, and the queue variant looked far better than it should, because it had faced a lighter sale.

k6 reports this as dropped_iterations, and our summary script flagged it. The fix was to create every VU up front and fail the run on dropped_iterations: ['count==0']. Later, a burst of unrelated work on the Pi pushed the load average near 9 on four cores. One peak run in that window skipped 253 shoppers, and another started 2,557 of 2,559 without reporting any as dropped, which only surfaced when the summary script compared runs. Both were re-run once the machine was quiet and kept with a note. The rule for throwing a run away was incomplete delivery of the schedule, not load: peak-queue-3 started under a load average of 8.9, delivered all 2,559 shoppers, landed inside its predicted ranges and was kept. Before you believe anything an arrival-rate test says, check that it delivered its schedule.

What this does not show

Scale. A few hundred requests a second on one small machine, with k6 and the server sharing four cores. The direction of each result should hold; the absolute numbers do not transfer.
Hot-row contention. SQLite serializes writes, so the single-row conditional UPDATE never fought for a row lock the way it would in MySQL or Postgres. Shopify says that design "couldn't handle the contention" at its scale. The demo shows that reserving before payment is necessary, not that one row is enough.
The cost of a held request. Node.js holds a waiting connection cheaply. In a thread-per-request server, the queue and deadline variants would look worse.
A fair waiting room. Ours is the lottery Shopify replaced. A timestamp-ordered queue should change who gets served and narrow the spread of waits; we did not build one.
Real payment latency. The stand-in and the predictor use a uniform 100 to 400 ms with no long tail. Real providers have tails, which make every queue worse.
Rare cases. Three runs per variant (one for low) agree with each other, and their order counts sit inside the simulated ranges, but three runs cannot rule out an occasional bad one.

The playbook, condensed

Rehearse the shape, not just the volume. Model the drop as a burst on baseline traffic with an open-model load test, and run it on a schedule.
Make the load test check correctness. After the run, ask the system what it sold. Fail on oversells, on charges to buyers who had left and on dropped arrivals, not only on p95.
Have the capacity before the minute you need it. Forecast, agree capacity with providers, scale up ahead of the peak and stop risky changes well before it.
Bound the work that reaches checkout. Admit what payment and inventory can finish and park everyone else somewhere cheap, ideally at the edge and in arrival order.
Reserve before you charge, and release on failure. The database decides who gets the last unit, not the application's memory of an earlier read.

Shopify's own posts are worth reading in full, starting with the 2025 readiness program, the checkout throttle story and the inventory reservations post. For the neighbouring problems, our post on how Stripe avoids double-charging covers retries around the payment call, and the rate limit simulator shows how leaky and token buckets behave under bursts.

Originally published at devops-daily.com.

Touch Grass: 6 Project Ideas for People Who Live in a Terminal (Hacktoberfest DEV Challenge, Week 1)

DevOps Daily — Mon, 05 Oct 2026 20:01:41 +0000

The theme for Week 1 of the Hacktoberfest DEV Challenge is out, and it is a little personal for anyone in DevOps: Touch Grass. Build something with open-weight models or open-source AI that gets people off the screen and into the world. Submissions are due by October 11, 2026 at 11:59 PM PDT.

The organisers put it well: the best builds should make the screen the shortest part of the experience. That is a fun constraint for people whose job is mostly screens. This post gives you six project ideas with a DevOps flavour. Each one is small enough for a week of evenings, gets someone outside, and has a clear answer to the question the judges ask in every entry: why does open matter for what you built?

The rules in 30 seconds

From the challenge page and the Week 1 launch post:

The prompt: build something with open-source AI at its core: an open-weight model, an open-source agent harness or framework, local inference, or all three. The open pieces should be what makes the project work.
The theme: get people off the screen and into the world. Hiking, gardening, birding, run clubs, fall foliage: if it gets someone outside, it counts. Bonus points if you take it outside, use it, and tell us how it went.
A new project, built during the challenge. Pull requests to existing projects do not count this year.
The write-up matters most. Writing quality has the biggest weight in judging. Explain why open innovation matters for what you built: does it run with no internet, keep data off someone else's server, let you swap or fine-tune the model, cost nothing to run?
One submission per challenge. It is in the running for the overall prize and every partner category it qualifies for, but it can win only once per challenge. To enter a partner category, use that partner's technology and list the category in the Prize Categories section of your post. Missed the weekend challenge? Every week is a fresh start.

1. The pager-safe walk, for your on-call friend

The problem: on-call engineers stay inside because they are afraid of missing a page. A whole week indoors, for alerts that mostly need nothing.

What it does: answers one question before you leave: "How far can I walk and still be back at my laptop within 15 minutes?" It draws that area on a map, so the walk is planned in seconds and the rest is outside. While you walk, a small watcher reads incoming alerts and sends one short verdict to your phone: "stay out" or "head back, this one looks real". Plain rules decide whether a page needs you; the model only explains why in one sentence.

Why open matters: alert payloads are full of hostnames, customer names and internal URLs. With a local model, they never leave your machine. The walking area comes from open map data: OpenStreetMap plus an open-source routing engine such as Valhalla, which has an isochrone API for exactly this "how far in N minutes" question. Use pedestrian costing, and show the area as an estimate: hills and crossings make the real walk back slower.

Week scope: one isochrone request for your home address, one map page, one watcher that reads a webhook or a JSON file of alerts. Skip live paging integrations; a sample alert file is enough for the demo.

Partner fits: Temporal (a watcher that must not lose an alert is a good fit for a durable workflow), Mastra (orchestrate the steps over an open model).

Background: our on-call rotation and escalation guide explains which pages should wake a human, which is exactly the rule set your watcher needs.

2. Field notes from your camera roll, for the friend who hikes

The problem: after a hike, the photos sit unsorted on a phone, and the "what was that plant?" question never gets answered.

What it does: you drop a folder of hike photos on your laptop. A local vision model writes field notes: what each photo shows, with a confidence level, and the place it was taken (from the photo's GPS data). The result is a one-page trail log with a map. The screen time is five minutes after the hike, not during it.

Why open matters: photos with GPS coordinates show where you live and where you walk. With an open-weight vision model running locally (Gemma 4 takes text and image input), the photos never leave your laptop. If you add hosted search, say in the post that only the notes and embeddings go up, and drop the exact coordinates first.

Week scope: one folder in, one Markdown or HTML page out. Read EXIF with an existing library. Ask the model to say "not sure" instead of guessing a species, and show its uncertainty in the page.

Partner fits: Gemma (run it locally), Tiger Data or MongoDB Atlas (store notes and embeddings so you can search "all the mushrooms from September").

Background: none needed, just a recent hike. If you have not been on one, that is the first step of the project.

3. A cron job for going outside, for everyone on your team

The problem: "I'll go for a walk later" never happens, because later is a meeting.

What it does: reads your calendar (an .ics export), the weather forecast, and today's sunset time, and books the best 30 to 45 minutes outside before it gets dark. Then it learns from your history which suggestions you actually took. If you never go out at 13:00 on Mondays, it stops suggesting that.

Why open matters: your calendar shows who you meet and when. It stays on your machine. The weather comes from Open-Meteo, which is free for non-commercial use with no API key.

Week scope: a script that prints today's best remaining window and writes it back to your calendar as an event. The learning part can be a CSV of past suggestions with "taken" or "skipped".

Partner fits: TabPFN (its category is about predicting from a historical CSV, which is exactly your "taken or skipped" log), Gemma (write the suggestion in a friendly sentence).

Background: if cron syntax still surprises you, try the cron expression simulator before you schedule anything.

4. Alerting for tomatoes, for a friend with a garden

The problem: garden apps either nag every day or stay quiet until the plants are dead. It is the same alert fatigue we fight at work.

What it does: an Arduino UNO Q in the garden reads soil moisture and temperature, combines it with the local frost forecast, and pages you only when the garden needs you: "frost tonight, cover the peppers" or "the bed by the fence has been dry for three days". Everything else goes into a weekly summary. Apply the rule we use for production: page on symptoms, not on every reading.

Why open matters: the model runs on the board itself, with no cloud account and no subscription. The network is only needed to fetch the daily frost forecast and to send the message; the readings and the decisions stay on the board.

Week scope: one sensor, one plant bed, two alert rules, one weekly summary. A soil moisture sensor and a frost check are enough for a strong demo video.

Partner fits: Arduino (run a model on the UNO Q, sense and act), DigitalOcean (host the weekly summary page, if you want one outside your home network).

Background: our guide to SLOs, SLIs and error budgets is about services, but the same thinking decides when a tomato deserves a page.

5. A trail buddy that works with no signal, for your homelab friend

The problem: the best trails have no mobile signal, and that is exactly when you want to know how far it is to water, or when you must turn back to beat the sunset.

What it does: before the hike, it downloads what it needs: the route, water sources and shelters from OpenStreetMap (through the Overpass API), and the sunset time. On the trail, a Raspberry Pi or an old phone answers questions by voice: "How far to the next water?" "When do I need to turn back?" The distances and times come from plain code; the model only turns them into a short spoken answer.

Why open matters: it has to work with no internet at all. A small local model, local speech recognition such as whisper.cpp, and local speech output such as Piper make that possible.

Week scope: one trail, three questions, voice in and voice out. Test it in airplane mode before you leave home, then test it on the trail.

Partner fits: Gemma (the local model), SerpApi (a pre-trip check for trail closures and news while you still have signal).

Background: if you build this on a Pi, the Linux terminal simulator is a quick refresher on the commands you will need over SSH.

6. The walking standup, for a remote team

The problem: remote teams spend the whole day in calls. The daily standup could be the one meeting you take outside.

What it does: everyone records a 60-second voice note on their walk: what they did, what they will do, what blocks them. Back at the desk, a local model transcribes the notes and writes the standup summary for the team channel. Nobody has to look at a screen during the meeting itself.

Why open matters: team updates mention customers, incidents and people. With local transcription and a local model on a team server you control, they stay inside the team.

Week scope: a folder of audio files in, one summary out. Use your own notes for a week, then try it with a friend's team.

Partner fits: DigitalOcean (run the team server or an open-weight model on a GPU Droplet), ElevenLabs (transcription, if you are fine with sending audio to a hosted service, or a narrated version of the summary for the demo), Mastra (orchestrate transcription, summary and posting).

Background: keep the summary short and factual. The rule from our on-call guide applies here too: if nobody acts on a message, it should not be sent.

A starter for the weather-aware ideas

Ideas 3, 4 and 5 all start with the same two pieces: some facts from an open data source, and a local model that turns them into one useful sentence. With Ollama, Gemma 4 and Open-Meteo:

ollama pull gemma4:e4b

# today's sunset and today's hourly rain chance (example coordinates: Berlin)
WEATHER=$(curl -fsS "https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&hourly=precipitation_probability&daily=sunset&timezone=auto&forecast_days=1") || exit 1
NOW=$(TZ=Europe/Berlin date +%H:%M)

jq -n --arg w "$WEATHER" --arg now "$NOW" '{
  model: "gemma4:e4b",
  stream: false,
  messages: [
    {role: "system", content: "Suggest one 30 to 45 minute window to go outside today. It must start after the current time and end before sunset. Use only the weather data you are given. If no window fits, set possible to false."},
    {role: "user", content: ("Current time: " + $now + "\nWeather: " + $w)}
  ],
  format: {
    type: "object",
    properties: {possible: {type: "boolean"}, start: {type: "string"}, end: {type: "string"}, reason: {type: "string"}},
    required: ["possible", "start", "end", "reason"]
  }
}' | curl -fsS http://localhost:11434/api/chat -H 'Content-Type: application/json' -d @- | jq -e '.message.content | fromjson'

The format field constrains the model to that JSON shape, and the answer comes back as a JSON string in .message.content, which is why the example parses it with fromjson. In your own code, check the result against the data before you use it: the window must start after now, end before sunset, and fall in hours with a low chance of rain. After sunset there is no valid window, so expect possible: false. Better still, let plain code pick the window and use the model only for the reason.

How to fit it into one week

Monday or Tuesday: pick the person and the place. Write one sentence: "My on-call friend wants to go for a walk without worrying about pages." That sentence is your scope.
Midweek: build the ugly version. One input, one output, from the command line.
Saturday: take it outside. Use it on a real walk, hike or garden. What breaks out there is the best paragraph in your post.
Sunday: write. Writing has the most weight. Leave yourself at least three hours.

What to put in the write-up

Who it is for and what got them outside, in their words.
Why open, concretely. "It works in airplane mode on the trail" is stronger than "open source is important".
What happened outside. The organisers ask for this as a bonus, and it is also the most interesting part to read.
What did not work: a model that guessed a species with too much confidence, a route that ignored a river. Judges trust posts that show the hard parts.
A demo: photos from outside, a short video, or a terminal recording.
Your agent session, if you used one. The organisers suggest DevRelay. It is optional, but it helps judges see how you built it.

Next week

A new challenge starts every Monday in October, each with a new theme. We post DevOps project ideas for each one on the day it launches.

And if you want to practise classic open source pull requests this month (they do not count for the DEV Challenge, but they are still worth learning), our 7-day Hacktoberfest challenge gives you one small, reviewed PR a day.

What will you build to get outside this week? Tell us in the comments, and then go for a walk.

GITHUB_TOKEN Is Now 377 Characters Long. We Tested Which Redaction Rules Still Catch It

DevOps Daily — Mon, 05 Oct 2026 09:00:00 +0000

On October 2, GitHub finished the rollout of a new format for GitHub App installation tokens. By default, every newly minted ghs_ token, including the GITHUB_TOKEN that Actions gives each job, is now ghs_<app id>_<JWT> instead of a 40-character opaque string. GitHub's checklist tells you to look at length checks, database columns, proxies, and "logging and secret redaction rules that only match the legacy token pattern." We wanted to know how many of those rules actually break, so we ran 40 real tokens through the usual regexes, and one of them through three popular secret scanners and two databases.

Most of them missed. The latest gitleaks release, with its default rules, found nothing. The common hand-written patterns hid only the first 40 to 46 characters, and in our sample that part never changed from one token to the next, so the "redacted" log line still held everything an attacker needs.

TLDR

The Actions token we measured is 377 characters, not the "about 520" in GitHub's changelog. All 40 samples had the same length and shape: ghs_15368_ plus a three-part JWT signed with ES256.
The first 46 characters were identical in all 40 tokens. They are the app ID of GitHub Actions and a base64url JWT header that never changed. A rule that redacts only that part hides nothing secret.
Five common patterns redacted the whole token in 0 of 40 cases. Three never matched at all, so the full token would stay in the log. Two matched only the 40 or 46 public characters.
GitHub's own first recommended regex left the hyphen out. The May 15 version, ghs_[A-Za-z0-9\._]{36,}, fully matched 8 of 40 tokens, because 32 had a - in the JWT. GitHub corrected it on May 26, and the fixed regex matched all 40.
gitleaks 8.30.1 (latest release, March 21) reported 0 findings. trufflehog 3.97.9 found a 46-character fragment and marked it unverified. detect-secrets 1.5.0 flagged the line. Fixes for gitleaks and trufflehog have been open pull requests since July.
Storage fails loudly, except one case. Postgres and default MySQL 8.4 reject the token in a VARCHAR(255) column. MySQL with strict mode off stores the first 255 characters with only a warning.
Two custom rules, gitleaks and trufflehog, catch the full 377-character token today. Both are below and in the companion repo.

Prerequisites

Code, logs, or a database that handles GitHub App installation tokens or the Actions GITHUB_TOKEN
ripgrep (rg) to search a codebase for old patterns
Optional: a GitHub account to fork the test repo and run it against your own rules

What changed, and when

GitHub announced the change on April 24. The staged rollout started on April 27 with the Actions GITHUB_TOKEN and first-party integrations, then moved to all App installation tokens from mid-May to late June. The October 2 post marks it as complete.

What changed:

Format: ghs_APPID_JWT. The ghs_ prefix stays.
Length:"~520 characters" in GitHub's words, and it "will vary based on the data stored within it."
Contents: GitHub says the JWT "contains details about the token such as the target installation, the application, and basic validation details," and that clients must not validate it or depend on its contents.

What did not change: permissions, repository scoping, the one-hour expiration, and the REST endpoint that mints the token. GitHub Enterprise Server is not affected.

One date still matters. GitHub added a temporary X-GitHub-Stateless-S2S-Token request header in May so apps could force either format. Apps that send disabled to keep getting old tokens lose that option on November 30, 2026 , when GitHub stops respecting the header.

How we tested without leaking a token

We did not have a third-party GitHub App to mint tokens from, so we used the token every Actions job already has. GitHub's April notice says the new format covers "GitHub App installation server-to-server tokens, including Actions GITHUB_TOKEN." A workflow in a private test repository read secrets.GITHUB_TOKEN and passed it to small Python scripts that print only structure: lengths, separator positions, the decoded JWT header, the claim names, and how many characters each regex leaves visible. No step printed the payload values or the signature, and we checked the downloaded logs for any token value before making the repository public.

Two workflows ran on October 5, 2026, on ubuntu-latest:

check.yml: one token through the shape script, the regex list, gitleaks, trufflehog, detect-secrets, and Postgres and MySQL service containers.
sample.yml: 40 matrix jobs, one token each, to see whether the results hold across tokens.

This is the shape script's output from the recorded check run:

check.yml, Token shape step

# TOKEN is the job's GITHUB_TOKEN; the script prints structure only
$ python3 scripts/shape.py
length: 377
prefix: ghs_
separators (index, char): [(3, '_'), (9, '_'), (46, '.'), (290, '.'), (321, '-'), (360, '-')]
app id segment: 15368 (public: the app's numeric id)
segment lengths header/payload/signature: 36 243 86
chars used outside [A-Za-z0-9]: ['-']
header: {"alg":"ES256","typ":"JWT"}
payload claim names: ['aud', 'ctx', 'exp', 'iat', 'iss', 'jti', 'ver']
exp - iat (s): 3600
signature bytes: 64

Put together, a current GITHUB_TOKEN looks like this:

ghs_15368_eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.<243-character payload>.<86-character signature>
| ____________________________________________ |
  46 characters, the same in all 40 samples:
  ghs_ + 15368 (the app ID of GitHub Actions) + _ +
  the base64url of {"alg":"ES256","typ":"JWT"}

A few things we did not expect:

377, not 520. All 40 Actions tokens were exactly 377 characters. GitHub's "about 520" probably describes tokens for other apps, which carry more claims. We could not mint one, so we cannot confirm that. Plan for at least 520, as GitHub asks, and do not assume a fixed length either way.
The prefix is public. The 46-character head (ghs_15368_ plus the header segment) had one SHA-256 value across all 40 samples. It decodes to {"alg":"ES256","typ":"JWT"}, which anyone can encode.
The JWT uses base64url, so - and _ appear inside it. 32 of the 40 tokens had a -, 27 had a _, and only 3 had neither. This detail breaks the most regexes.
Actions masks the token in its own logs, but nobody else does. GitHub masks secret values in workflow logs and says that this redaction is not guaranteed. Your application logs, proxies, error trackers, and log pipelines do not get that masking at all.

The redaction test

For each pattern, the scripts ran re.search against the token and counted the characters outside the first match. Every pattern here starts with ghs_, which appears once per token, so that is also what a re.sub replacement would leave visible. The patterns are ones you find in real redaction code: GitHub's old documented shape, the regexes from the current gitleaks and trufflehog releases, the two regexes GitHub recommended in May, and the regexes from the open scanner fixes.

Pattern	Matched	Whole token redacted (40 tokens)	Most characters left visible
`ghs_[0-9a-zA-Z]{36}` (legacy shape)	never	0	377
`(?:ghu	ghs)_[0-9a-zA-Z]{36}` (gitleaks 8.30.1)	never	0
`\bghs_[A-Za-z0-9_]{36}\b`	never	0	377
`(?:ghu	ghs)[A-Za-z0-9]{36}`	first 40 characters	0
trufflehog 3.97.9 GitHub detector	first 46 characters	0	331
`ghs_[A-Za-z0-9\._]{36,}` (GitHub, May 15)	up to the first `-`	8	86
`ghs_[A-Za-z0-9\.\-_]{36,}` (GitHub, May 26)	whole token	40	0
gitleaks PR #2193	whole token	40	0
trufflehog PR #5156	whole token	40	0

Tokens fully redacted, out of 40 real GITHUB_TOKENs

	Value	Series
Legacy ghs_ + 36	0	Old shape
gitleaks 8.30.1 rule	0	Old shape
Word-bounded 36	0	Old shape
Underscore 36	0	Old shape
trufflehog 3.97.9 rule	0	Old shape
GitHub, May 15	8	New-format aware
GitHub, May 26	40	New-format aware
gitleaks PR #2193	40	New-format aware
trufflehog PR #5156	40	New-format aware

Each pattern run with Python re against 40 Actions tokens minted on 2026-10-05 (sample.yml in The-DevOps-Daily/ghs-token-check). 'Fully redacted' means re.sub would replace all 377 characters.

The failures fall into three groups.

Never matches, so the full token stays in the log. The legacy shape and the gitleaks rule expect 36 letters or digits right after ghs_. The new token has five digits (15368) and then an underscore, so the match fails at character 10. The word-bounded variant allows underscores, but its 36 characters end in the middle of the header, where no word boundary exists.

Matches the public prefix only. Allow underscores and the pattern matches ghs_15368_ plus 30 header characters. Make it open-ended ({36,255}, which is what trufflehog uses) and it runs to the first . and stops at 46 characters. Either way, the payload and the full signature stay in the log line. In our sample the part that disappeared was the same in every token, so anyone who reads the redacted line can paste the prefix back. The result is a working token for as long as it lives, which for an installation token is up to an hour.

Stops at the first hyphen. GitHub's May 15 changelog recommended ghs_[A-Za-z0-9\._]{36,} "to match both new and current format tokens." A snapshot from May 19 shows that version. An editor's note dated May 26 updated it to ghs_[A-Za-z0-9\.\-_]{36,}. The first version has no -, and base64url uses -. In our sample, the match stopped somewhere in the signature on 32 of 40 tokens and left 8 to 86 characters visible. For redaction this is the least bad failure, because the hidden part always includes the payload, and in most of those 32 tokens part of the signature too. In one sample the whole signature stayed visible. For validation it is worse: code that checks a token with fullmatch against the May 15 regex rejects every token that contains a hyphen. In our sample that was 80% of real tokens.

If you copied GitHub's regex in the second half of May, check which version you have.

What the scanners found

The check workflow wrote the token into a file as GITHUB_TOKEN=<token> and pointed the latest release of each scanner at it:

gitleaks v8.30.1
gitleaks findings: 0
trufflehog 3.97.9
trufflehog findings: 1
  Github verified False raw_len 46
detect-secrets 1.5.0
detect-secrets findings: 2
  JSON Web Token
  GitHub Token

gitleaks 8.30.1 found nothing. Its github-app-token rule is (?:ghu|ghs)_[0-9a-zA-Z]{36}, the first "never matches" case above. Someone reported the format change in issue #2192 on July 15, and PR #2193 with a fix has been open since July 16. The latest release is from March 21. If gitleaks 8.30.1 with its default rules guards your pre-commit hook or CI, its rule cannot match this token shape, and it did not flag our test file.
trufflehog 3.97.9 found a fragment. Its detector matched the 46-character prefix and reported that fragment as an unverified finding. Most examples in trufflehog's README run with --results=verified, and that filter drops this finding. PR #5156, open since July 27, adds a pattern for the new format.
detect-secrets 1.5.0 flagged the line twice, once as a JSON Web Token and once as a GitHub Token. Its GitHub pattern also matches only a prefix, but a scanner only needs to flag the line, and it did.

That last point explains why the same partial match is fine in one tool and bad in another. A scanner that matches 46 characters still points a person at the right line. A redactor that matches 46 characters prints the rest of the token.

Storing the token

GitHub asks for columns that "accept at least 520 characters." We inserted the 377-character token into narrow columns to see how each database reacts:

postgres t40:
ERROR: value too long for type character varying(40)
postgres t255:
ERROR: value too long for type character varying(255)
mysql 8.4 default sql_mode: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
mysql strict t255:
ERROR 1406 (22001) at line 1: Data too long for column 'tok' at row 1
mysql non-strict t255:
stored_len
255

Postgres and MySQL 8.4 with its default strict mode fail on insert, which is the good outcome: the error is in your logs the first time a new token arrives. MySQL with sql_mode cleared, which some older applications set on purpose, stores the first 255 characters and keeps going. The insert succeeds with a warning that is easy to miss, and the failure shows up later as a 401 from the GitHub API.

Installation tokens last an hour, so most apps cache them rather than store them for long. Look anyway: caches with fixed-size keys or values, cookie-backed sessions, and old migrations from when the token was always 40 characters.

What to change

1. Find the old patterns. Search code and config for anything that encodes the old shape:

# regexes that expect the old 36-character body, and anything mentioning ghs_
rg -n --hidden --glob '!.git' --glob '!node_modules' -e 'ghs_' -e 'gh\[[a-z]+\]_' -e '\{36\}' .

# columns that might hold a token and are too short
rg -n -i -e 'varchar\((40|64|100|128|255)\)' --glob '*.sql' --glob '*migration*' .

If your log pipeline or error tracker has its own scrubbing rules, check those too. A rule written for the old shape lives wherever someone pasted it.

2. Redact with a pattern that covers the whole JWT. GitHub's corrected regex works:

import re

# GitHub's recommendation as of May 26, 2026; matches old and new ghs_ tokens
GHS = re.compile(r"ghs_[A-Za-z0-9\.\-_]{36,}")

def redact(line: str) -> str:
    return GHS.sub("ghs_[REDACTED]", line)

It is greedy, so it also eats a trailing . or - that ends a sentence. For redaction that is the right tradeoff.

3. Do not validate the structure. GitHub says clients "must not take a dependency on the contents of this JWT." If you check tokens at all, check the ghs_ prefix and a generous maximum length, and let the GitHub API decide if the token is valid.

4. Patch your scanners until the upstream fixes ship. Both of these caught the full 377-character token in the recorded run (gitleaks with config/gitleaks.toml findings: 1, secret_len 377 and CustomRegex verified False raw_len 377):

Custom rules for the new installation token format

gitleaks

# .gitleaks.toml: keep the default rules, add one
[extend]
useDefault = true

[[rules]]
id = "github-app-installation-token-jwt"
description = "GitHub App installation token, stateless ghs_<app id>_<JWT> format"
regex = '''ghs_[0-9]+_eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'''
keywords = ["ghs_"]

trufflehog

# trufflehog --config trufflehog.yaml
detectors:
  - name: GitHubInstallationTokenJWT
    keywords:
      - ghs_
    regex:
      token: 'ghs_[0-9]+_eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'

A trufflehog custom detector without a verification endpoint reports its findings as unverified. A run with --results=verified drops them too, so let unverified results through for this detector.

5. Size storage for the documented length, not the one you measured. Use text in Postgres, or at least VARCHAR(1024) where you need a bound, and make sure MySQL runs in strict mode so a short column fails on insert.

6. Remove the override header. If your app sends X-GitHub-Stateless-S2S-Token: disabled to keep old-format tokens, that stops working on November 30, 2026. Fix whatever needed it before then.

Test your own rules

The workflows, scripts, and recorded results are public. Fork the repository, add your redaction or validation regex to patterns.txt, and run the workflows. The test uses the fork's own GITHUB_TOKEN, prints no token, and reports how much of a live token your rule leaves visible.

The-DevOps-Daily/ghs-token-check on GitHub

What we could not test

Tokens from other GitHub Apps. We measured only the Actions GITHUB_TOKEN. GitHub says other installation tokens are about 520 characters, and their payloads may differ. Every regex that covers the full JWT structure should still match, but we did not run one.
GitHub's own secret scanning and push protection. Testing them means pushing a live token to a repository, which we chose not to do, and it would say little about your own pipeline.
Commercial log scrubbers. We tested regexes and three open source scanners, not the built-in rules in Datadog, Splunk, or Sentry. Run your own provider's rule through the test repo if you can export it as a regex.
How long the prefix stays fixed. In our 40 samples the header segment never changed. GitHub could change the signing algorithm or add a header field at any time, so treat "the first 46 characters are public" as a property of today's tokens, not a guarantee.

Summary

GitHub warned about this change for five months and lists "logging and secret redaction rules" in its own checklist. The tools many teams use for that job have not caught up: the latest gitleaks release does not detect the new tokens, trufflehog sees only a fragment that it cannot verify, and the common hand-written patterns redact a prefix that, in our tests, was identical in every token. The fixes are small. Use a pattern that covers the whole JWT, add one custom rule to each scanner, give the column room for 520 characters or more, and test the result against a real token instead of a 40-character example in a unit test.

If you are reviewing the rest of your GitHub Actions setup, our posts on self-hosted runner version enforcement and on finding secrets before you have to rotate them cover two related changes.

Originally published at devops-daily.com.

Knowing What Your AI Feature Costs Before Finance Does

DevOps Daily — Mon, 05 Oct 2026 09:00:00 +0000

The invoice for your model provider arrives with one line per model. Finance reads it and asks a fair question: which feature spent that? If three features share one API key, the invoice cannot tell you, and neither can the provider dashboard. You find out the hard way, when one of them grows.

We wanted to know how far off the obvious guesses are, so we measured. Three features on one key, two models on DigitalOcean Serverless Inference, 151 recorded calls, and every number below comes from the usage block the provider returned. The feature we expected to be expensive was not the problem. The one-word classifier was.

TLDR

On Kimi K2.6, an incident answer the engineer read was about 100 tokens long. The request billed about 2,900 tokens, and 928 of them were reasoning tokens that nobody reads.
A one-word alert label used a median of 566.5 reasoning tokens on Kimi K2.6 and 184.5 on DeepSeek V4.1 Flash, with single calls as high as 1,590 and 1,783. On mean costs, one label cost between 32 and 48 percent of a full incident investigation.
Turning reasoning off for the label (reasoning_effort: "none" on Kimi K2.6) cut the cost per 1,000 labels 41 times at list price. It also changed 6 of the 20 labels.
The agent loop was not the multiplier we expected. In 19 of 20 runs the model asked for all four tools in one step, so an investigation took two calls, not the ten we had feared.
The same prompt used anywhere from 551 to 1,652 reasoning tokens across ten runs. Per-request cost is a distribution, not a number.
The fix is one wrapper: every call carries the feature and tenant that caused it, and writes the provider's usage in OpenTelemetry GenAI attribute names.

Prerequisites

Node.js 20 or later
An API key for an OpenAI-compatible endpoint that returns a usage block (we used DigitalOcean Serverless Inference)
A rough idea of per-token pricing: you pay one rate for input tokens and a higher rate for output tokens

The scripts, the fixtures, every raw response's usage and the report are in the companion repo:

The-DevOps-Daily/ai-feature-cost-ledger on GitHub

The invoice has the wrong shape

A cloud bill has the same problem, and the cloud answer is cost allocation tags. Model spend has no tags. The provider sees an API key and a model name, so that is what it bills by. Everything you would want to group by, such as the feature, the customer and the environment, exists only in your code at the moment you make the call.

There is a second problem, and it is the one that surprised us. Even per request, what a user sees has little to do with what you pay for. A request bills four things:

Input tokens : the system prompt, the conversation so far, tool definitions and tool results, every time
Cached input tokens : the part of the input the provider served from its prompt cache, at a lower rate
Output tokens : the answer
Reasoning tokens : thinking the model does before it answers, billed as output and usually never shown

The first and last are the ones people underestimate. So we built three small features that share one key and measured each.

The three features

All three run on the same key, which is the point: the provider sees one stream of calls.

Incident assistant. An engineer asks why checkout-api is returning 502s. The assistant has four tools: list pods, read logs, list recent deploys and read metrics. The tools return fixed text describing pods that are OOM-killed after a deploy raised a cache size, so every run sees the same incident. We ran it two ways: as a tool-calling agent, and single-shot with all four tool outputs pasted into one prompt. The system prompt asks for at most three short sentences.
Ticket summary. Two sentences for an account manager from a nine-message support thread.
Alert classifier. One word, page, ticket or ignore, for 20 different alert lines.

Each incident condition ran 10 times on each of two models, DeepSeek V4.1 Flash and Kimi K2.6. The classifier ran all 20 alerts under four settings. Both models report reasoning tokens and cached tokens separately in usage, which is why we picked them. All 40 incident answers identified the intended root cause (a keyword check for the memory exhaustion and the cache change behind it), so the cost comparisons below are between answers that found the right cause.

Prices are DigitalOcean's published Standard rates per million tokens, read from the pricing page on 5 October 2026:

Model	Input	Cached input	Output
DeepSeek V4.1 Flash	$0.30	$0.006	$1.20
Kimi K2.6	$0.95	$0.19	$4.00

What one answer actually billed

Here is the single-shot incident answer on Kimi K2.6, straight from the report file:

ai-feature-cost-ledger

# one incident answer on Kimi K2.6, median of 10 runs
$ jq '.incident["kimi-k2.6 single"] | {medianInput, medianOutput, medianReasoning, medianVisibleAnswer, medianBilledPerVisible}' data/report.json
{
  "medianInput": 1893,
  "medianOutput": 1023.5,
  "medianReasoning": 927.5,
  "medianVisibleAnswer": 102,
  "medianBilledPerVisible": 30.6
}

The engineer read a 102-token answer. The request billed a median of 1,893 input tokens and 1,024 output tokens, and 928 of the output tokens were reasoning. For every token the engineer read, the request billed about 30 (the median of the ten per-run ratios). On DeepSeek V4.1 Flash the same question billed about 17 tokens per token read, because it reasoned less: a median of 126 reasoning tokens against Kimi's 928.

One incident answer: what was billed vs what was read

	Value	Series
Input (prompt and tool output)	1893 tokens	Billed
Reasoning (billed as output)	927.5 tokens	Billed
Answer the engineer read	102 tokens	Read

Kimi K2.6, single-shot, median of 10 runs. Each bar is its own median, so the bars need not add up. Source: data/report.json in the companion repo.

Warning

Reasoning tokens are in completion_tokens, and you pay the output rate for them. If you estimate cost by counting the tokens in the answer text, you miss most of the output bill on a reasoning model. Read usage.completion_tokens_details.reasoning_tokens where the provider reports it.

The agent loop was not the problem

We expected the agent to be the expensive version. An agent resends the whole conversation on every step: system prompt, tool definitions, every earlier tool result. Ten sequential steps means paying for the first prompt ten times.

That is not what happened. In 19 of the 20 agent runs, the model asked for all four tools in its first step, received the results, and answered in the second. The one exception, a Kimi run, asked for three tools, then the fourth, then answered: three calls. With two calls, the resend costs one extra copy of a short prompt.

Model	Single-shot, median	Agent, median	Agent / single-shot
DeepSeek V4.1 Flash	$0.000926	$0.001518	1.64x
Kimi K2.6	$0.005892	$0.005871	1.00x

Those are list prices with no cache discount, which is the fair comparison here (the cache section explains why). On Kimi the agent was no dearer at all, because it reasoned less once the tool results were in front of it.

This does not mean agent loops are cheap. It means the step count decides, and the step count belongs to the model and the task, not to you. A model that calls one tool per step, or a task where each tool result decides the next call, turns two calls into eight. Measure steps per request as its own number, and alert on it.

The one-word answer that cost the most

The classifier is the feature nobody worries about. The prompt is roughly 50 to 90 tokens and the answer is one word. At list price, it cost $0.48 per 1,000 labels on DeepSeek V4.1 Flash with default settings, and $2.45 on Kimi K2.6.

Almost all of that is reasoning. The answer was 2 to 4 tokens every time. The reasoning was not:

Reasoning tokens spent on a one-word alert label

Series	Samples	Min	Median	p95	Max
DeepSeek V4.1 Flash, default	20	52 tokens	184.5 tokens	1218 tokens	1783 tokens
DeepSeek V4.1 Flash, effort=low	20	46 tokens	107.5 tokens	832 tokens	1138 tokens
Kimi K2.6, default	20	154 tokens	566.5 tokens	1168 tokens	1590 tokens
Kimi K2.6, effort=none	20	0 tokens	0 tokens	0 tokens	0 tokens

20 alerts per setting, one call each. The visible answer was 2 to 4 tokens every time. Source: data/feature-runs.jsonl.

The tail is where the money goes. A DNS failure rate of 14 percent for three minutes drew 1,783 reasoning tokens from DeepSeek before it said ticket. A node memory-pressure alert drew 1,218. On DeepSeek at default settings, 14 of the 20 alerts used between 50 and 250. Nothing in the request tells you in advance which call will be the expensive one.

Put next to the incident assistant, this is the number that changes priorities. Comparing mean costs, one label cost 32 to 48 percent of a full agent investigation : 32 percent on DeepSeek at list price, 48 percent on Kimi with the cache discounts both features got. A feature that runs on every alert is billed at a third to a half of a feature that runs when an engineer is paged.

Turning reasoning down, and what it changes

Both models accept a reasoning_effort parameter on this endpoint, but not the same values, and the values do not mean the same thing. DeepSeek V4.1 Flash rejected none and minimal (reasoning_effort must be one of [low high xhigh max] for this model) and accepted low. Kimi K2.6 accepted all three, but on the probe alert low and minimal used more reasoning than the default (1,210 and 1,021 tokens against 656), and only none turned it off. That is one call each, recorded in data/effort-probe.json, so read it as "check what the setting does", not as a rule.

ai-feature-cost-ledger

# the alert classifier on Kimi K2.6, default vs reasoning_effort none
$ jq '.classifier["kimi default"] | {runs, medianOutput, medianReasoning, maxOutput, medianVisible, costPer1000}' data/report.json
{
  "runs": 20,
  "medianOutput": 569.5,
  "medianReasoning": 566.5,
  "maxOutput": 1593,
  "medianVisible": 3,
  "costPer1000": 2.4195
}
$ jq '.classifier["kimi effort=none"] | {runs, medianOutput, medianReasoning, costPer1000, agreesWithDefault}' data/report.json
{
  "runs": 20,
  "medianOutput": 2,
  "medianReasoning": 0,
  "costPer1000": 0.0381,
  "agreesWithDefault": 14
}

Cost per 1,000 alert labels

	Value	Series
Kimi K2.6, default	2.4451$	Kimi K2.6
Kimi K2.6, effort=none	0.06$	Kimi K2.6
DeepSeek V4.1 Flash, default	0.4812$	DeepSeek V4.1 Flash
DeepSeek V4.1 Flash, effort=low	0.2671$	DeepSeek V4.1 Flash

Mean cost of 20 real calls per setting, scaled to 1,000, at DigitalOcean list prices on 5 Oct 2026 with no cache discount (costPer1000NoCache). Source: data/report.json.

With reasoning off, Kimi produced 2 output tokens per label. At list price, the cost per 1,000 labels fell from $2.45 to $0.06, 41 times less. As billed, with the cache discounts some of those calls got, it fell from $2.42 to $0.04, 63 times less. DeepSeek at low saved 1.8 times either way, and its tail did not go away: one call still used 1,138 reasoning tokens.

The catch is in the last field. Without reasoning, Kimi gave a different label on 6 of the 20 alerts. Four moved up to page: the disk at 91 percent, a node under memory pressure, a CI queue delay and a single OOM-killed pod. The other two moved from ignore to ticket. DeepSeek at low changed 2 labels. Each alert ran once per setting, so these are observed differences, not a measured rate, and we have no ground truth for the alerts, so we cannot say which setting was right. We can say that "turn reasoning off" is a product decision about who gets woken up, not a free saving. Run your own labelled set through both settings before you flip it.

Caching is a discount you cannot schedule

Prompt caching is the other big lever. DeepSeek V4.1 Flash bills cached input at $0.006 per million tokens against $0.30 uncached, 50 times less. On the nine single-shot runs that hit the cache, the median cost was $0.000342, against $0.000925 for the same tokens at list price.

Two things stop you from budgeting on that number:

The first call always paid full price. The first call of every incident condition had zero cached tokens, and so did the first call of the second Kimi agent run, even though the run before it had just sent the same prefix.
What repeats in our test does not repeat in production. We sent the identical incident ten times, so later runs found almost the entire single-shot prompt in cache (1,984 of 2,019 tokens on DeepSeek). A real incident brings new logs and new metrics. Only the shared prefix (system prompt and tool definitions, 576 tokens on DeepSeek here) can repeat across different incidents, which is why the agent comparison above uses list prices.

Record cache_read tokens per call and watch the hit rate as its own metric. Put the stable part of every prompt first and the per-request part last, so the prefix the cache can match is as long as possible. Then budget at list price and treat the cache as a discount, not a plan.

Tag every call with the feature that caused it

None of the numbers above helps finance unless each call says which feature it belongs to. The fix is boring, and that is the point. Every model call goes through one function. The function knows the feature and the tenant, reads the usage block from the response, prices it, and writes one row:

// src/ledger.mjs, trimmed
import { appendFileSync } from 'node:fs';
import { chat } from './client.mjs';
import { costUsd, splitUsage } from './cost.mjs';

export async function meteredChat({ feature, tenant, ledger }, body) {
  const res = await chat(body);
  const s = splitUsage(res.data.usage);
  const row = {
    ts: new Date().toISOString(),
    'app.feature': feature, // who caused this call
    'app.tenant': tenant, // who you would bill or rate limit
    'gen_ai.operation.name': 'chat',
    'gen_ai.provider.name': 'digitalocean',
    'gen_ai.request.model': body.model,
    'gen_ai.response.model': res.data.model,
    'gen_ai.usage.input_tokens': s.input, // includes cached tokens
    'gen_ai.usage.cache_read.input_tokens': s.cached,
    'gen_ai.usage.output_tokens': s.output, // includes reasoning tokens
    'gen_ai.usage.reasoning.output_tokens': s.reasoning,
    'app.cost_usd': costUsd(body.model, res.data.usage),
  };
  appendFileSync(ledger, JSON.stringify(row) + '\n');
  return res;
}

And the pricing function it calls, which is where most homemade cost trackers go wrong:

// src/cost.mjs, trimmed
export function splitUsage(u = {}) {
  const input = u.prompt_tokens ?? 0;
  const cached = u.prompt_tokens_details?.cached_tokens ?? u.cache_read_input_tokens ?? 0;
  const output = u.completion_tokens ?? 0;
  const reasoning = u.completion_tokens_details?.reasoning_tokens ?? 0;
  return { input, cached, uncached: input - cached, output, reasoning };
}

export function costUsd(model, usage) {
  const p = pricing.models[model]; // USD per million tokens, with source and date
  const s = splitUsage(usage);
  const cachedRate = p.cachedInput ?? p.input;
  return (s.uncached * p.input + s.cached * cachedRate + s.output * p.output) / 1e6;
}

Three details matter more than the rest.

Use the provider's numbers, not your own estimate. The usage block is what you are billed for. A tokenizer on your side counts the text you sent; it does not see reasoning tokens or what the cache served.
Split before you price. prompt_tokens already includes cached tokens and completion_tokens already includes reasoning tokens. Add them on top and you bill yourself twice for the same tokens.
Name the fields the way OpenTelemetry does. The GenAI semantic conventions define gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens and gen_ai.usage.reasoning.output_tokens, and state that the cached and reasoning counts are included in the totals, the same rule as above. The conventions are still marked Development, so pin the version you emit. The names still mean a JSON line today can become a span attribute later without a rename.

Once every row carries app.feature, the roll-up is a GROUP BY. The 151 calls in this experiment cost $0.19 as billed and $0.22 at list price. As billed, the 80 classifier calls cost more than the 41 agent calls; at list price, the agent calls were slightly ahead:

Feature	Calls	As billed	List price	Reasoning tokens
alert-classifier	80	$0.0641	$0.0651	23,500
incident-assistant (agent)	41	$0.0619	$0.0713	7,948
incident-assistant (single-shot)	20	$0.0566	$0.0733	11,831
ticket-summary	10	$0.0053	$0.0066	2,958

The mix of calls here is our test plan, not a production workload, so the shares mean nothing on their own. The shape is the lesson: the feature with the shortest answers spent the most reasoning tokens.

Where a gateway fits

You can stop at the wrapper. It is about 30 lines, it lives in your code, and it adds no one who sees your prompts beyond the inference provider. Keeping it consistent gets harder when the calls come from many services in several languages, and it reports spend without enforcing a budget. That is the problem the AI gateway and LLM observability vendors sell into, and they solve the tagging part in similar ways:

Portkey takes an x-portkey-metadata header with any keys you choose, such as feature and _user, and has budget limits in front of the provider.
Helicone uses one header per property, Helicone-Property-<Name>, and lets you segment requests and cost by them.
Langfuse records usage and cost per generation. It infers cost from model definitions it ships for OpenAI, Anthropic and Google models, so for a model like DeepSeek V4.1 Flash on DigitalOcean you add your own definition with the prices, or send usage and cost yourself. Ingested values take priority over inferred ones, and you should send them.
Braintrust logs traces with token counts, cost and metadata alongside its evaluations, which fits the question this post leaves open: whether a cheaper reasoning setting still labels alerts correctly.

The trade-offs are the usual ones. A proxy gateway adds a network hop to every model call, and a hosted one is another company that sees your prompts. Helicone, Langfuse and Portkey's gateway all publish source you can run yourself, which answers the second point and makes the first your problem. Whichever you pick, check that it records the provider's reported usage, including reasoning and cached tokens, rather than estimating from the text. Counting only what the user saw would have put the incident assistant's token bill far too low in this test: the median billed-to-read ratio per condition ran from 17 to 40, and single investigations from 14 to 50.

What we could not conclude

Whether failed attempts are billed. The 151 recorded calls all succeeded on the first attempt, so the client's retries for 429 and 5xx never fired in the record. A request that fails or times out on the client side may still reach the provider and be billed, and nothing we recorded can show whether it was.
Which classifier labels were right. We have no ground truth for the 20 alerts, so the 6 changed labels are a difference, not an error rate.
Whether reasoning helps the incident answers. All 40 identified the intended root cause by our keyword check, which is not a full accuracy review, so we cannot price the accuracy reasoning buys on this task. A harder incident might change that.
Anything about other models. Two models, one endpoint, one day. Others reason more or less, and the OpenAI GPT-5 models on the same pricing page were not available on our account tier, so we could not include them.
Steady-state cache hit rates. Our repeated prompts overstate hits for anything that changes per request.

Summary

Price every request from the provider's usage block. The answer text is not the bill.
On reasoning models, reasoning tokens can be most of the output bill, even for a one-word answer. Track reasoning_tokens as its own metric and look at the tail, not the median.
Steps per request decides what an agent costs. The median was 2 here: 19 investigations took two calls and one took three. Measure it and alert when it climbs.
reasoning_effort is a large lever and a product decision. Test the labels it changes before you ship it.
Budget at list price. Treat the prompt cache as a discount, put the stable prefix first, and watch the hit rate.
Wrap every model call once, tag it with the feature and tenant, and use the OpenTelemetry GenAI names. Then finance gets one line per feature, and you get it before they ask.

For two other ways a model bill grows quietly, see why most LLM calls are classification and what a semantic cache really saves.

Originally published at devops-daily.com.

A Repo per Agent: What Cloudflare Artifacts Changes About Git

DevOps Daily — Sat, 03 Oct 2026 09:00:00 +0000

Most Git workflows assume that changes come from people, arrive at human speed, and get reviewed by other people. Coding agents break all three assumptions. A team that runs fifty agents in parallel does not have fifty developers; it has fifty processes that may all want to land work in the same minute. When they all push to one branch, they spend much of that time fetching, rebasing and retrying.

On October 1, 2026, Cloudflare announced new capabilities for Artifacts , its Git-compatible store that you drive from Cloudflare Workers, together with a contest to build what comes next on top of it. Artifacts is in open beta on the Workers Paid plan. Its core idea is easy to say and has big consequences: stop sharing one repository between workers, and give each unit of work its own. This post explains what Artifacts is, measures why a shared branch stops working as agents multiply, shows the repo-per-agent pattern in code, and works out what it costs.

TL;DR

Artifacts is a Git-compatible repository store you create and control from a Worker. Any Git client can clone and push with a short-lived bearer token.
Cloudflare's own guidance is one repo per unit of autonomous work: 10,000 agents, 10,000 repos. Repos are cheap to create and fork.
In our local experiment, failed pushes grew roughly with the square of the number of agents sharing one branch: 40 agents produced about 700 failed push attempts.
A repo per agent removes the collisions but moves the hard part to merging the work back. Cloudflare left that layer open and is running a contest for it.
Pricing: 10,000 operations and 1 GB free per month, then $0.15 per 1,000 operations and $0.50 per GB-month. Each repo is capped at 1 GB.

Prerequisites

To follow the code:

A Cloudflare account on the Workers Paid plan (Artifacts is not available on the free plan)
Wrangler 4.145.0 or later, so wrangler types knows the Artifacts binding
Git 2.x on the machine that clones and pushes
Basic familiarity with Workers bindings and wrangler.toml

The push experiment below needs Git, Bash and standard Unix tools (bc, paste, sort).

What Artifacts is

An Artifacts namespace holds repositories. Each repository is a real Git remote: you get an HTTPS URL of the form https://<ACCOUNT_ID>.artifacts.cloudflare.net/git/<namespace>/<repo>.git, and any Git client can talk to it. Authentication is a bearer token passed as an extra HTTP header, so the token never lands in .git/config:

# Clone and push with a repo-scoped token; nothing is written to the remote URL
git -c http.extraHeader="Authorization: Bearer $ARTIFACTS_TOKEN" clone "$ARTIFACTS_REMOTE" work
cd work
# ...edit and commit...
git -c http.extraHeader="Authorization: Bearer $ARTIFACTS_TOKEN" push origin main

What makes it different from a hosted Git server is the control plane. A Worker gets a binding with methods to create, import, fork, list and delete repositories, mint read or write tokens with a lifetime in seconds, and read history and files without cloning:

Need	Binding call
New empty repo	`env.ARTIFACTS.create(name)`
Copy from GitHub or another remote	`env.ARTIFACTS.import({ source: { url }, target: { name } })`
Branch off a reviewed baseline	`repo.fork(name, { defaultBranchOnly: true })`
Credentials for one agent	`repo.createToken("write", 900)`
Inspect what an agent did	`repo.log({ ref: "main" })`, `repo.readFile({ ref, path })`
Clean up	`env.ARTIFACTS.delete(name)`

A few more pieces make it a platform rather than storage:

Workers Builds can deploy a Worker from an Artifacts repo. Pushes to main run the deploy command, and once you enable builds for preview branches, pushes to other branches produce preview URLs. Only main can be the production branch for now.
Repository events (creates, forks, pushes, clones and so on) can be delivered to Workers Queues, so a push can start a test run or a review agent.
Jurisdictions keep a namespace's data in the US or the EU.
Metrics per repository: operations, pulls, pushes and error rates.

Under the hood, Cloudflare describes each repo as a single logical instance that it can route to from any region, with data replicated synchronously across data centers and copied to object storage in the background. It has not published how concurrent pushes to the same ref are ordered, which matters for the next section.

Why one shared branch falls apart

Git protects a branch with a compare-and-swap. A push says "move main from commit A to commit B". If someone else moved main first, the server refuses, because B was built on a history that is no longer the tip. Two agents are enough to see it:

two agents, one branch

# both agents cloned the same baseline and made one commit each
$ git push origin main # in agent-a
To ../shared.git
   35bfd50..11799a8 main -> main
$ git push origin main # in agent-b
To ../shared.git
 ! [rejected] main -> main (fetch first)
error: failed to push some refs to '../shared.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Agent B now has to fetch, rebase and try again. That is fine for two humans. To see what happens with more writers, we ran a small experiment: N agents each commit one file (no two agents touch the same file, so the work never conflicts), are launched concurrently, and push to main of one shared repository, retrying immediately with git pull --rebase until the push lands.

#!/usr/bin/env bash
# N agents each commit one file and push to the same branch of one shared repo,
# retrying with fetch + rebase until the push lands. Prints total attempts.
set -u
N=$1; W=$(mktemp -d); cd "$W"
git init -q --bare -b main shared.git
git clone -q shared.git seed 2>/dev/null
(cd seed && git -c user.email=s@x -c user.name=seed commit -q --allow-empty -m baseline && git push -q origin main)
for i in $(seq 1 "$N"); do git clone -q shared.git "a$i"; done
agent() {
  cd "a$1"
  echo "$1" > "agent-$1.txt"; git add .; git -c user.email=a@x -c user.name="agent-$1" commit -q -m "agent-$1"
  tries=0
  until git push -q origin main 2>/dev/null; do
    tries=$((tries+1))
    git -c user.email=a@x -c user.name="agent-$1" pull -q --rebase origin main 2>/dev/null
  done
  echo "$tries" > ../fails-$1
}
for i in $(seq 1 "$N"); do agent "$i" & done; wait
total=$(cat fails-* | paste -sd+ | bc); max=$(cat fails-* | sort -n | tail -1)
commits=$(git --git-dir=shared.git rev-list --count main)
echo "agents=$N failed_pushes=$total worst_agent_retries=$max commits_on_main=$commits"
rm -rf "$W"

We ran it three times for each size on a 4-core Raspberry Pi 4 with Git 2.39 and a bare repository on local disk. The counter is failed push attempts; the script does not record why each one failed.

race.sh, three runs per size

$ for n in 5 10 20 40; do for run in 1 2 3; do ./race.sh $n; done; done
agents=5 failed_pushes=10 worst_agent_retries=4 commits_on_main=6
agents=5 failed_pushes=10 worst_agent_retries=4 commits_on_main=6
agents=5 failed_pushes=10 worst_agent_retries=4 commits_on_main=6
agents=10 failed_pushes=45 worst_agent_retries=9 commits_on_main=11
agents=10 failed_pushes=45 worst_agent_retries=9 commits_on_main=11
agents=10 failed_pushes=45 worst_agent_retries=9 commits_on_main=11
agents=20 failed_pushes=190 worst_agent_retries=19 commits_on_main=21
agents=20 failed_pushes=186 worst_agent_retries=19 commits_on_main=21
agents=20 failed_pushes=190 worst_agent_retries=19 commits_on_main=21
agents=40 failed_pushes=711 worst_agent_retries=36 commits_on_main=41
agents=40 failed_pushes=716 worst_agent_retries=37 commits_on_main=41
agents=40 failed_pushes=702 worst_agent_retries=38 commits_on_main=41

commits_on_main is the baseline plus one commit per agent, so every agent's work landed in the end. The cost is in the retries.

Failed push attempts when N agents share one branch

	5 agents	10 agents	20 agents	40 agents
Measured (median of 3 runs)	10	45	190	711
N(N-1)/2	10	45	190	780

race.sh on a Raspberry Pi 4, Git 2.39, local bare repository. No agents edited the same file, so every rebase succeeded; real conflicts make it worse.

A simple model explains the shape. If the agents move in rounds and only one push wins each round, every other pending push fails, and the total is N(N-1)/2. Our results sit close to that model. They fall below it at 40 agents, which is what you expect when an agent fetches several new commits in one rebase and skips some rounds. The model gives 499,500 failed pushes at 1,000 agents; we did not measure that scale.

Keep the limits of this test in mind. It is a conflict-free workload on local disk with immediate retries; backoff and spread-out arrival times would lower the numbers, and a hosted server adds a network round trip to every attempt. Real agents editing the same code would also hit merge conflicts that a retry loop cannot fix. And the work around Git has its own budgets: GitHub generally allows up to 80 content-generating requests per minute and 500 per hour across its API and web interface, and REST and GraphQL share a limit of 100 concurrent requests. Those limits apply to things like creating branches, pull requests and comments, not to the Git pushes counted here.

The usual answer is to give each agent its own branch in the shared repo. That removes the race on main, but the repository is still one shared object for permissions, clone size and blast radius. A token that can push to the repo can usually push to every unprotected branch in it.

The repo-per-agent pattern

Artifacts makes the next step cheap: a fork per session. A reviewed baseline repo holds the code you trust. Every agent session gets its own fork, a write token that only works on that fork and expires in minutes, and nothing else.

baseline reviewed main
agent-a-s41 fork + 15 min token
agent-b-s42 fork + 15 min token
agent-c-s43 fork + 15 min token
Orchestrator tests, review, merge

Connections:

baseline -> agent-a-s41 (fork)
baseline -> agent-b-s42 (fork)
baseline -> agent-c-s43 (fork)
agent-a-s41 -> Orchestrator (push event)
agent-b-s42 -> Orchestrator (push event)
agent-c-s43 -> Orchestrator (push event)

The Worker that hands out sessions is short. Configure the binding:

name = "agent-sessions"
main = "src/index.ts"
compatibility_date = "2026-10-02"

[[artifacts]]
binding = "ARTIFACTS"
namespace = "agents"

Then fork the baseline per session and return a scoped, short-lived token. This assumes a repo named baseline already exists in the namespace with your reviewed code on main (create it with create and push, or import it from GitHub), and that you ran wrangler types so Env includes the binding and the secret:

// src/index.ts: one fork and one 15-minute write token per agent session
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method !== 'POST') return new Response('POST only', { status: 405 });
    // This endpoint hands out write tokens: only your orchestrator may call it.
    // ORCHESTRATOR_SECRET is a Worker secret (wrangler secret put ORCHESTRATOR_SECRET).
    if (request.headers.get('Authorization') !== `Bearer ${env.ORCHESTRATOR_SECRET}`) {
      return new Response('Unauthorized', { status: 401 });
    }
    const { agent, session } = (await request.json()) as { agent: string; session: string };

    // Keep names readable but unique: validate the inputs and add a random suffix
    if (!/^[a-z0-9]{1,24}$/.test(agent) || !/^[a-z0-9]{1,24}$/.test(session)) {
      return new Response('agent and session must be lowercase letters and digits', {
        status: 400,
      });
    }
    const name = `${agent}-${session}-${crypto.randomUUID().slice(0, 8)}`;

    // `using` disposes the repo handle when the block ends, as the binding requires
    using baseline = await env.ARTIFACTS.get('baseline');
    const fork = await baseline.fork(name, { defaultBranchOnly: true });

    using repo = await env.ARTIFACTS.get(name);
    const token = await repo.createToken('write', 900); // 900 s = 15 minutes

    // plaintext is the Git token string; expiresAt tells the agent when to stop
    return Response.json({
      name,
      remote: fork.remote,
      token: token.plaintext,
      expiresAt: token.expiresAt,
    });
  },
};

The agent clones the remote, works, and pushes with the token. When it is done, the orchestrator does not need to clone anything to see what happened:

// Inspect a finished session without cloning it
async function summarize(env: Env, name: string) {
  using repo = await env.ARTIFACTS.get(name);
  const commits = await repo.log({ ref: 'main', limit: 20 }); // newest first
  const plan = await repo.readFile({ ref: 'main', path: 'PLAN.md' });
  return { commits, plan: plan ? await plan.text() : null };
}

Cloudflare's best-practice notes add three habits worth copying:

Give each token the least it needs, for as little time as possible. Read tokens for indexing and review, write tokens only for the agent doing the work.
Fork from a reviewed baseline instead of copying files into each new repo, so every session starts from the same known state.
Keep run metadata out of the tree. Attach prompts, model output and run IDs with git notes, so the commit holds the work and the notes hold the context.

And one of our own: delete forks once their work is merged or rejected. Storage is billed, and a pile of abandoned agent repos is the 2026 version of a pile of abandoned feature branches.

Getting the work back is the hard part

A repo per agent does not make conflicts go away. It moves them from push time, where they cost retries, to integration time, where something has to decide what lands. That is a better place for them, because you can be deliberate there, but someone still has to build it. Cloudflare says so directly: Artifacts is the storage primitive, and its contest asks developers to build the coordination, review and merge layer on top.

The options teams use today, from simplest to most ambitious:

Orchestrator merges in order. One process fetches finished forks, rebases each onto the baseline, runs the tests and pushes. It is a merge queue with exactly one writer to the baseline, so the race from the experiment above cannot happen.
Human review gate. Each fork's push event creates a review item. With Workers Builds, a branch push can also produce a preview URL to look at before anything merges.
Agent reviewers. A second agent with a read-only token reviews the diff and either approves it into the queue or sends it back. Keep the merge itself with the orchestrator, so no reviewing agent ever holds a write token to the baseline.
Best-of-N. Fork several sessions from the same baseline for the same task, test them all, and merge only the winner. Forks are cheap enough that this stops being wasteful.

None of these are new ideas. What changes is that the per-session repository, its credentials and its cleanup are now an API call instead of something you script around a hosted Git service.

What it costs

Artifacts bills two things. The numbers below are from the pricing page. It lists October 14, 2026 as the day billing starts; the announcement says October 15:

Dimension	Included each month	Then
Operations (create, push, pull, clone and similar)	10,000	$0.15 per 1,000
Storage	1 GB-month	$0.50 per GB-month

To size it, assume one agent session costs seven operations: a fork, a clone, three pushes, one fetch by the reviewer and a delete. That is an assumption, not a published figure; count your own workflow before you budget.

Estimated monthly operations cost by agent sessions per day

	Value
100/day	1.65$
1,000/day	30$
10,000/day	313.5$
100,000/day	3148.5$

Assumes 7 billable operations per session, 30 days, and the 10,000 free operations a month. Excludes storage and other Cloudflare charges.

Storage depends on your repos and how quickly you delete forks. It is billed on the average of each day's peak, so deleting a fork stops it from adding up over the month but does not remove that day's peak. Cloudflare has not documented whether a fork shares objects with its parent or counts its full size, so measure that in the beta before you plan around hundreds of long-lived forks.

Limits and open questions

Know these before you commit:

1 GB per repository and 32 MB per file or blob. A repository above 1 GB, or one with a single file above 32 MB, does not fit. Account storage is 1 TB by default and can be raised.
Rate limits of 2,000 requests per 10 seconds per namespace for the control plane, and 2,000 Git requests per 10 seconds per repository. Split busy workloads across namespaces.
Open beta. The API and limits can still change, and it requires the Workers Paid plan.
Production deploys from main only in the Workers Builds integration.
Undocumented so far: how concurrent pushes to one ref are ordered, whether forks share storage, and what Artifacts guarantees about publishing events. Queues itself delivers at least once and without ordering guarantees, so make event handlers idempotent. Test the rest before you rely on it.

Note

The contest runs until October 14, 2026. Entries need a 5 to 10 minute demo video, open source code under MIT, Apache or BSD, and instructions to run it. Up to two members of each of the three winning teams are flown to Cloudflare Connect in San Francisco, and first place also gets $25,000 in Cloudflare credits. Details are on Cloudflare's announcement.

Summary

Shared branches assume few writers. Our small experiment shows what happens when that assumption fails: with immediate retries, failed pushes grew roughly with the square of the number of agents, before a single real conflict appeared. The fix is not a faster retry loop. It is isolation, and Artifacts makes isolation an API call: fork a reviewed baseline per session, hand the agent a 15-minute token that works on nothing else, and read its work back without cloning.

What Artifacts does not give you is the decision about what merges. Plan that layer first. A single-writer merge queue plus tests is enough to start, and it is the part where your team's judgment matters most. If you want to start small, move one agent workflow, such as dependency updates or test generation, to forks of a baseline and watch the error rate and the bill for a month. The Artifacts documentation covers the binding, tokens and Workers Builds setup.

Originally published at devops-daily.com.

We Took Down Our Mesh VPN Control Plane. Here Is What Kept Working

DevOps Daily — Sat, 03 Oct 2026 09:00:00 +0000

When you replace a VPN concentrator with a mesh VPN such as Tailscale, NetBird or ZeroTier, traffic stops flowing through one box. Devices connect to each other directly wherever the network allows it. But the mesh still has a centre: the coordination server (Tailscale calls it the control plane) that hands out keys, addresses, peer lists and access rules. The old question "what happens when the VPN gateway dies?" becomes "what happens when the control server is unreachable?"

The vendors' answer is reassuring: the data plane keeps working. We wanted to know what that means in practice, so we ran an open-source control server (Headscale) with real Tailscale clients, stopped it, and probed the network about once a minute for 30 minutes. Existing tunnels survived the whole outage. Joining, revoking and renewing did not, and what happens when a client restarts during the outage is the result most worth adding to your runbook.

TLDR

During a control outage	What we measured
Existing tunnels	Kept working for the full 30 minutes: 27 of 27 probes in each direction
A new device joins	Failed (`NeedsLogin`); it joined by itself 14 to 22 s after control came back
An admin revokes a device	Impossible: the admin API is the control server
A device's key expires	The first probe after the expiry time failed; peers enforce it locally
A client restarts (no netmap cache)	Came back with no address and stayed off the network until control returned
A client restarts (netmap cache on)	Reachable again in 15 to 19 s in 11 of 12 timed restarts (one took 140 s); stayed reachable in two 10-minute checks, dropped once in another run

With the control server up	What we measured
Policy change blocks a connection	The first probe after the change already failed (within 6 s, mostly our probe timeout)
Deleting a device	Same: blocked at the first probe

Prerequisites

A Linux machine with tailscale and tailscaled installed (we used 1.102.4)
Basic familiarity with Tailscale or another mesh VPN

Everything runs as a normal user on one host, and it does not touch an existing Tailscale install. The scripts are in the companion repo:

The-DevOps-Daily/tailnet-control-outage on GitHub

The setup

One Headscale process plays the control server. Five tailscaled processes play devices, each in userspace-networking mode (no TUN device and no root, so they can share one host). Each node serves hello from <name> on a local port, and a probe fetches that page through another node's SOCKS5 proxy. A probe only succeeds if traffic really crossed the tailnet.

Each node has one job:

Node	Job during the outage
a, b	A long-lived pair. Never touched.
c	Tries to join.
d	Gets restarted.
e	Its key expires two minutes in.

a SOCKS5 probe
WireGuard direct path
b hello from b

Control plane stopped during the test
- Headscale 0.29.4 keys, peers, policy
Devices tailscaled 1.102.4, userspace mode
- a, b long-lived pair
- c joins mid-outage
- d restarted mid-outage
- e key expires mid-outage

The outage script stops Headscale and then works through a timeline: probe every node, start c (its join attempt runs from about 25 seconds to 70 seconds), try an admin command, let e's key expire at about two minutes, and restart d at five to six minutes, probing roughly once a minute throughout. Another script then brings the control server back and times the recovery.

What kept working: the tunnels you already had

The long-lived pair never noticed. In the 30-minute run, a reached b and b reached a on every probe, 27 probes in each direction, the last one 1,793 seconds after the control server stopped. (In one of the shorter runs, a single probe from b to a failed once and the next one succeeded.) Here is the start and the end of that run, from runs/nocache-30min/02-outage.txt:

16:34:16 t=0s control server stopped (e's key expires at 2026-10-02T16:36:10Z)
16:34:16 t=0s a->b ok (hello from b) | b->a ok (hello from a) | a->d ok (hello from d) | a->e ok (hello from e)
16:34:41 t=25s a's own status line: offline
...
17:03:09 t=1723s a->b ok (hello from b) | b->a ok (hello from a) | a->d FAIL | a->e FAIL
17:04:19 t=1793s a->b ok (hello from b) | b->a ok (hello from a) | a->d FAIL | a->e FAIL

This matches what Tailscale documents: every node keeps its peers, their endpoints and the packet filter, and traffic never goes through the coordination server in the first place. Each probe is a fresh HTTP connection, so this shows that new connections between existing peers keep working; we did not hold one long-lived session open.

Note the third line. While a was moving traffic, tailscale status showed a itself as offline, because a could not reach the control server. If your monitoring alerts on that, it will tell you the mesh is down while it is working. Probe real traffic between real nodes instead.

What stopped working

New devices cannot join

Node c started with a valid, reusable pre-auth key and never got past NeedsLogin:

t=71s new node c tries to join: timeout waiting for Tailscale service to enter a Running state; check health with "tailscale status" (state NeedsLogin)

It kept retrying on its own, and got an address 14 to 22 seconds after the control server came back (three runs). Nobody had to touch it. During the outage, though, a replacement laptop, a new CI runner or an autoscaled node simply cannot get on the network.

Nobody can revoke anything

The admin interface is the control server. The headscale nodes list command we ran to find a node to remove failed with context deadline exceeded. With Tailscale's hosted service the equivalent is the admin console and API, and they are part of the same control plane.

This is the security cost of the design. Tailscale's own documentation lists it: during an outage, "existing users cannot have their keys revoked." If you need to cut off a stolen laptop or a departing employee during a control plane incident, the mesh cannot do it for you. The device keeps every peer and every rule it had when the outage started.

Expiring keys still expire

We set e's key to expire about two minutes into the outage (114 seconds in the 30-minute run). The first probe after that time failed, and every probe after it. The client does this itself: it marks peers whose key has expired and stops talking to them (ipn/ipnlocal/expiry.go in the client source).

So key expiry is enforced by the peers themselves, without the control server. That is good for security, and it means a control outage can become a data plane outage: every device whose key expires during the outage drops off, and cannot renew until control is back. After recovery, e stayed Logged out until we cleared its expiry on the server, and then it came back within 1 to 2 seconds. In real life an admin extends or clears the expiry, or the user re-authenticates.

If your tailnet uses short key expiry for servers, a long control outage will take them off the network one by one. Tailscale lets you disable or temporarily extend expiry per device, and a device that is tagged when it first authenticates has key expiry disabled by default (key expiry docs).

The restart trap

Five minutes into the outage we restarted d's tailscaled, keeping its state directory, the way an unattended upgrade, a reboot or a crashed process does.

Without the netmap cache, d came back with no address and no peers:

t=317s restarted d's tailscaled while control is down: state NoState, ip , netmap cache files: 0

Its health check said why (from d's log, in log-excerpts.txt in the repo):

You are logged out. The last login error was: fetch control key: Get "http://127.0.0.1:18080/key?v=142": dial tcp 127.0.0.1:18080: connect: connection refused

d stayed off the network for the rest of the outage. The client keeps its keys on disk, but without the cache it does not keep the network map: the list of peers, their addresses and the packet filter only live in memory. Without the control server, a restarted client knows who it is but not who anyone else is. It kept retrying, and once the control server was back it logged in again by itself, within 3 to 4 seconds according to its own log.

During a control incident, the devices that restart are exactly the ones that disappear. A device that is replaced rather than restarted, such as a new Kubernetes node or a container without persistent state, is in the same position as c: it has never been on the network and cannot join until control is back.

Netmap caching fixes it

Tailscale has a fix: netmap caching. With it, each client writes its network map to disk and loads it at startup. Tailscale's September 22 post says it is a feature flag in the current client and is expected to be on by default from version 1.104, after more testing (mobile clients later). The cache only helps a device that restarts with its state directory intact; a fresh replacement has nothing cached.

In the 1.102 client we tested, the switch is on the server side. The client only writes the cache when the control server grants it the cache-network-maps node attribute. The TS_USE_CACHED_NETMAP environment variable defaults to on and works as an off switch. Headscale 0.29 passes node attributes through from its policy file, so turning it on was one block:

{
  "acls": [{ "action": "accept", "src": ["lab@"], "dst": ["lab@:*"] }],
  "nodeAttrs": [{ "target": ["*"], "attr": ["cache-network-maps"] }]
}

d then kept nine files under its state directory (profile-data/<id>/netmap-cache/), covering itself, its peers, the user, the packet filter, the DERP map and DNS. With the cache on, the same restart came back like this:

Start: loaded netmap from disk cache; 3 peers
t=352s restarted d's tailscaled while control is down: state Running, ip 100.64.0.3, netmap cache files: 9

The first line is from d's log (log-excerpts.txt), the second from the outage script, both from the same 10-minute run.

d had its address immediately, but peers could not reach it straight away. A restarted client gets a new disco key (the key Tailscale uses for path discovery), and peers normally learn it from the control server. With the cache, the client advertises the new key to its peers directly over the tunnel instead; the logs show sending TSMP disco key advertisement and the peers receiving it. We timed how long until a could reach d again, with a retrying constantly (a failed attempt times out after 5 seconds, and the next one starts a second later):

Restart (netmap cache on)	Port	Outage before the first restart	a reaches d again after
3 restarts	new random port	under 1 minute	19 s, 16 s, 15 s
3 restarts	same fixed port	under 1 minute	19 s, 15 s, 15 s
3 restarts	new random port	6 minutes	140 s, 15 s, 17 s
3 restarts	same fixed port	6 minutes	18 s, 15 s, 16 s

Time until a peer reaches a restarted node, control server down

Series	Samples	Min	Median	p95	Max
new random port	6	15s	16.5s	140s	140s
same fixed port	6	15s	15.5s	19s	19s

12 restarts with the cache-network-maps attribute, Tailscale 1.102.4 and Headscale 0.29.4. Measured from when the restarted daemon answered, with a new attempt about every 6 s. Without the cache, the node was still unreachable after 300 s.

Eleven of the twelve restarts were reachable again in 15 to 19 seconds. One took 140 seconds, on a clean setup, so slow recoveries do happen. The restarts run one after another, so only the first in each row waited the full outage time; within that limit, a longer outage made no consistent difference, and neither did keeping the same UDP port.

Does the recovered path last? We restarted d once more and probed every 30 seconds for 10 minutes, once with a fixed port and once with a random port. Both times the first probe, right after the restart, failed, and the next 19, over the following 10 minutes, all reached d. In our 10-minute cache outage run, though, d answered for two minutes after its restart, then failed two probes in a row (at 484 and 559 seconds) and only answered again at 630 seconds, just before control returned. We could not reproduce that, so treat a cached restart as a big improvement, not a guarantee.

Without the cache, the same test (fixed port) gave up after 300 seconds.

The cache has a security side. The packet filter is on disk too, so a restarted node enforces the rules it had cached, not newer ones. That is the same trade-off a running node already makes during an outage, now extended across restarts.

When the control server comes back

Recovery was fast and needed no help, except for the expired key. From the 10-minute run without the cache:

16:21:27 control server back
16:21:48 c (tried to join during the outage) gets an address after 21s
16:21:55 a->c reachable after 7s
16:21:55 a->b ok (hello from b) | b->a ok (hello from a)
16:21:55 a->d (restarted during the outage) reachable after 0s
16:22:00 a->e (key expired during the outage): FAIL; e's state: Logged out.
16:22:01 admin clears e's expiry: a->e reachable after 1s

With the control server up, access changes are fast

The flip side is how quickly the control plane enforces a change when it is up. We changed the policy so that a may no longer reach b, reloaded Headscale with SIGHUP, and probed again and again until the result changed. Then we restored it, and then deleted b:

17:04:59 policy changed to block a->b: blocked after 6s
17:05:00 policy restored: a->b works again after 1s
17:05:06 node b deleted: a->b blocked after 5s

In both blocking cases the first probe after the change already failed. The 5 to 6 seconds is mostly that probe's own 5-second timeout. So with the control server up, a policy change or a removed device takes effect in seconds. With it down, it does not take effect at all. Your access control is only as live as your control plane.

This is also where ACLs as code pays off. A policy file in Git, reviewed and applied by CI, is a common way to run both Tailscale and Headscale. During an outage you cannot apply it, but once control is back, the reviewed version goes out in seconds. Git tells you which policy you intended at any point; it does not prove that every device received it, so check from the devices too.

What we got wrong first

Three mistakes in our first runs are worth sharing, because they are easy to make in your own health checks:

We restarted half of the long-lived pair. The first run used b both as a long-lived peer and as the restart target, so after about six minutes the "do existing tunnels survive?" measurement was gone. We added d as a separate restart target and ran it again.
Our recovery script read d's address once, at the start. A node restarted without a cache has no address until it logs in again, so every probe went to an empty address, and the script reported that d never recovered. d's own log showed it had logged in again seconds after control returned. If you write tailnet health checks, look up addresses on every attempt.
Our first probe accepted any response, and our restart check misread a logged-out client. A review of the scripts pointed out that the probe only checked for a non-empty body; it now requires curl to succeed and the page to name the node we meant to reach. The restart helper used tailscale status, which exits with an error when a node is logged out, so it reported the uncached restart as a failed start instead of measuring it; it now uses tailscale status --json. Every number in this post comes from runs with the fixed scripts.

What this means if you are replacing a VPN

Count the control plane in your access path. For Tailscale's hosted service, that means their availability. For Headscale, it is one process with one database, so back up the database and treat upgrades like any other change.
Turn on netmap caching where your client and control server support it, or plan to upgrade to the release where it is the default. In our tests, without it, a restart during the outage took the device off the network until control returned.
Do not restart clients during a control incident. Pause automatic client updates and node rotation until control is back.
Choose key expiry on purpose. Expiry limits the damage of a stolen key and also turns a long control outage into lost devices. Decide per device class, and alert before keys expire.
Keep a way to contain a device that does not need the control plane , such as host firewalls or cloud security groups in front of your most sensitive servers that you can change without the mesh. Revoking sessions at your identity provider helps for applications that check it, but it does not close the network path.
Monitor real traffic , not the self status. tailscale status called a working node offline.

What we could not test

Relays. Everything ran on one host, and the client status showed direct paths; we did not force relayed connections. If your traffic goes through DERP relays, and especially if you run the relay on the same host as Headscale, an outage may take the relays down as well. We did not measure that.
Other products. NetBird, ZeroTier, Twingate and Teleport split control and data differently. The questions in this post (joins, revocation, expiry, restarts) are the right ones to ask of any of them, but our numbers only describe Tailscale clients with a Headscale control server.
Tailscale's hosted control plane. It may roll out features like netmap caching differently from Headscale.
Long outages. Our longest was 30 minutes. Anything key-related scales with your expiry settings, not with our test.

Summary

With the control server down for 30 minutes, existing tunnels kept working in both directions on every probe.
New devices could not join, and nobody could revoke a device. Revocation is the real security cost of the outage.
Key expiry is enforced by the peers. A key that expires during an outage takes its device off the network until someone re-authenticates it.
Without the netmap cache, a client restarted during the outage came back with no peers and stayed off the network until control returned. With it (the cache-network-maps node attribute, expected as the default from Tailscale 1.104), peers reached it again in 15 to 19 seconds in 11 of 12 restarts, and it stayed reachable in two 10-minute checks (one earlier run saw an unexplained two-minute drop).
With the control server up, a policy change or a deleted device took effect at the first probe, within seconds.

Originally published at devops-daily.com.

CVE-2026-80521: The Ubuntu Container Escape Is Not Just an Ubuntu Problem

DevOps Daily — Fri, 02 Oct 2026 09:00:00 +0000

On September 22, DepthFirst published a working exploit for CVE-2026-80521 : an unprivileged process inside a default Docker or Kubernetes container gets a root shell on the host. Their exploit targets one specific Ubuntu 26.04 kernel, but the bug behind it is much wider. Ten days later, Ubuntu's tracker still lists the main kernels for 24.04 LTS and 26.04 LTS as vulnerable, with no fixed package to install.

Most coverage calls it "the Ubuntu container escape". That framing is too narrow, and it can make you look in the wrong place. The bug is in upstream Linux code that arrived in 6.10 and was later backported to the 6.1 and 6.6 long-term kernels. Upstream has fixed 6.12, 6.18 and 7.1, but as of today there is no fix in the 6.1 or 6.6 branches at all. Debian 12, for example, is listed as vulnerable.

This post shows how to tell whether a machine has the affected code (in one command that usually works inside a pod too), where fixes exist, and what to do on nodes that cannot be patched yet.

TLDR

Detail	Info
CVE	CVE-2026-80521, CVSS 7.8
Class	Use-after-free in the AF_UNIX socket garbage collector, container escape
Reported by	Kyle Zeng
Public exploit	September 22, 2026 (DepthFirst), built for an Ubuntu 26.04 LTS kernel
Vulnerable code	Linux 6.10 and later, plus backports in 6.1.141+ and 6.6.93+
Fixed upstream	6.12.111, 6.18.53, 7.1.10, 7.2
Not fixed upstream	6.1 and 6.6 long-term branches (as of October 2)
Default seccomp	Docker's default profile allows the calls the exploit needs
Quick check	`grep -cE ' unix_(add

Prerequisites

Shell access to the hosts you want to check, or {% raw %}kubectl access to a cluster
For the cluster check: permission to run kubectl debug node/...
Your distribution's security tracker for the final answer on a distro kernel

What happened

Unix domain sockets can carry open file descriptors from one process to another (SCM_RIGHTS). A socket can even be sent over itself, so the kernel needs a garbage collector to find groups of sockets that only reference each other and free them.

Linux 6.10 replaced that garbage collector with a new design that tracks sockets as vertices and edges in a graph and groups them into strongly connected components (SCCs). The fix commit describes a race in that code: while one thread sends a socket and another closes the sockets involved, the collector can free a vertex but leave it linked in a cached SCC list. The next garbage collection run walks that list and touches freed memory. That use-after-free is what the exploit turns into a host root shell.

Two details make this bad for container platforms:

Ordinary containers can reach it. Creating Unix sockets and passing file descriptors are everyday operations. Docker's default seccomp profile allows them, and so does the RuntimeDefault profile of the common container runtimes, because almost all software needs them.
The kernel is shared. A container is a process on the host kernel. Namespaces, cgroups and dropped capabilities limit what a process can do through normal interfaces, and a kernel memory bug like this one goes around them.

Upstream fixed it in August (af_unix: Unlink scc_entry in unix_del_edge()), and the kernel CVE team published the CVE on August 26. The fix is one line. Getting it into every distribution kernel is the slow part.

Who is affected

According to the kernel CVE record, the vulnerable code is in:

every kernel from 6.10 onward, until the fixed releases below
6.1.141 and later in the 6.1 series, and 6.6.93 and later in the 6.6 series, because the new garbage collector was backported to those long-term branches

The fix is in 6.12.111 , 6.18.53 , 7.1.10 and 7.2. The CVE record lists no fixed 6.1 or 6.6 release, and a search of the 6.1.y and 6.6.y stable branch history on October 2 found no commit with the fix's title (the same search finds it in 6.12.y and 6.18.y). The newest releases there (6.1.188 and 6.6.157, both from September 14) still have the bug.

What the main distributions say, checked on October 2:

Distribution	Kernel	Status
Ubuntu 26.04 LTS	`linux` and the cloud kernels (aws, azure, gcp, gke, oracle)	Vulnerable, work in progress
Ubuntu 24.04 LTS	`linux` and the cloud kernels	Vulnerable
Ubuntu 22.04 LTS	`linux` (5.15)	Not affected
Debian 13 (trixie)	6.12	Fixed in `6.12.111-1` (DSA-6528-1)
Debian 12 (bookworm)	6.1	Vulnerable (`6.1.176-1`), no fix yet
Debian 11 (bullseye)	5.10	Not affected
Amazon Linux 2023	6.18.35	Livepatch for `kernel-livepatch-6.18.35-68.129`, September 30 (ALAS2023LIVEPATCH-2026-397)

Sources: the Ubuntu CVE page, the Debian security tracker and the Amazon Linux advisory. We did not check every vendor. If you run RHEL and its rebuilds, Azure Linux, Container-Optimized OS, Bottlerocket or Talos, check that vendor's tracker.

Note

Some early reports listed Ubuntu 22.04 as vulnerable. Ubuntu's own tracker marks the 22.04 linux kernel (5.15) as not affected, which matches the upstream record: 5.15 never received the new garbage collector. The 22.04 HWE kernels are newer and are tracked separately, so check them on the tracker.

Why the version number is not enough

The natural first step is to compare uname -r against the list above. That works for upstream kernels and fails for distribution kernels, in both directions.

Ubuntu 24.04 ships a 6.8 kernel. Upstream 6.8 never had the new garbage collector, so a version check says "not affected". Ubuntu's tracker says vulnerable: distribution kernels pull in fixes from the stable branches, and the backport that brought the new code into 6.6 can bring it into a distro's 6.8 too. The reverse also happens. A distro can backport the one-line fix and keep the old version number.

So check for the code itself. The new garbage collector adds functions named unix_add_edges and unix_del_edges, and the kernel lists its function names in /proc/kallsyms. That file is world-readable by default, and unprivileged users usually see the names with the addresses zeroed. An ordinary container sees the host kernel's list, because there is only one kernel (a gVisor or Kata sandbox does not).

grep -cE ' unix_(add|del)_edges$' /proc/kallsyms

0: the new garbage collector was not found. On a standard distribution kernel, that means this CVE does not apply. On a custom or heavily modified kernel, confirm with whoever builds it.
1 or more: the new garbage collector is present. Now the version and your vendor decide whether the fix is in.

The presence check cannot see the fix itself, because the fix adds one line to a different function, unix_del_edge(). That is why the script below combines it with the version, and sends you to your vendor when the kernel is a distro build. Treat it as a fast first filter, not a certificate.

A check script

#!/usr/bin/env bash
# Heuristic check for CVE-2026-80521 (AF_UNIX garbage collector use-after-free).
# Usually works inside a container too: /proc/kallsyms lists the host kernel's symbols.
# Exit codes: 0 not detected or fixed, 1 affected, 2 check your vendor, 3 cannot tell.
set -u

release=${KERNEL_RELEASE:-$(uname -r)} # KERNEL_RELEASE only for testing the logic
if [[! $release =~ ^([0-9]+)\.([0-9]+)(\.([0-9]+))?(.*)$ ]]; then
  echo "verdict: CANNOT TELL (unrecognised kernel release '$release')"; exit 3
fi
major=${BASH_REMATCH[1]} minor=${BASH_REMATCH[2]} patch=${BASH_REMATCH[4]:-0}
suffix=${BASH_REMATCH[5]} # anything after x.y.z: a distro, custom or -rc build

# The bug is in the garbage collector that Linux 6.10 rewrote (backported to
# 6.1.141+ and 6.6.93+). unix_add_edges and unix_del_edges exist only in that code.
if ! head -n 1 /proc/kallsyms 2>/dev/null | grep -q .; then
  new_gc=unknown
else
  grep -qE ' unix_(add|del)_edges$' /proc/kallsyms
  case $? in 0) new_gc=yes ;; 1) new_gc=no ;; *) new_gc=unknown ;; esac
fi

# Upstream releases that contain the fix (kernel CVE record, 2 October 2026).
fixed_upstream=no
case "$major.$minor" in
  6.12) (( patch >= 111 )) && fixed_upstream=yes ;;
  6.18) (( patch >= 53 )) && fixed_upstream=yes ;;
  7.1) (( patch >= 10 )) && fixed_upstream=yes ;;
esac
if (( major > 7 || (major == 7 && minor >= 2) )); then fixed_upstream=yes; fi

echo "kernel: $release"
echo "new AF_UNIX GC found: $new_gc"

case $new_gc in
  unknown) echo "verdict: CANNOT TELL (could not read /proc/kallsyms); check $major.$minor with your vendor"; exit 3 ;;
  no) echo "verdict: NOT DETECTED (no new garbage collector; on a standard distro kernel this means not affected)"; exit 0 ;;
esac
if [[-n $suffix]]; then
  if [[$fixed_upstream == yes && $suffix != -rc*]]; then
    echo "verdict: PROBABLY FIXED (base $major.$minor.$patch has the fix upstream); confirm with your vendor"; exit 2
  fi
  echo "verdict: CHECK YOUR VENDOR (new garbage collector present; distro kernels backport fixes without changing the version)"; exit 2
fi
if [[$fixed_upstream == yes]]; then
  echo "verdict: FIXED (if this is an unmodified upstream $release build)"; exit 0
fi
echo "verdict: AFFECTED (upstream $release has the bug and no fix)"; exit 1

We ran it as an unprivileged user on two machines: a Raspberry Pi test box on Raspberry Pi OS, and an Ubuntu 22.04 server. The KERNEL_RELEASE variable is only there to test the version logic with made-up strings.

af-unix-gc-check

# Raspberry Pi OS, kernel 6.12.62: new garbage collector present, below the 6.12.111 fix
$ ./af-unix-gc-check.sh; echo "exit=$?"
kernel: 6.12.62+rpt-rpi-v8
new AF_UNIX GC found: yes
verdict: CHECK YOUR VENDOR (new garbage collector present; distro kernels backport fixes without changing the version)
exit=2
# Ubuntu 22.04, kernel 5.15: old garbage collector
$ ./af-unix-gc-check.sh; echo "exit=$?"
kernel: 5.15.0-191-generic
new AF_UNIX GC found: no
verdict: NOT DETECTED (no new garbage collector; on a standard distro kernel this means not affected)
exit=0

We also read /proc/kallsyms from inside a running Docker container on the 22.04 server. The symbol names were there (addresses zeroed), so the one-line check worked from inside the container. Hardened setups can mask that file; then the script says it cannot tell.

Check a whole cluster

Start with the kernel and OS image of every node:

kubectl get nodes -o custom-columns=NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion

Then run the presence check on every distinct kernel build you see, or simply on every node during a rollout, when old and new images run side by side:

# Starts a debugging pod on the node; /proc/kallsyms is the node's kernel symbol list
kubectl debug node/<node-name> -it --image=busybox:1.36 -- \
  grep -cE ' unix_(add|del)_edges$' /proc/kallsyms

kubectl debug node creates a pod that shares the node's PID, network and IPC namespaces, so it needs the matching RBAC and must be allowed by your admission policy. Delete the debug pod when you are done (kubectl get pods shows it as node-debugger-...).

Node kernel uname -r
kallsyms check unix_add_edges?
Fix in this build? version or vendor tracker

Outcomes:

0 matches or fixed build: not exposed
Code present, no fix: isolate untrusted workloads

What to do now

1. Patch where a fix exists

If your kernel line has a fix (Debian 13, upstream 6.12.111+, 6.18.53+, 7.1.10+ or 7.2+), install it. A new kernel package does nothing until the node boots into it, so plan the reboots: cordon and drain one node at a time, or replace node pools with a new image. On Amazon Linux 2023 with the matching kernel, the livepatch applies without a reboot once kernel livepatching is enabled; verify that it is applied.

2. Find the workloads that run code you did not write

Where there is no fix yet (Ubuntu 24.04 and 26.04, Debian 12, anything on a 6.1 or 6.6 kernel at or above the backport), the question is who can run arbitrary code on those nodes. A container escape needs code execution inside a container first. The usual places where strangers get exactly that:

CI runners and build agents that run pull or merge request code from forks
multi-tenant clusters where teams or customers deploy their own images
notebooks, online code runners, plugin systems and "run your script" features
any pod with a known remote code execution bug that you have not patched yet

Single-tenant nodes that only run your own images are still worth patching, but they are not where you start.

3. Move untrusted workloads into a sandbox or onto unaffected nodes

For the workloads in step 2, the most reliable option you can use today is a different isolation boundary:

gVisor runs the container on a user-space kernel (the Sentry) that implements Unix sockets itself, so the sandbox's own Unix sockets do not use the host kernel's AF_UNIX garbage collector. (Access to host Unix sockets is a separate option and is off by default.) On GKE this is GKE Sandbox. Elsewhere, install runsc on the nodes, register it as a containerd runtime handler, and then add a RuntimeClass.
Kata Containers runs each pod in a lightweight VM, and can use Firecracker as its VM monitor. The guest kernel may have the same bug, but an escape then lands in a VM, not on the host.
A separate node pool on an unaffected kernel , such as Ubuntu 22.04 with its 5.15 kernel, for the untrusted workloads until your main image is fixed. It is a short-term measure with its own tradeoffs (an older kernel and different hardware support), not a destination.

A RuntimeClass for gVisor, and a pod that uses it:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc # the containerd runtime handler configured on the node
---
apiVersion: v1
kind: Pod
metadata:
  name: untrusted-build
spec:
  runtimeClassName: gvisor
  containers:
    - name: build
      image: registry.example.com/ci/build-runner:2026.10

Test your workloads under gVisor before you switch: it does not implement every system call, and I/O heavy jobs can run slower.

4. Know what does not help

Default seccomp. The default profiles allow Unix sockets and file descriptor passing.
Blocking AF_UNIX with a custom seccomp profile, in general. For the Copy Fail AF_ALG bug earlier this year, blocking one rarely used socket family was a clean mitigation. AF_UNIX is different: DNS resolution through NSS, database clients on local sockets, process managers, logging and many language runtimes use it. Blocking it breaks most real applications. A custom profile can still make sense for a specific workload that you have tested without Unix sockets.
Rootless containers, user namespaces and dropped capabilities. They are good hardening, but they are not a fix here: the vulnerable code is reachable without any special privilege.

5. Watch for the fix and re-check

Subscribe to your distribution's security notices, and keep the check above in your node image pipeline. Once the fixed kernel is out, the version check plus the vendor's fixed package version tells you which nodes still need a reboot. Then move the sandboxed workloads back, or keep them sandboxed. Copy Fail and Fragnesia were container escapes too, earlier this year, so the sandbox may be worth keeping.

Summary

CVE-2026-80521 is a use-after-free in the Linux AF_UNIX garbage collector. A public exploit escapes default Docker and Kubernetes containers to host root.
It is not limited to Ubuntu. The vulnerable code is in 6.10 and later and in the 6.1.141+ and 6.6.93+ long-term kernels. Upstream fixed 6.12, 6.18 and 7.1, but not 6.1 or 6.6.
Ubuntu 24.04 and 26.04 and Debian 12 had no fixed kernel on October 2. Ubuntu 22.04 (5.15), Debian 11 and other kernels without the new garbage collector are not affected.
Version numbers mislead on distro kernels. grep -cE ' unix_(add|del)_edges$' /proc/kallsyms tells you whether the new garbage collector is there, usually even from inside a container.
Until you can patch, put workloads that run untrusted code behind gVisor, Kata or a VM, or on nodes with an unaffected kernel. Default seccomp and user namespaces do not stop this one.

Originally published at devops-daily.com.

Hacktoberfest 2026 Stopped Counting PRs. Make Your First Ones Anyway, One a Day

DevOps Daily — Thu, 01 Oct 2026 15:04:37 +0000

Hacktoberfest started today, and if you have taken part before, the first thing to know is that the rule everyone remembers is gone. Pull requests no longer count toward Hacktoberfest rewards. There is no PR target and no PR-based swag this year.

That is a good change. It is also a slightly awkward one if Hacktoberfest was the push you needed to make your first open source contribution. This post covers what changed, why a first PR is still worth making, and a 7-day challenge we built: one small task a day, 5 to 15 minutes each, and each one ends with a real pull request that a maintainer reviews.

What changed in Hacktoberfest 2026

From the official FAQ:

Pull requests and merge requests will no longer count toward Hacktoberfest rewards. It’s easier than ever to submit low-effort spam PRs to projects, so we’re listening to maintainer feedback and no longer actively incentivizing PRs.

The rest of the event changed with it:

New organizers. MLH and DEV run Hacktoberfest this year, in partnership with DigitalOcean.
New format. 300+ in-person and online community events, focused on hands-on building and learning with open-source AI and open-weight models. The in-person ones are called Fests.
New rewards. You collect stickers. Two are required: sign in with MyMLH and add your mailing address. You earn more by checking in at a Fest, entering the code shown during a livestream, or submitting to a DEV Challenge. Fifteen stickers enter you in a raffle for a Hacktoberfest t-shirt or an Arduino Uno Q board (details on the online participation page).

If you maintain a repo, you know why. Every October, maintainers spent hours closing pull requests that changed one word in a README, and AI tools made those PRs even cheaper to produce. Taking the swag off the PR removes the main reason to send them.

Why your first PR still matters

The spam was the problem, not the pull request. The things a first contribution teaches you did not change:

Forking, branching and keeping a fork in sync without breaking it.
Reading a codebase you did not write, well enough to change one thing in it.
Running a project's tests before a reviewer has to tell you they fail.
Writing a PR description that a busy maintainer can approve in one read.
Taking review feedback and pushing a fix to the same branch.

Those are daily skills in any DevOps or platform job. A merged PR in a public repo is also evidence of them, which a line on a CV is not.

What you need is a project that wants small contributions, says exactly how to make them, and reviews them. So we made one.

The challenge: one small PR a day for 7 days

The DevOps Daily Hacktoberfest challenge is 7 daily tasks, plus a bonus, in the devops-daily repo: an open source DevOps learning site with 560+ posts, 45 quizzes and 27 flashcard decks, all stored as Markdown and JSON in the repo. Each day has its own page with the file to change and an example, and most days include a command to check your work.

To be clear about what this is: it is a DevOps Daily challenge that runs during Hacktoberfest, not an official Hacktoberfest event, and these PRs do not count toward Hacktoberfest stickers. What you get is a real contribution, reviewed by a maintainer, with a test suite in CI, on your GitHub profile. The first challenge PRs came in within hours.

Set up once (about 10 minutes)

You need Git, Node.js 22.13.1 or later (below 25), and pnpm 10.

# Fork https://github.com/The-DevOps-Daily/devops-daily on GitHub first, then:
git clone https://github.com/<your-username>/devops-daily.git
cd devops-daily
git remote add upstream https://github.com/The-DevOps-Daily/devops-daily.git

pnpm install
pnpm test          # the unit tests CI runs on every PR

Before each day's task, start a new branch from a fresh copy of main, so your PR contains only that day's change. Change the day number each time:

git fetch upstream
git checkout -b hacktoberfest/day-1 upstream/main

The one exception is Day 7, which adds to your Day 1 profile. If your Day 1 PR is not merged yet, branch from your Day 1 branch instead.

The 7 days

Day 1: Add yourself to the experts directory (5 min). Create a Markdown profile in content/experts/ with your name, a short bio and how people can reach you. It is the smallest possible PR, and it shows you the whole fork, branch, commit, push and PR flow once before the content gets harder.

Day 2: Add your favorite DevOps tool (5 min). Add one entry to the toolbox page: name, one-line description, link and category. Pick a tool you use, not one you have heard of. The description is where your experience shows.

Day 3: Add a quiz question (5 min). Add a question to one of the existing quizzes in content/quizzes/. The tests check every question's shape, so run them before you push. Write the explanation for the wrong answers too: "why not B" teaches more than "why A".

Day 4: Add a flashcard (5 min). Add one or two cards to a deck in content/flashcards/. A good card has one fact on the front and the shortest complete answer on the back.

Day 5: Share a tip (10 min). Add a gotcha or "thing I wish I knew" to an existing post or guide. This is the first day where you edit someone else's writing, so add your tip where a reader needs it and leave the rest of the post alone.

Day 6: Find and fix something (10 min). Browse devops-daily.com and find a typo, a broken link, a deprecated command or a formatting problem. Fix exactly one thing, and put the URL where you found it in the PR description.

Day 7: Share your stack (15 min). Add a "My Stack" section to the profile you made on Day 1: what you run, why you picked it, and what you would change.

Bonus: Build something (30+ min). A full quiz on a topic that is not covered yet, a tool comparison, a production checklist, or for the ambitious, an interactive game or simulator.

How to get your PR merged (and not closed as spam)

The 2026 change is a good reminder of what maintainers are tired of. These habits make a PR easy to review in any project, not only this one:

One change per PR. If you spot a typo in another quiz while doing Day 3, fix it in a separate PR. A small diff is quick to check.
Run the tests first. Run pnpm test before you push, so you find a failing check before a reviewer does.
Say what and why. Two sentences: what you changed, and why it is correct. For a fix, add the link to where you found the problem.
Read what you submit. If an AI tool helped you write a quiz question or a tip, check it like you would check a stranger's code. A confident wrong answer in a quiz is worse than no question.
Answer review on the same branch. Push the fix to the same branch and the PR updates. Do not open a new PR for each round.

Collect your Hacktoberfest stickers too

The challenge and the official event go together well. Do one small PR a day here, and collect your stickers at hacktoberfest.com: check in at a Fest near you, enter the code during a livestream, or submit to a DEV Challenge, since DEV co-runs the event this year.

Start with Day 1 today: devops-daily.com/hacktoberfest. And if you get stuck on any day, open an issue on the repo or ask in your PR. Questions are contributions too.