Monitoring a Kubernetes CronJob (and catching the runs that never happen)

A Kubernetes CronJob looks healthy right up until you need what it was supposed to produce. The pods crashed through their retries on Tuesday, or the image tag was deleted, or someone set suspend: true during an incident and forgot. kubectl get cronjobs still shows a schedule and a last schedule time, and nothing tells you the job hasn’t succeeded in a week.

How Kubernetes CronJobs fail without telling you

The job runs and fails. The container exits non-zero, Kubernetes retries up to backoffLimit, and the Job gets marked failed. Unless something’s watching Job status, that’s the end of it.

The job never really runs. The pod sits in Pending waiting for capacity, or in ImagePullBackOff because a tag or registry credential changed. The CronJob did its part by creating the Job, so LAST SCHEDULE looks fine.

The schedule stops.

• Someone suspended the CronJob with spec.suspend: true.
• The controller was down long enough to miss more than 100 schedules. It then stops starting runs until you intervene, unless startingDeadlineSeconds limits how far back it counts.
• The CronJob was deleted or never applied to the new cluster after a migration.

The job succeeds at the wrong time. Without spec.timeZone, the schedule follows the controller manager’s time zone, usually UTC. A report you meant for 02:00 in Chicago runs at 21:00 the day before.

Whykubectland cluster metrics aren’t enough

kubectl get cronjob shows when a Job was last scheduled. It doesn’t say whether it succeeded. Prometheus with kube-state-metrics can alert on failed Jobs, but it lives inside the cluster it’s watching. If the cluster, the node pool or the monitoring stack is what broke, the alert goes down with it. And a CronJob that’s suspended or deleted doesn’t produce a failure to alert on.

What you want is a signal from outside that has to arrive for everything to be OK. That’s a heartbeat. The job reports success to an outside check, and the check alerts you when the report doesn’t show up on time.

Set up Kubernetes CronJob monitoring

1. Create a heartbeat check with the same schedule

Take the CronJob’s schedule, for example 0 2 * * *, and create a heartbeat check with the same cron expression. Set spec.timeZone on the CronJob (stable since Kubernetes 1.27) and choose the same time zone on the check, so both agree about when 02:00 is. If you’re unsure what an expression means, paste it into the cron expression explainer.

Give the check a grace period that covers pod scheduling, the image pull, the job’s normal runtime and any retries backoffLimit allows. A job that sometimes succeeds on its third try will report late, and that’s fine.

2. Put the ping URL in a Secret

Anyone with the ping URL can report a run, so keep it out of manifests you commit:

read -rs URL
kubectl create secret generic pingstack-nightly-report \
  --from-literal=ping-url="$URL"
unset URL

If you use Sealed Secrets or External Secrets, store it there instead.

3. Report only after the work succeeds

I like running the work in an init container and the report in the main container. Kubernetes only starts the main container after every init container exits successfully, so a failed run never reports. Your job’s image doesn’t need curl either:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          initContainers:
            - name: report
              image: registry.example.com/reports:1.4.0
              command: ["./run-report"]
          containers:
            - name: pingstack
              image: curlimages/curl:8.10.1
              command: ["sh", "-c"]
              args:
                - curl --fail --silent --show-error --max-time 10 --retry 3 "$PINGSTACK_PING_URL" > /dev/null
              env:
                - name: PINGSTACK_PING_URL
                  valueFrom:
                    secretKeyRef:
                      name: pingstack-nightly-report
                      key: ping-url

If your image already has a shell and curl, one container is enough. Chain the report with && so it only runs after a successful command:

command: ["sh", "-c", "./run-report && curl --fail --silent --show-error --max-time 10 \"$PINGSTACK_PING_URL\""]

concurrencyPolicy: Forbid stops a slow run overlapping the next one. startingDeadlineSeconds: 600 lets the controller still start a run up to ten minutes late, for example after a control-plane restart, and keeps it from counting a long backlog of missed schedules.

4. Test it without waiting for 02:00

kubectl create job --from=cronjob/nightly-report nightly-report-test
kubectl logs -f job/nightly-report-test --all-containers

The check should show a heartbeat and its next deadline. Delete the test Job when you’re done.

Catch “succeeded but did nothing”

Exit code 0 doesn’t mean the job did its work. A report that ran against an empty table succeeds in seconds. Send a number with the ping, like rows processed or file size, and set warning and critical thresholds on the check:

COUNT=$(./run-report --count) && \
  curl --fail --silent --show-error --max-time 10 \
    --get --data-urlencode "value=$COUNT" "$PINGSTACK_PING_URL"

Several clusters or CronJobs

Give each CronJob its own check and Secret. If the same CronJob runs in several clusters and each run matters, use one named location per cluster on the check. Each location expects its own heartbeat, so a healthy cluster can’t hide a silent one.

Common questions

Can I monitor a CronJob from inside the cluster? You can watch Job status with kube-state-metrics, and I’d definitely keep it. It can’t tell you about a suspended or deleted CronJob, though, and it fails along with the cluster. An outside heartbeat covers both.

Why did my check alert when the job was fine? Usually a time zone mismatch between the check and spec.timeZone, or a grace period shorter than pod scheduling plus retries. Pods stuck in Pending during a capacity crunch also report late.

Should the job also ping when it starts? No. Every report counts as a run, so a ping at the start would mark the check healthy even when the work then fails. Report once, after success.

What if the job runs more often than every minute? Kubernetes CronJobs can’t schedule more often than once a minute. For anything more frequent, use a long-running worker and report on an interval check.

Try it

Pingstack’s cron job monitoring takes your CronJob’s cron expression and time zone, alerts when a run is missing or late, and tracks a reported number against thresholds. Free covers 15 checks. The full Kubernetes recipe is in the docs, and the cron monitoring guide covers the same ideas for plain crontab. Start free and add a check to the CronJob you’d least like to fail silently.