CronTask controller
The CronTask controller creates a Task from a CronTask's template at each scheduled time. It decides from the CronTask, the Tasks that carry its uid label and the time. The manager performs each create and each delete.
| Read | Answers |
|---|---|
| CronTask | What do I apply, and what does the status hold? |
| State machine | What happens at a scheduled time? |
| Interface | What does the controller read, and what does the manager do? |
| Task controller | Who runs each Task that a CronTask creates? |
One run #
- You apply a CronTask. Its first reconcile writes
nextScheduleTimeandnotBefore. It creates no Task for a time before the CronTask existed. - The manager reconciles the CronTask again at
notBefore, or at once when its spec changes. - The controller finds the latest scheduled time at or before now. If the controller already decided about that time, it does nothing.
- Otherwise it decides about that time. It creates the Task, or it skips
the time and says why in the
Skippedcondition. - It deletes the Concluded Tasks beyond
historyLimit, oldest first. - It sets
notBeforeto the next scheduled time. While a create or a delete is in flight, the first attempt included, it setsnotBeforeto one minute later at most, so a create that never lands is retried long before the next scheduled time.
When the manager was down for several scheduled times, the controller decides only about the latest one. It drops the earlier ones, as a Kubernetes CronJob does.
Implementation and evidence #
| Responsibility | Source | Tests |
|---|---|---|
| Pure decisions: the time, the policies, recovery, history | CronTaskController | Controller passes, CronTask E2E |
| Resource, status and schedule math | CronTask types, validation | Unit tests in cron_task.rs; CronTask E2E |
| Sweep, intent before effect, fenced create and retire, receipts | Manager | CronTask E2E |
The crontask.fleet.dev/ labels, retire_scheduled_task |
Store | CronTask E2E |
| Crate edges and the field manager | architecture.toml | Architecture suite |
The E2E suite runs the manager on the file store with a test clock and the
fake runtime. It covers one Task per scheduled time, Forbid, Allow,
Replace and its wait for the old session, a missed deadline, suspend,
historyLimit, and recovery from an intent with no receipt: adoption, the
limits in force at the retry, and NeedsAttention for a create in doubt.
Through the fleet binary, it covers apply, get, list and delete, and the
refusals of a bad schedule, a bad template, template amendments and the
controller's labels.
CronTask state machine #
First, the controller gives up each pending create that a fresh decision
would refuse now: the CronTask is suspended, the create's scheduled time is
past its deadline, or the policy is Forbid and another Task of this
CronTask runs. Such a create never blocks a later time. A retry always
obeys the spec in force, as a fresh decision does.
At a scheduled time T, the controller applies these rules in order:
suspendis true: no decision. The phase isSuspended.startingDeadlineSecondsis set, and now is later thanTplus the deadline: skipT.Skippedis True with reasonMissedDeadline.concurrencyPolicyisForbid, and a Task of this CronTask is not Concluded, or a create is still pending: skipT.Skippedis True with reasonForbidden.concurrencyPolicyisReplace: delete each Task of this CronTask that is not Concluded, and give up a create that is still pending.- Write the create's intent for
T.Skippedis False with reasonCreated.
Then the controller makes each pending create, under the limits in force at that moment:
suspendis true: the create is given up.- The deadline passed: the create is given up.
Skippedis True with reasonMissedDeadline. concurrencyPolicyisReplace, and another Task of this CronTask is not Concluded or is being deleted: the create waits.Skippedis False with reasonReplacing. The controller looks again after one minute. A delete only marks the old Task, and the Task controller releases its session when the agent is at rest, or 5 minutes after the delete if the agent still reports work. The new Task is created only when the old one is gone from the store, so two agents never work at once.- Otherwise the manager creates the Task.
A Task that is deleting does not count as running for Forbid.
Conditions #
| Type | Status | Reason | Meaning |
|---|---|---|---|
Scheduled |
True | Scheduled |
The schedule parses, and the CronTask is not suspended. |
Scheduled |
False | Suspended |
spec.suspend is true. |
Scheduled |
False | InvalidSchedule |
The schedule does not parse. |
Skipped |
True | MissedDeadline |
The last scheduled time came later than its deadline. |
Skipped |
True | Forbidden |
A Task still ran at the last scheduled time. |
Skipped |
False | Created |
The last scheduled time decided to create its Task. |
Skipped |
False | Replacing |
The Task waits for the Task it replaces to be released. |
NeedsAttention |
True | CreateUnknown |
The store could not say, through three attempts, whether a create happened. |
NeedsAttention |
False | Resolved |
No create is in doubt any more. |
CronTask controller interface #
Controller API #
CronTaskController is in
fleet-controllers. It
is pure.
| Item | Meaning |
|---|---|
CronTaskInput |
The CronTask, the Tasks whose crontask.fleet.dev/uid label is its uid, and now. |
CronTaskController::due(obj, now) |
True when the CronTask has no status, its spec changed, or notBefore came. |
CronTaskController::reconcile(input) |
Returns a Plan: the status, a Create for a new Task, and a RetireScheduled effect for each Task to delete. |
The schedule math is in
fleet_api::cron_task. It
uses the croner crate, and it never reads the clock.
Manager execution protocol #
The manager reconciles the due CronTasks after the Tasks in each sweep. It lists the Tasks once for all of them. When it cannot list the Tasks, it reconciles no CronTask: an Unknown listing is never read as no Task.
For each due CronTask the manager does these steps:
- It writes the planned status, compare-and-swap under the lease. The status holds the intent of each create and delete, by key.
- It performs each delete with
Store::retire_scheduled_task, fenced on the Task's uid. Then it writes the receipt under the key. - It performs the create with
Store::create_fenced. Then it writes the receipt under the key.
A delete only marks the Task. Under Replace the controller plans the
replacement in a later pass, once the old Task is gone from the listing.
Recovery #
An intent with no receipt means that the manager stopped between the two writes. The controller answers it from the Task listing:
- The Task exists and carries this CronTask's uid: the controller records
the receipt
Created, withadoptedtrue. - The Task does not exist: the create is pending again. The controller
makes it under the same key and the same name, under the limits in force
now (
suspend, the deadline,Replace), and it counts as a running Task forForbid. - A create that the store refused three times is given up, with the
receipt
Failed. - A delete keeps the permission it was decided with,
running. A retry never widens it: a history delete stays Concluded-only, so it never removes a Task that a person reopened withfleet amendmeanwhile. Only a newerReplacedecision widens it. It is narrowed to Concluded-only while the spec no longer saysReplace. - A history delete that the store refused is decided again when the Task
is Concluded again and still past
historyLimit. - A delete, like a create, brings
notBeforewithin one minute, so one that never lands is retried long before the next scheduled time. - A create whose outcome is Unknown is tried again every minute, with no
limit: Fleet never reads Unknown as absence, so it never drops the create.
While it waits it counts as a running Task, so under
Forbidit holds later times. The record counts the attempts in doubt inunknowns: an attempt that read Unknown, or whose receipt never came. Only a Known outcome clears the count. After three attempts in doubt,NeedsAttentionis True with reasonCreateUnknown.
A create that finds a Task with the same name but without this uid is refused. The controller never takes another CronTask's Task.
Store port and write ownership #
| Write | Who | Rule |
|---|---|---|
| CronTask status | cron-task-controller |
It owns every field. |
| A Task, created | cron-task-controller |
Only through create_fenced, under the lease. |
The crontask.fleet.dev/name and -uid labels |
cron-task-controller |
Set at create. fleet apply refuses them, and nobody changes them later. |
| A Task, deleted | cron-task-controller |
Only through retire_scheduled_task: the Task's uid must match, the Task must carry this CronTask's uid, and it must be Concluded unless the policy is Replace. |