# How to schedule recurring jobs on a Quake AI VM

Source: https://docs.quake.ai/docs/compute/how-to/schedule-cron-jobs
Markdown: https://docs.quake.ai/docs/compute/how-to/schedule-cron-jobs.md

---

# How to schedule recurring jobs on a Quake AI VM

Run a command or script on a fixed schedule inside a Quake AI instance, using either `cron` or `systemd` timers.



Scheduling runs in-guest on a VM you operate. Use `cron` or `systemd` timers from the Linux images Quake AI provides, and configure them over SSH inside the instance.



<PrerequisiteBlock>

- A running Linux instance. See [How to create an instance](/docs/compute/how-to/create-instance).
- SSH access to the instance. See [How to add an SSH key](/docs/tools/add-ssh-key).
- A user with `sudo` privileges on the instance.

</PrerequisiteBlock>

## Floating IP requirements

These scheduling patterns require zero floating IPs. Both `cron` and `systemd` timers run entirely inside the guest operating system, so a scheduled job needs no inbound network path. A job that calls the Quake AI API or reaches another instance uses the project's private network, which also needs no floating IP. Allocate a floating IP only if you need to reach the instance directly over SSH from outside the project, and one floating IP per instance covers that access.

## Choose cron or systemd timers

Both mechanisms run a command on a schedule. Pick one per job and keep that job's definition in a single place.

| Use `cron` when | Use `systemd` timers when |
|---|---|
| You want the shortest possible setup for a simple recurring command | You want per-run logging in the system journal |
| The job is self-contained and has no service dependencies | The job depends on another unit, the network, or a mounted volume |
| You are comfortable redirecting output to a file yourself | You want built-in failure handling, run-time limits, and missed-run catch-up |

`systemd` timers need more configuration upfront and give you logging, dependency ordering, and reliability controls. The rest of this guide shows both, so you can match the mechanism to the job.

## Schedule a recurring job

The example below runs a maintenance script, `/usr/local/bin/cleanup-tmp.sh`, every day at 02:30 in the instance's local time zone. Replace the script path and schedule with your own.




Each user has a crontab, a table of scheduled commands. Edit the current user's crontab:

```bash
crontab -e
```

Add one line. The five fields are minute, hour, day of month, month, and day of week, followed by the command:

```cron
30 2 * * * /usr/local/bin/cleanup-tmp.sh
```

Save and exit the editor. List the active entries to confirm the change:

```bash
crontab -l
```

The command runs as the user who owns the crontab. For a job that must run as `root`, install it in the root crontab with `sudo crontab -e`, or drop a file in `/etc/cron.d/`:

```cron
# /etc/cron.d/cleanup-tmp
30 2 * * * root /usr/local/bin/cleanup-tmp.sh
```

Files in `/etc/cron.d/` include an extra field for the user that runs the command. Keep the schedule readable by adding a comment above each entry that describes what the job does.



Set the crontab time zone explicitly so the schedule does not shift when you move the instance between regions. Add `CRON_TZ=UTC` as the first line of the crontab to run every entry in UTC.






A `systemd` timer pairs two units: a service unit that defines the work and a timer unit that defines the schedule.

Create the service unit at `/etc/systemd/system/cleanup-tmp.service`:

```ini
[Unit]
Description=Clean up temporary files
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/cleanup-tmp.sh
```

Create the timer unit at `/etc/systemd/system/cleanup-tmp.timer`:

```ini
[Unit]
Description=Run the temporary-file cleanup daily

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target
```

`OnCalendar` sets the schedule, `Persistent=true` runs a job that was missed while the instance was off, and the timer and service share the same base name so `systemd` links them automatically.

Reload `systemd`, then enable and start the timer:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now cleanup-tmp.timer
```

Confirm the timer is active and check its next run time:

```bash
systemctl list-timers cleanup-tmp.timer
```




## Capture output and logs

A scheduled job produces no terminal output, so route its output somewhere you can read later.




`cron` mails a job's output to the local user by default, which most cloud instances do not deliver. Redirect both standard output and standard error to a log file in your home directory instead. A user crontab cannot create files under `/var/log/` without extra permissions.

Create the log directory, then add the redirect to your crontab:

```bash
mkdir -p ~/logs
```

```cron
30 2 * * * /usr/local/bin/cleanup-tmp.sh >> ~/logs/cleanup-tmp.log 2>&1
```

Rotate the log so it does not grow without bound. `logrotate` needs the full path to your home directory:

```bash
sudo tee /etc/logrotate.d/cleanup-tmp <<EOF
/home/$(whoami)/logs/cleanup-tmp.log {
    weekly
    rotate 8
    compress
    missingok
    notifempty
}
EOF
```

To write logs under `/var/log/`, install the job in the root crontab with `sudo crontab -e` or in `/etc/cron.d/` as shown earlier.




`systemd` captures a service's standard output and standard error in the journal, so no redirection is needed. Read the job's history:

```bash
journalctl -u cleanup-tmp.service
```

Follow the most recent run as it happens:

```bash
journalctl -u cleanup-tmp.service -f
```

The journal applies its own size limits and rotation, so a chatty job does not fill the disk. Set `SystemMaxUse` in `/etc/systemd/journald.conf` to cap journal storage if you need a tighter limit.




## Handle failures and overlapping runs

A job that runs longer than its interval can start a second copy before the first finishes, and a job that fails silently can go unnoticed for days. Guard against both.




Wrap the command in `flock` to take an exclusive lock, so a slow run blocks the next start instead of overlapping it:

```cron
30 2 * * * /usr/bin/flock -n /tmp/cleanup-tmp.lock /usr/local/bin/cleanup-tmp.sh >> ~/logs/cleanup-tmp.log 2>&1
```

The `-n` flag makes `flock` exit immediately if the lock is held, which skips the run rather than queueing it. For failure visibility, have the script exit non-zero on error and append a timestamped status line to its log, then alert on that line with your monitoring agent.




A `oneshot` service does not start a second run while the first is active, so overlap is handled for you. Add an `OnFailure` handler to react when a run exits non-zero. First reference a handler service in the job's `[Unit]` section:

```ini
[Unit]
Description=Clean up temporary files
After=network-online.target
OnFailure=notify-failure@%n.service
```

Then define `notify-failure@.service` as a template unit that sends an alert. To smooth load when several timers share a schedule, add a jitter window to the timer:

```ini
[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=300
Persistent=true
```

`RandomizedDelaySec=300` spreads the start time across a five-minute window.




## Worked example: schedule a volume snapshot

A common scheduled job is a recurring volume snapshot. Install the OpenStack CLI and an application credential on the instance, then schedule a snapshot of a target volume.

Create the script at `/usr/local/bin/snapshot-data-volume.sh`. Set `YOUR_CLOUD` to the cloud name in your `~/.config/openstack/clouds.yaml` or from your downloaded openrc file. See [How to generate application credentials](/docs/tools/generate-app-credentials).

```bash
#!/usr/bin/env bash
set -euo pipefail

export OS_CLOUD=YOUR_CLOUD
VOLUME_ID="YOUR_VOLUME_ID"
STAMP="$(date -u +%Y%m%d-%H%M%S)"

openstack volume snapshot create \
  --volume "$VOLUME_ID" \
  --force \
  "data-volume-$STAMP"
```

Make the script executable, create the log directory, then schedule it with whichever mechanism you chose above. The `cron` form runs it nightly at 01:00 UTC under a lock:

```bash
chmod +x /usr/local/bin/snapshot-data-volume.sh
mkdir -p ~/logs
```

```cron
0 1 * * * /usr/bin/flock -n /tmp/snapshot-data-volume.lock /usr/local/bin/snapshot-data-volume.sh >> ~/logs/snapshot-data-volume.log 2>&1
```

Use an application credential rather than a password so the scheduled job keeps working without interactive login. See [How to generate application credentials](/docs/tools/generate-app-credentials) and [How to create an instance snapshot](/docs/compute/how-to/create-snapshot) for the snapshot lifecycle. Prune old snapshots on the same schedule with a retention loop so the count stays bounded.

## Run a dedicated control VM for ops jobs

Scheduling each job on the instance it acts against works for instance-local maintenance. For jobs that act on many instances or on Quake AI resources through the API, run them from one small, long-lived control instance instead.

A control VM keeps scheduled operations in a single place you can audit and back up, holds one application credential rather than spreading credentials across the fleet, and isolates ops workloads from production traffic. A shared-vCPU flavor is enough for most control workloads. The control VM needs no floating IP of its own when it reaches the Quake AI API and other instances over the project's private network.

## Verify the result

Confirm the schedule is registered and inspect the most recent run.




List the active crontab entries:

```bash
crontab -l
```

After the first scheduled run, check the log file you configured:

```bash
tail -n 20 ~/logs/cleanup-tmp.log
```




List the timer and read its last run from the journal:

```bash
systemctl list-timers cleanup-tmp.timer
journalctl -u cleanup-tmp.service -n 20
```

The `list-timers` output shows the `LAST` and `NEXT` columns once the timer has fired at least once.




## See also

- [How to create an instance](/docs/compute/how-to/create-instance): provision the VM that runs your scheduled jobs
- [How to create an instance snapshot](/docs/compute/how-to/create-snapshot): the snapshot operation used in the worked example
- [How to generate application credentials](/docs/tools/generate-app-credentials): non-interactive credentials for jobs that call the API
- [Install the OpenStack CLI](/docs/tools/install-openstack-client): set up the CLI used inside the instance
- [How to use the virtual machine console](/docs/compute/how-to/use-vm-console): reach the instance when SSH is unavailable