# How to ship application logs off your VMs

Source: https://docs.quake.ai/docs/operate/monitoring/how-to-ship-logs
Markdown: https://docs.quake.ai/docs/operate/monitoring/how-to-ship-logs.md

---

# How to ship application logs off your VMs

Forward application and system logs to a central store for search and alerting. Quake AI does not provide a managed log service. Run self-hosted Loki on a Quake AI VM, or forward to a SaaS log platform over HTTPS.



Quake AI does not meter outbound transfer. Your log SaaS may charge per gigabyte ingested at their endpoint. Size batching and log levels accordingly.



## Path A: Promtail to self-hosted Loki

Deploy Loki alongside Prometheus/Grafana ([monitoring how-to](/docs/operate/monitoring/how-to-deploy-monitoring)), then install and run Promtail on each application VM.

Install the Promtail binary. Match the version to your Loki release:

```bash
sudo apt-get update && sudo apt-get install -y unzip
PROMTAIL_VERSION="3.1.1"
curl -fsSLO "https://github.com/grafana/loki/releases/download/v${PROMTAIL_VERSION}/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
sudo install -D -m 0755 promtail-linux-amd64 /usr/local/bin/promtail
sudo mkdir -p /etc/promtail /var/lib/promtail
```

Write the configuration. Replace `LOKI_HOST` with the private IP or hostname of the Loki instance you deployed in the [monitoring how-to](/docs/operate/monitoring/how-to-deploy-monitoring):

```yaml
# /etc/promtail/config.yml
server:
  http_listen_port: 9080
positions:
  filename: /var/lib/promtail/positions.yaml
clients:
  - url: http://LOKI_HOST:3100/loki/api/v1/push
scrape_configs:
  - job_name: varlogs
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*.log
```

Create a systemd unit:

```ini
# /etc/systemd/system/promtail.service
[Unit]
Description=Promtail log shipper
After=network-online.target

[Service]
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/config.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

Reload systemd and start Promtail:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now promtail
```

Query logs in Grafana with LogQL.

## Path B: Vector to a SaaS HTTP endpoint

Install [Vector](https://vector.dev/) and point a `http` sink at your provider (Grafana Cloud, Datadog, Better Stack, Axiom, and similar):

```toml
[sources.app_logs]
type = "file"
include = ["/var/log/myapp/*.log"]

[sinks.saas]
type = "http"
inputs = ["app_logs"]
uri = "https://logs.example-provider.com/v1/ingest"
encoding.codec = "json"
```

Store the provider API key in [application secrets](/docs/security/how-to/inject-app-secrets), not in the config file committed to git.

## Path C: Fluent Bit DaemonSet on Kubernetes

On Magnum:

```bash
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit \
  --namespace logging --create-namespace \
  --set config.outputs='[OUTPUT]
    Name http
    Match *
    Host logs.example-provider.com
    Port 443
    tls On'
```

Tune `Match` rules so only application namespaces ship to the SaaS endpoint.

## Verify

- Generate a known log line (`logger.info("ship-logs-test")`) and locate it in Loki or the SaaS search UI within one minute.
- Confirm log volume and retention match your compliance requirements.

## See also

- [How to monitor your Quake AI workload with Prometheus and Grafana](/docs/operate/monitoring/how-to-deploy-monitoring)
- [Monitoring overview](/docs/operate/monitoring)
- [How to store application secrets and inject them at runtime](/docs/security/how-to/inject-app-secrets)