How to ship application logs off your VMs
Coming from another cloud?
▸AWS·Cloudwatch Logs
This Quake AI feature maps to AWS’s Cloudwatch Logs.
▸DigitalOcean·LOG Forwarding
This Quake AI feature maps to DigitalOcean’s LOG Forwarding.
How to ship application logs off your VMs
Forward application and system logs to a central store for search and alerting. Quake AI does not provide a managed log service. Run self-hosted Loki on a Quake AI VM, or forward to a SaaS log platform over HTTPS.
Path A: Promtail to self-hosted Loki#
Deploy Loki alongside Prometheus/Grafana (monitoring how-to), then install and run Promtail on each application VM.
Install the Promtail binary. Match the version to your Loki release:
sudo apt-get update && sudo apt-get install -y unzip
PROMTAIL_VERSION="3.1.1"
curl -fsSLO "https://github.com/grafana/loki/releases/download/v${PROMTAIL_VERSION}/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
sudo install -D -m 0755 promtail-linux-amd64 /usr/local/bin/promtail
sudo mkdir -p /etc/promtail /var/lib/promtailWrite the configuration. Replace LOKI_HOST with the private IP or hostname of the Loki instance you deployed in the monitoring how-to:
# /etc/promtail/config.yml
server:
http_listen_port: 9080
positions:
filename: /var/lib/promtail/positions.yaml
clients:
- url: http://LOKI_HOST:3100/loki/api/v1/push
scrape_configs:
- job_name: varlogs
static_configs:
- targets: [localhost]
labels:
job: varlogs
__path__: /var/log/*.logCreate a systemd unit:
# /etc/systemd/system/promtail.service
[Unit]
Description=Promtail log shipper
After=network-online.target
[Service]
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/config.yml
Restart=on-failure
[Install]
WantedBy=multi-user.targetReload systemd and start Promtail:
sudo systemctl daemon-reload
sudo systemctl enable --now promtailQuery logs in Grafana with LogQL.
Path B: Vector to a SaaS HTTP endpoint#
Install Vector and point a http sink at your provider (Grafana Cloud, Datadog, Better Stack, Axiom, and similar):
[sources.app_logs]
type = "file"
include = ["/var/log/myapp/*.log"]
[sinks.saas]
type = "http"
inputs = ["app_logs"]
uri = "https://logs.example-provider.com/v1/ingest"
encoding.codec = "json"Store the provider API key in application secrets, not in the config file committed to git.
Path C: Fluent Bit DaemonSet on Kubernetes#
On Magnum:
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit \
--namespace logging --create-namespace \
--set config.outputs='[OUTPUT]
Name http
Match *
Host logs.example-provider.com
Port 443
tls On'Tune Match rules so only application namespaces ship to the SaaS endpoint.
Verify#
- Generate a known log line (
logger.info("ship-logs-test")) and locate it in Loki or the SaaS search UI within one minute. - Confirm log volume and retention match your compliance requirements.
See also#
Usage Guidelines
The sample code, software libraries, command line tools, proofs of concept, templates, and other related technology on this page (including any of the foregoing that is provided by Quake AI personnel) is provided to you as Quake AI Content under the Quake AI Customer Agreement, or the relevant written agreement between you and Quake AI (whichever applies). Do not use this Quake AI Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the Quake AI Content (such as sample code) as appropriate for production grade use based on your specific quality control practices and standards. Deploying Quake AI Content may incur Quake AI charges for creating or using Quake AI chargeable resources, such as running Compute instances or storing data in Object Storage. Your use is also subject to the Acceptable Use Policy.
For the full policy, see Usage Guidelines.