Stream Logs
In this guide, we will cover how to enable the streaming of syslogs from vJailbreak VMs to Fluentd and integrate with Loki running on the PCD-CE. For this tutorial, you will need working knowledge of k3s, Fluentd and Loki.
Prerequisites
- A running vJailbreak instance with SSH enabled.
- A running Fluentd instance on a remote host (in this case, the PCD-CE Management host).
- A running grafana/loki-stack instance; for this, we are using PCD-CE.
Set up vJailbreak VM for rsyslog
- Enable k3s to write to
syslog. - Install rsyslog on the vJailbreak VM - if not already present.
- Configure rsyslog to forward syslogs to Fluentd.
- Configure Loki to read logs from the Fluentd log directory.
Enable syslog for k3s
To the service section add the following:
sudo vi /etc/systemd/system/k3s.service[Service]Type=notifyNotifyAccess=allTo the ExecStart section, add --log=/var/log/syslog after server. Keep the existing arguments: the vJailbreak VM runs k3s with --secrets-encryption, and removing it would turn off encryption of Kubernetes secrets at rest.
ExecStart=/usr/local/bin/k3s \ server --log=/var/log/syslog \ '--secrets-encryption' \ '--disable' \ 'traefik' \sudo systemctl daemon-reloadsudo systemctl restart k3sConfigure rsyslog
- Edit the rsyslog configuration file
- Add the following configuration:
sudo sh -c 'echo "*.* @<fluentd-host>:5140" >> /etc/rsyslog.d/90-fluentd.conf'Note: Change <fluentd-host> to the IP address of the Fluentd host.
- Restart rsyslog
sudo systemctl restart rsyslog- Test rsyslog
sudo journalctl -fInstall Fluentd
- Install Fluentd on the PCD-CE Management host
- Test Fluentd
$ ulimit -n65536Please add the following lines to your /etc/security/limits.conf file:
root soft nofile 65536root hard nofile 65536* soft nofile 65536* hard nofile 65536- Set up sysctl conf
Edit /etc/sysctl.conf and add the following:
net.core.somaxconn = 1024net.core.netdev_max_backlog = 5000net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_wmem = 4096 12582912 16777216net.ipv4.tcp_rmem = 4096 12582912 16777216net.ipv4.tcp_max_syn_backlog = 8096net.ipv4.tcp_slow_start_after_idle = 0net.ipv4.tcp_tw_reuse = 1net.ipv4.ip_local_port_range = 10240 65535fs.inotify.max_user_instances = 1024# If forward uses port 24224, reserve that port number for use as an ephemeral port.# If another port, e.g., monitor_agent uses port 24220, add a comma-separated list of port numbers.# net.ipv4.ip_local_reserved_ports = 24220,24224net.ipv4.ip_local_reserved_ports = 24224Then check if these are in effect:
$ sysctl -pnet.core.somaxconn = 1024net.core.netdev_max_backlog = 5000net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_wmem = 4096 12582912 16777216net.ipv4.tcp_rmem = 4096 12582912 16777216net.ipv4.tcp_max_syn_backlog = 8096net.ipv4.tcp_slow_start_after_idle = 0net.ipv4.tcp_tw_reuse = 1net.ipv4.ip_local_port_range = 10240 65535fs.inotify.max_user_instances = 1024net.ipv4.ip_local_reserved_ports = 24224- Install Fluentd
sudo curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-fluent-package5.sh | bash- Restart Fluentd
sudo systemctl restart fluentd- Test Fluentd
sudo systemctl status fluentdConfigure Fluentd
- Edit the Fluentd configuration file
/etc/fluentd/fluentd.conf - Add the following configuration:
<source> @type syslog port 5140 bind 0.0.0.0 tag system</source>
<match system.**> @type stdout</match>Ref: https://docs.fluentd.org/how-to-guides/parse-syslog
- Restart Fluentd
sudo systemctl restart fluentdGenerally, the logs should now show up in /var/log/fluent/fluentd.log.
Verify
- Check that rsyslog is running
sudo systemctl status rsyslog- Check that Fluentd is running
sudo systemctl status fluentd- Check that syslogs from vJailbreak are being sent to Fluentd
vjb$ logger -p vjailbreak.notice "This is a test message from Rsyslog - Hello Openstack!"- Check that Fluentd is receiving the logs
pcd$ tail -f /var/log/fluent/fluentd.logSet up Loki on PCD-CE
- Log in to the PCD-CE Management Host
- Then export the kubeconfig
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml- Install Loki using Helm Use the following loki-config.yaml
loki: image: tag: 2.9.3 enabled: true
grafana: enabled: false
promtail: enabled: true
config: server: http_listen_port: 3101 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://loki:3100/loki/api/v1/push snippets: extraScrapeConfigs: |- - job_name: fluentd static_configs: - targets: - localhost labels: job: fluentd __path__: /hostlogs/fluent/*.log pipeline_stages: - match: selector: '{job="fluentd"}' stages: - regex: expression: '.*' - timestamp: source: time format: RFC3339 - output: source: message
extraVolumes: - name: host-logs hostPath: path: /var/log/fluent type: Directory - name: tmp emptyDir: {}
extraVolumeMounts: - name: host-logs mountPath: /hostlogs/fluent readOnly: true - name: tmp mountPath: /tmp
serviceAccount: create: true
rbac: create: true
persistence: enabled: true size: 10Gi storageClassName: "" accessModes: - ReadWriteOncehelm upgrade --namespace pcd-community --install loki grafana/loki-stack -f loki-config.yaml- Check if all the Loki pods are
running
kubectl get pods -n pcd-community | grep loki- Add Loki as a data source in Grafana
-
Add it manually in the Grafana UI Configuration > “Add Datasource” > Loki > “url: http://loki:3100” > “Save & Test”
-
Add it using a ConfigMap
Add the ConfigMap
Terminal window kubectl apply -f loki-datasource.yamlRestart the deployment
Terminal window kubectl rollout restart deployment prometheus-stack-grafana
Go to “Explore” > “Loki” to start exploring the logs.
- You can use the query below to browse the logs
{job="fluentd"} |= ``Flow of Logs
architecture-beta
group vJailbreak(server)[vJailbreak VM]
service k3s(logos:kubernetes)[K3s] in vJailbreak
service syslog(disk)[Syslog] in vJailbreak
service rsyslogd(internet)[rsyslogd] in vJailbreak
group PCD(server)[PCD]
service fluentd(disk)[fluentd] in PCD
service loki(database)[Loki] in PCD
service grafana(logos:grafana)[Grafana] in PCD
k3s:L -- R:syslog
syslog:B -- T:rsyslogd
rsyslogd:R -- R:fluentd
fluentd:B -- T:loki
loki:L -- R:grafana
Version of tools used
- Fluentd
fluentd --versionfluent-package 5.2.0 fluentd 1.18.0 (46372ddd521870f6a203baefb5a598209486d0bc)- Loki & Grafana
NAME CHART APP VERSIONgrafana grafana-8.11.1 11.6.0loki loki-stack-2.10.2 v2.9.3