Heya.
I’m still pretty new to the homelab scene, so the more detail you can add the better. I’d like to add some sort of log aggregation tool, something like Elastic, where I can go to look at logs from any of my systems that aren’t working, or just make sure I don’t miss any errors.
Pretty much everything I run is set up as a Proxmox LXC from Proxmox helper scripts, which most of the time means it’s running as a systemctl service. Sometimes they run in Alpine instead, and a few of my apps also run in Docker.
What’s a good app to aggregate logs from those sources? I’ve heard of Prometheus, Grafana and Loki but not sure if they do what I’m after, they seem pretty overwhelming and more focused on metrics, whereas I want to be able to search for and view logs. I’d appreciate if you also mention the basic steps to send the logs from each container to said app.
this is the main pointer
On every device (but the central syslogserver or you create a loop that fills the drive in mere seconds)
/etc/rsyslog.d/99-centralsyslog.conf
$PreserveFQDN on *.* @192.168.1.66then on the central syslog server 192.168.1.66
/etc/rsyslog.d/01-syslog_receiver.conf
# provides UDP syslog reception module(load="imudp") input(type="imudp" port="514") # provides TCP syslog reception module(load="imtcp") input(type="imtcp" port="514")and also
/etc/rsyslog.d/20-save2postgresql.conf
# Load the PostgreSQL output module module(load="ompgsql") # Template for inserting logs template(name="pgsql-template" option.sql="on" type="string" string="INSERT INTO system_events (hostname, facility, priority, tag, message) VALUES ('%HOSTNAME%', %syslogfacility%, %syslogpriority%, '%syslogtag%', '%msg%' )") # Send logs to PostgreSQL *.emerg :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template *.panic :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template *.alert :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template *.crit :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template *.error :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template *.err :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template #*.warning :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template #*.warn :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template #*.notice :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template #*.info :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template #*.debug :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-templateMake sure you install postgres, the rsyslog-psql module and create the database and tables.
Grafana can run on the same or any other server.