Docker Logging Through Syslog
You can send syslog from your applications to Loggly by linking them with the Loggly Docker container. It uses ryslog to listen for syslog events and then forwards them to Loggly. Docker will automatically manage the port mapping. It’s made available by SendGrid Labs on GitHub or Docker Hub. These instructions were tested with Docker client version 1.3.0 and nginx 1.7.7. For alternatives, see the Advanced Options below.
Docker Syslog Setup
1. Start the Loggly Docker Container
Run the following command to download and run the Loggly docker container. The rsyslog daemon is running inside this container and will send syslog to Loggly. This will also open a high numbered port on the host machine, which maps to port 514 inside the container where rsyslog will receive it and send it to Loggly.
sudo docker run -d -p 514/udp --name loggly-docker -e TOKEN=TOKEN -e TAG=Docker sendgridlabs/loggly-docker
Replace:
- TOKEN: your customer token from the source setup page
Note: You may get some standard output like shown below. You can ignore this and rsyslog will continue running as usual.
rsyslogd: imklog: cannot open kernel log(/proc/kmsg): Operation not permitted. rsyslogd: activation of module imklog failed [try https://www.rsyslog.com/e/2145 ]
2. Send Test Logs From The Host
You can verify the container is running and see the port that Docker has opened up for syslog by running
sudo docker ps -a
Here you can see it’s redirecting port 49154 on the host to port 514 in the container
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1a9d7496ae42 sendgridlabs/loggly-docker:latest "/tmp/run.sh" 18 minutes ago Up 5 seconds 514/tcp, 0.0.0.0:49154->514/udp loggly-docker
If you send test messages to the host’s port, they will be sent to Loggly
echo netcat:"Host test log" | nc -u -w 1 127.0.0.1 UDP_PORT
Replace:
- UDP_PORT: the high numbered port as shown above that maps to port 514 inside the loggly container
3. Link To Other Containers
You can link other containers to Loggly’s container so that all your syslog gets sent to Loggly. Docker will automatically inject environment variables telling you the IP and Port to send syslog to.
In this example, we will configure an Nginx container to send syslog to Loggly. Run the Nginx Docker container in interactive terminal mode and link it with the running loggly-docker container
sudo docker run -i -t --name nginx --link loggly-docker:loggly nginx /bin/bash
Now run the following command inside the container to check the linked environment variables.
env | grep LOGGLY_PORT_514_UDP
Here is an example output with the values for my variables. Yours will be different.
LOGGLY_PORT_514_UDP_PORT=514 LOGGLY_PORT_514_UDP_ADDR=172.17.0.5
4. Send Test Logs From Nginx Container
Try sending a test event from inside the Nginx container. You can use netcat to confirm the link is working and that logs are reaching to Loggly.
apt-get install netcat echo netcat:"Nginx test log" | nc -u -w 1 $LOGGLY_PORT_514_UDP_ADDR $LOGGLY_PORT_514_UDP_PORT
5. Configure Nginx for Syslog
We can change nginx’s configuration to log over syslog to our Loggly container instead. Here’s how you can edit the Dockerfile of the existing Nginx container.
Insert the line below in the Nginx Docker file before the command which starts the Nginx server inside the container. This will configure the Nginx container to send logs to the “loggly” link we just set up.
RUN sed -i "s/server {/server {n n error_log syslog_server=loggly;n access_log syslog_server=loggly;n/" /etc/nginx/conf.d/default.conf
Build the image and run your container normally. Then, visit a webpage from the host machine to generate a log. The access log event should show up inside Loggly.
6. Verify Events
Search Loggly for events with the Docker tag over the past 20 minutes. It may take a few minutes to index the events. If it doesn’t work, see the troubleshooting section below.
tag:Docker
Advanced Docker Syslog Options
- Docker Logging Strategies – See the top ways to log on Docker and pick the best one for you
- Linking Docker Containers – More details on how container linking works in Docker
- Shippable and Ansible – A customer blog post about how to set these up with Loggly
- Loggly Libraries Catalog – You can bypass Docker and log from applications directly using our HTTP/S libraries
- Search or post your own Docker syslog and Docker logging question in the community forum.
Docker Logging Troubleshooting
If you don’t see any data show up in the verification step, then check for these common problems.
Check Docker Container:
- Wait a few minutes in case indexing needs to catch up
- Verify the container is running and that it has mapped port 514 by running sudo docker ps -a
- Send test events from inside each of the containers and from the host to see which point in the chain is dropping logs
- Make sure your app is sending syslog to the injected environment variable address instead of the usual syslog address
- See our Rsyslog Troubleshooting Guide if the files are not being sent to Loggly
- If you see any GnuTLS error then you can use the updated docker image from Docker Hub also available on GitHub. See Pull Request where you can get both the updated Loggly certificate and updated rsyslog format.
Still Not Working?
- Search or post your own Docker logs, Docker daemon, or other Docker question in the community forum.
