Docker 1.8 with td-agent 2.2.1 container

This is a quick guide on setting up td-agent container with Docker 1.8

1. Create a directory to store your dockerfile and td-agent config file. 

mkdir ~/td-agent-es && cd ~/td-agent-es

2. Create a dockerfile as shown below
FROM ubuntu:14.04 ENV DEBIAN_FRONTEND noninteractive RUN apt-get update &&\ apt-get install -y --force-yes libssl1.0.0 curl RUN curl https://packages.treasuredata.com/GPG-KEY-td-agent | apt-key add - # add treasure data repository to apt RUN echo "deb [arch=amd64] http://packages.treasuredata.com/2/ubuntu/trusty/ trusty contrib" > /etc/apt/sources.list.d/treasure-data.list RUN apt-get update &&\ apt-get install -y --force-yes td-agent RUN /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch ADD td-agent.conf /etc/td-agent/td-agent.conf EXPOSE 24224 CMD ["td-agent"]
3. Create td-agent.conf file as shown below
## built-in TCP input ## @see http://docs.fluentd.org/articles/in_forward <source></source> type forward # match docker logs <match docker.=""> #type stdout type elasticsearch logstash_format true host "#{ENV['ES_PORT_9200_TCP_ADDR']}" port 9200 flush_interval 5s </match>
4. Build the docker image

docker build -t td-agent-es .

5. Start elasticsearch container. As you can see from the td-agent.conf, the td-agent container will link to this elasticsearch container. Please note that this setup is not using any persistent storage mapping. You can refer to this link for more information on how to setup that mapping.

docker run -d -p 9200:9200 -p 9300:9300 --name es elasticsearch

6. Start your td-agent container 

docker run -d -p 24224:24224 --link es:es --name td-agent td-agent-es

7. Launch a hello-world to see if td-agent is forwarding log to elasticsearch container. Please note that Docker 1.8 add fluentd as one of the logging-driver. So this command will not work if you are using Docker with older version

docker run --log-driver=fluentd --log-opt fluentd-address=[your-host-IP-address]:24224 hello-world

8. Use the following command to check if elasticsearch container is receiving log. You can check container ID or container name using docker ps -a

curl -XGET 'http://[your-host-IP-address]:9200/_all/_search?q=*'