Skip to content

D
docker-elk
Commit 2ed8ce30 authored by Antoine Cotten's avatar Antoine Cotten Committed by Anthony Lapenna
Browse files
Options

Use new official images (#97)

parent a785d5e9
Hide whitespace changes
Inline Side-by-side
Showing with 97 additions and 126 deletions
README.md
View file @ 2ed8ce30
......@@ -8,9 +8,9 @@ It will give you the ability to analyze any data set by using the searching/aggr
Based on the official images:
* [elasticsearch](https://registry.hub.docker.com/_/elasticsearch/)
* [logstash](https://registry.hub.docker.com/_/logstash/)
* [kibana](https://registry.hub.docker.com/_/kibana/)
* [elasticsearch](https://github.com/elastic/elasticsearch-docker)
* [logstash](https://github.com/elastic/logstash-docker)
* [kibana](https://github.com/elastic/kibana-docker)
**Note**: This version has [Search Guard support](https://github.com/floragunncom/search-guard).
......@@ -28,7 +28,7 @@ Default configuration of Search Guard in this repo is:
Existing users:
* admin (password: admin): No restrictions for this user, can do everything
* logstash (password: logstash): CRUD permissions for logstash-* index
* logstash (password: logstash): CRUD permissions for logstash-\* index
* kibanaro (password: kibanaro): Kibana user which can read every index
* kibanaserver (password: kibanaserver): User for the Kibana server (all permissions for .kibana index)
......@@ -72,10 +72,10 @@ You can also choose to run it in background (detached mode):
$ docker-compose up -d
```
After elasticsearch is started Search Guard have to be initialized:
Search Guard must be initialized after Elasticrsearch is started:
```bash
$ docker exec -it dockerelk_elasticsearch_1 /init_sg.sh
$ docker exec dockerelk_elasticsearch_1 bin/init_sg.sh
```
_This executes sgadmin and load the configuration in elasticsearch/config/sg*.yml_
......@@ -115,9 +115,27 @@ The Kibana default configuration is stored in `kibana/config/kibana.yml`.
## How can I tune Logstash configuration?
The logstash configuration is stored in `logstash/config/logstash.conf`.
The Logstash container is using the [shipped configuration](https://github.com/elastic/logstash-docker/blob/master/build/logstash/config/logstash.yml).
The folder `logstash/config` is mapped onto the container `/etc/logstash/conf.d` so you
If you want to override the default configuration, create a file `logstash/config/logstash.conf` and add your configuration in it.
Then, you'll need to map your configuration file inside the container in the `docker-compose.yml`. Update the logstash container declaration to:
```yml
logstash:
build: logstash/
volumes:
- ./logstash/pipeline:/usr/share/logstash/pipeline
- ./logstash/config:/usr/share/logstash/config
ports:
- "5000:5000"
networks:
- docker_elk
depends_on:
- elasticsearch
```
In the above example the folder `logstash/config` is mapped onto the container `/usr/share/logstash/config` so you
can create more than one file in that folder if you'd like to. However, you must be aware that config files will be read from the directory in alphabetical order.
## How can I specify the amount of memory used by Logstash?
......@@ -129,9 +147,8 @@ If you want to override the default configuration, add the *LS_HEAP_SIZE* enviro
```yml
logstash:
build: logstash/
command: -f /etc/logstash/conf.d/
volumes:
- ./logstash/config:/etc/logstash/conf.d
- ./logstash/pipeline:/usr/share/logstash/pipeline
ports:
- "5000:5000"
networks:
......@@ -147,7 +164,7 @@ logstash:
To add plugins to logstash you have to:
1. Add a RUN statement to the `logstash/Dockerfile` (ex. `RUN logstash-plugin install logstash-filter-json`)
2. Add the associated plugin code configuration to the `logstash/config/logstash.conf` file
2. Add the associated plugin code configuration to the `logstash/pipeline/logstash.conf` file
## How can I enable a remote JMX connection to Logstash?
......@@ -158,9 +175,8 @@ Update the container in the `docker-compose.yml` to add the *LS_JAVA_OPTS* envir
```yml
logstash:
build: logstash/
command: -f /etc/logstash/conf.d/
volumes:
- ./logstash/config:/etc/logstash/conf.d
- ./logstash/pipeline:/usr/share/logstash/pipeline
ports:
- "5000:5000"
networks:
......@@ -173,7 +189,7 @@ logstash:
## How can I tune Elasticsearch configuration?
The Elasticsearch container is using the shipped configuration and it is not exposed by default.
The Elasticsearch container is using the [shipped configuration](https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml).
If you want to override the default configuration, create a file `elasticsearch/config/elasticsearch.yml` and add your configuration in it.
......@@ -193,17 +209,18 @@ elasticsearch:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
```
You can also specify the options you want to override directly in the command field:
You can also specify the options you want to override directly via environment variables:
```yml
elasticsearch:
build: elasticsearch/
command: elasticsearch -Des.network.host=_non_loopback_ -Des.cluster.name: my-cluster
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xms1g -Xmx1g"
network.host: "_non_loopback_"
cluster.name: "my-cluster"
networks:
- docker_elk
```
......@@ -219,12 +236,13 @@ In order to persist Elasticsearch data even after removing the Elasticsearch con
```yml
elasticsearch:
build: elasticsearch/
command: elasticsearch -Des.network.host=_non_loopback_ -Des.cluster.name: my-cluster
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xms1g -Xmx1g"
network.host: "_non_loopback_"
cluster.name: "my-cluster"
networks:
- docker_elk
volumes:
......
docker-compose.yml
View file @ 2ed8ce30
......@@ -8,13 +8,19 @@ services:
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xms1g -Xmx1g"
# disable X-Pack
# see https://www.elastic.co/guide/en/x-pack/current/xpack-settings.html
# https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html#xpack-enabling
xpack.security.enabled: "false"
xpack.monitoring.enabled: "false"
xpack.graph.enabled: "false"
xpack.watcher.enabled: "false"
networks:
- docker_elk
logstash:
build: logstash/
command: -f /etc/logstash/conf.d/
volumes:
- ./logstash/config:/etc/logstash/conf.d
- ./logstash/pipeline:/usr/share/logstash/pipeline
ports:
- "5000:5000"
networks:
......@@ -24,7 +30,7 @@ services:
kibana:
build: kibana/
volumes:
- ./kibana/config/:/etc/kibana/
- ./kibana/config/:/usr/share/kibana/config
ports:
- "5601:5601"
networks:
......
elasticsearch/Dockerfile
View file @ 2ed8ce30
FROM elasticsearch:5.2.0
# https://github.com/elastic/elasticsearch-docker
FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.1
COPY config/ /etc/elasticsearch
COPY config/ config/
COPY bin/ bin/
RUN elasticsearch-plugin install --batch com.floragunn:search-guard-5:5.2.0-10
RUN printf "#!/bin/bash\n/usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin.sh -cd /etc/elasticsearch -ts /etc/elasticsearch/truststore.jks -ks /etc/elasticsearch/kirk-keystore.jks -nhnv -icl" > /init_sg.sh
RUN chmod +x /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin.sh
RUN chmod +x /init_sg.sh
CMD ["-E", "path.conf=/etc/elasticsearch", "-E", "network.host=0.0.0.0", "-E", "discovery.zen.minimum_master_nodes=1"]
RUN elasticsearch-plugin install --batch com.floragunn:search-guard-5:5.2.1-11 \
&& chmod +x \
plugins/search-guard-5/tools/hash.sh \
plugins/search-guard-5/tools/sgadmin.sh
elasticsearch/bin/init_sg.sh 0 → 100755
View file @ 2ed8ce30
#!/bin/sh
plugins/search-guard-5/tools/sgadmin.sh -cd config/ -ts config/truststore.jks -ks config/kirk-keystore.jks -nhnv -icl
elasticsearch/config/elasticsearch.yml
View file @ 2ed8ce30
## Defaults from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0
# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1
## Search Guard
#
searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
......
kibana/Dockerfile
View file @ 2ed8ce30
FROM kibana:5.2.0
# https://github.com/elastic/kibana-docker
FROM docker.elastic.co/kibana/kibana:5.2.1
RUN kibana-plugin install https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.2.0-1/searchguard-kibana-5.2.0-1.zip
RUN kibana-plugin install https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.2.1-1/searchguard-kibana-5.2.1-1.zip
kibana/config/kibana.yml
View file @ 2ed8ce30
searchguard.cookie.password: "123567818187654rwrwfsfshdhdhtegdhfzftdhncn"
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# This setting specifies the IP address of the back end server.
server.host: "0.0.0.0"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This setting
# cannot end in a slash.
# server.basePath: ""
# The maximum payload size in bytes for incoming server requests.
# server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
# server.name: "your-hostname"
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://elasticsearch:9200"
# When this setting’s value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
# elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn’t already exist.
# kibana.index: ".kibana"
# The default application to load.
# kibana.defaultAppId: "discover"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
---
## Default Kibana configuration from kibana-docker.
## from https://github.com/elastic/kibana-docker/blob/master/build/kibana/config/kibana.yml
#
server.name: kibana
server.host: "0"
elasticsearch.url: http://elasticsearch:9200
elasticsearch.username: elastic
elasticsearch.password: changeme
xpack.monitoring.ui.container.elasticsearch.enabled: false
## Disable X-Pack
## see https://www.elastic.co/guide/en/x-pack/current/xpack-settings.html
## https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html#xpack-enabling
#
xpack.security.enabled: false
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.reporting.enabled: false
## Custom configuration
#
elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"
# Paths to the PEM-format SSL certificate and SSL key files, respectively. These
# files enable SSL for outgoing requests from the Kibana server to the browser.
# server.ssl.cert: /path/to/your/server.crt
# server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
# elasticsearch.ssl.cert: /path/to/your/client.crt
# elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
# elasticsearch.ssl.ca: /path/to/your/CA.pem
# To disregard the validity of SSL certificates, change this setting’s value to false.
# elasticsearch.ssl.verify: true
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
# elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
# elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
# elasticsearch.requestHeadersWhitelist: [ authorization ]
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
# elasticsearch.shardTimeout: 0
# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
# elasticsearch.startupTimeout: 5000
# Specifies the path where Kibana creates the process ID file.
# pid.file: /var/run/kibana.pid
# Enables you specify a file where Kibana stores log output.
# logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
# logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
# logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
# logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 10000.
# ops.interval: 10000
searchguard.cookie.password: "123567818187654rwrwfsfshdhdhtegdhfzftdhncn"
logstash/Dockerfile
View file @ 2ed8ce30
FROM logstash:5.2.0
# https://github.com/elastic/logstash-docker
FROM docker.elastic.co/logstash/logstash:5.2.1
# Add your logstash plugins setup here
# Example: RUN logstash-plugin install logstash-filter-json
logstash/config/.placeholder 0 → 100644
View file @ 2ed8ce30
Ensure the existence of the parent folder.
logstash/config/logstash.conf → logstash/pipeline/logstash.conf
View file @ 2ed8ce30
......@@ -9,7 +9,7 @@ input {
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "logstash"
password => "logstash"
user => logstash
password => logstash
}
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment