Compare commits

..

1 Commits

Author SHA1 Message Date
AlfonsoRBJ
7c3cbbb83e Bump version 3729 2018-12-20 18:41:27 +01:00
26 changed files with 294 additions and 784 deletions

View File

@@ -1,77 +1,6 @@
# Change Log
All notable changes to this project will be documented in this file.
## Wazuh Docker v3.8.2_6.7.0
### Changed
- Update Elastic Stack version to 6.7.0. ([#144](https://github.com/wazuh/wazuh-docker/pull/144))
## Wazuh Docker v3.8.2_6.6.2
### Changed
- Update Elastic Stack version to 6.6.2. ([#130](https://github.com/wazuh/wazuh-docker/pull/130))
## Wazuh Docker v3.8.2_6.6.1
### Changed
- Update Elastic Stack version to 6.6.1. ([#129](https://github.com/wazuh/wazuh-docker/pull/129))
## Wazuh Docker v3.8.2_6.5.4
### Added
- Add Wazuh-Elasticsearch. ([#106](https://github.com/wazuh/wazuh-docker/pull/106))
- Store Filebeat _/var/lib/filebeat/registry._ ([#109](https://github.com/wazuh/wazuh-docker/pull/109))
- Adding the option to disable some xpack features. ([#111](https://github.com/wazuh/wazuh-docker/pull/111))
- Wazuh-Kibana customizable at plugin level. ([#117](https://github.com/wazuh/wazuh-docker/pull/117))
- Adding env variables for alerts data flow. ([#118](https://github.com/wazuh/wazuh-docker/pull/118))
- New Logstash entrypoint added. ([#135](https://github.com/wazuh/wazuh-docker/pull/135/files))
- Welcome screen management. ([#133](https://github.com/wazuh/wazuh-docker/pull/133))
### Changed
- Update to Wazuh version 3.8.2. ([#105](https://github.com/wazuh/wazuh-docker/pull/105))
### Removed
- Remove alerts created in build time. ([#137](https://github.com/wazuh/wazuh-docker/pull/137))
## Wazuh Docker v3.8.1_6.5.4
### Changed
- Update to Wazuh version 3.8.1. ([#102](https://github.com/wazuh/wazuh-docker/pull/102))
## Wazuh Docker v3.8.0_6.5.4
### Changed
- Upgrade version 3.8.0_6.5.4. ([#97](https://github.com/wazuh/wazuh-docker/pull/97))
### Removed
- Remove cluster.py work around. ([#99](https://github.com/wazuh/wazuh-docker/pull/99))
## Wazuh Docker v3.7.2_6.5.4
### Added
- Improvements to Kibana settings added. ([#91](https://github.com/wazuh/wazuh-docker/pull/91))
- Add Kibana environmental variables for Wazuh APP config.yml. ([#89](https://github.com/wazuh/wazuh-docker/pull/89))
### Changed
- Update Elastic Stack version to 6.5.4. ([#82](https://github.com/wazuh/wazuh-docker/pull/82))
- Add env credentials for nginx. ([#86](https://github.com/wazuh/wazuh-docker/pull/86))
- Improve filebeat configuration ([#88](https://github.com/wazuh/wazuh-docker/pull/88))
### Fixed
- Temporary fix for Wazuh cluster master node in Kubernetes. ([#84](https://github.com/wazuh/wazuh-docker/pull/84))
## Wazuh Docker v3.7.2_6.5.3
### Changed

View File

@@ -1,5 +1,5 @@
Portions Copyright (C) 2019 Wazuh, Inc.
Portions Copyright (C) 2018 Wazuh, Inc.
Based on work Copyright (C) 2003 - 2013 Trend Micro, Inc.
This program is a free software; you can redistribute it and/or modify

View File

@@ -1,6 +1,6 @@
# Wazuh containers for Docker
[![Slack](https://img.shields.io/badge/slack-join-blue.svg)](https://wazuh.com/community/join-us-on-slack/)
[![Slack](https://img.shields.io/badge/slack-join-blue.svg)](https://goo.gl/forms/M2AoZC4b2R9A9Zy12)
[![Email](https://img.shields.io/badge/email-join-blue.svg)](https://groups.google.com/forum/#!forum/wazuh)
[![Documentation](https://img.shields.io/badge/docs-view-green.svg)](https://documentation.wazuh.com)
[![Documentation](https://img.shields.io/badge/web-view-green.svg)](https://wazuh.com)
@@ -11,9 +11,8 @@ In this repository you will find the containers to run:
* wazuh-logstash: It is used to receive alerts generated by the manager and feed Elasticsearch using an alerts template
* wazuh-kibana: Provides a web user interface to browse through alerts data. It includes Wazuh plugin for Kibana, that allows you to visualize agents configuration and status.
* wazuh-nginx: Proxies the Kibana container, adding HTTPS (via self-signed SSL certificate) and [Basic authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#Basic_authentication_scheme).
* wazuh-elasticsearch: An Elasticsearch container (working as a single-node cluster) using Elastic Stack Docker images. **Be aware to increase the `vm.max_map_count` setting, as it's detailed in the [Wazuh documentation](https://documentation.wazuh.com/current/docker/wazuh-container.html#increase-max-map-count-on-your-host-linux).**
In addition, a docker-compose file is provided to launch the containers mentioned above.
In addition, a docker-compose file is provided to launch the containers mentioned above. It also launches an Elasticsearch container (working as a single-node cluster) using Elastic Stack Docker images.
## Documentation
@@ -21,6 +20,10 @@ In addition, a docker-compose file is provided to launch the containers mentione
* [Wazuh documentation for Docker](https://documentation.wazuh.com/current/docker/index.html)
* [Docker hub](https://hub.docker.com/u/wazuh)
## Current release
Containers are currently tested on Wazuh version 3.7.2 and Elastic Stack version 6.5.3. We will do our best to keep this repository updated to latest versions of both Wazuh and Elastic Stack.
## Directory structure
wazuh-docker
@@ -59,9 +62,9 @@ In addition, a docker-compose file is provided to launch the containers mentione
## Branches
* `stable` branch on correspond to the latest Wazuh-Docker stable version.
* `stable` branch on correspond to the last Wazuh-Docker stable version.
* `master` branch contains the latest code, be aware of possible bugs on this branch.
* `Wazuh.Version_ElasticStack.Version` (for example 3.8.2_6.7.0) branch. This branch contains the current release referenced in Docker Hub. The container images are installed under the current version of this branch.
* `Wazuh.Version_ElsaticStack.Version` (for example 3.7.0_6.4.3) branch. This branch contains the current release referenced in Docker Hub. The container images are installed under the current version of this branch.
## Credits and Thank you
@@ -74,7 +77,7 @@ We thank you them and everyone else who has contributed to this project.
## License and copyright
Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
## Web references

View File

@@ -1,2 +1,2 @@
WAZUH-DOCKER_VERSION="3.8.2_6.7.0"
REVISION="3803"
WAZUH-DOCKER_VERSION="3.7.2_6.5.3"
REVISION="3729"

View File

@@ -1,9 +1,9 @@
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
version: '2'
services:
wazuh:
image: wazuh/wazuh:3.8.2_6.7.0
image: wazuh/wazuh:3.7.2_6.5.3
hostname: wazuh-manager
restart: always
ports:
@@ -11,26 +11,40 @@ services:
- "1515:1515"
- "514:514/udp"
- "55000:55000"
# - "1516:1516"
networks:
- docker_elk
# volumes:
# - my-path:/var/ossec/data:Z
# - my-path:/etc/postfix:Z
# - my-path:/etc/filebeat
# - my-custom-config-path/ossec.conf:/wazuh-config-mount/etc/ossec.conf
# command: ["echo 'hello world'"]
depends_on:
- logstash
logstash:
image: wazuh/wazuh-logstash:3.8.2_6.7.0
image: wazuh/wazuh-logstash:3.7.2_6.5.3
hostname: logstash
restart: always
# volumes:
# - my-path:/etc/logstash/conf.d:Z
links:
- elasticsearch:elasticsearch
ports:
- "5000:5000"
networks:
- docker_elk
depends_on:
- elasticsearch
environment:
- LS_HEAP_SIZE=2048m
elasticsearch:
image: wazuh/wazuh-elasticsearch:3.8.2_6.7.0
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.3
hostname: elasticsearch
restart: always
ports:
- "9200:9200"
# - "9300:9300"
environment:
- node.name=node-1
- cluster.name=wazuh
@@ -42,26 +56,46 @@ services:
soft: -1
hard: -1
mem_limit: 2g
# volumes:
# - my-path:/usr/share/elasticsearch/data:Z
networks:
- docker_elk
kibana:
image: wazuh/wazuh-kibana:3.8.2_6.7.0
image: wazuh/wazuh-kibana:3.7.2_6.5.3
hostname: kibana
restart: always
# ports:
# - "5601:5601"
# environment:
# - ELASTICSEARCH_URL=http://elasticsearch:9200
networks:
- docker_elk
depends_on:
- elasticsearch
links:
- elasticsearch:elasticsearch
- wazuh:wazuh
nginx:
image: wazuh/wazuh-nginx:3.8.2_6.7.0
image: wazuh/wazuh-nginx:3.7.2_6.5.3
hostname: nginx
restart: always
environment:
- NGINX_PORT=443
- NGINX_CREDENTIALS
ports:
- "80:80"
- "443:443"
# volumes:
# - my-path:/etc/nginx/conf.d:Z
networks:
- docker_elk
depends_on:
- kibana
links:
- kibana:kibana
networks:
docker_elk:
driver: bridge
ipam:
config:
- subnet: 172.25.0.0/24

View File

@@ -1,32 +0,0 @@
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
FROM docker.elastic.co/elasticsearch/elasticsearch:6.7.0
ENV ALERTS_SHARDS="1" \
ALERTS_REPLICAS="0"
ENV API_USER="foo" \
API_PASS="bar"
ENV XPACK_ML="true"
ENV ENABLE_CONFIGURE_S3="false"
ENV TEMPLATE_VERSION=v3.8.2
ADD https://raw.githubusercontent.com/wazuh/wazuh/$TEMPLATE_VERSION/extensions/elasticsearch/wazuh-elastic6-template-alerts.json /usr/share/elasticsearch/config
COPY config/entrypoint.sh /entrypoint.sh
RUN chmod 755 /entrypoint.sh
COPY --chown=elasticsearch:elasticsearch ./config/load_settings.sh ./
RUN chmod +x ./load_settings.sh
RUN bin/elasticsearch-plugin install --batch https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-6.7.0.zip
COPY config/configure_s3.sh ./config/configure_s3.sh
RUN chmod 755 ./config/configure_s3.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["elasticsearch"]

View File

@@ -1,76 +0,0 @@
#!/bin/bash
set -e
# Check number of arguments passed to configure_s3.sh. If it is different from 4 or 5, the process will finish with error.
# param 1: number of arguments passed to configure_s3.sh
function CheckArgs()
{
if [ $1 != 4 ] && [ $1 != 5 ];then
echo "Use: configure_s3.sh <Elastic_Server_IP:Port> <Bucket> <Path> <RepositoryName> (By default <current_elasticsearch_major_version> is added to the path and the repository name)"
echo "or use: configure_s3.sh <Elastic_Server_IP:Port> <Bucket> <Path> <RepositoryName> <Elasticsearch major version>"
exit 1
fi
}
# Create S3 repository from base_path <path>/<elasticsearch_major_version> (if there is no <Elasticsearch major version> argument, current version is added)
# Repository name would be <RepositoryName>-<elasticsearch_major_version> (if there is no <Elasticsearch major version> argument, current version is added)
# param 1: <Elastic_Server_IP:Port>
# param 2: <Bucket>
# param 3: <Path>
# param 4: <RepositoryName>
# param 5: Optional <Elasticsearch major version>
# output: It will show "acknowledged" if the repository has been successfully created
function CreateRepo()
{
elastic_ip_port="$2"
bucket_name="$3"
path="$4"
repository_name="$5"
if [ $1 == 5 ];then
version="$6"
else
version=`curl -s $elastic_ip_port | grep number | cut -d"\"" -f4 | cut -c1`
fi
if ! [[ "$version" =~ ^[0-9]+$ ]];then
echo "Elasticsearch major version must be an integer"
exit 1
fi
repository="$repository_name-$version"
s3_path="$path/$version"
curl -X PUT "$elastic_ip_port/_snapshot/$repository" -H 'Content-Type: application/json' -d'
{
"type": "s3",
"settings": {
"bucket": "'$bucket_name'",
"base_path": "'$s3_path'"
}
}
'
}
# Run functions CheckArgs and CreateRepo
# param 1: number of arguments passed to configure_s3.sh
# param 2: <Elastic_Server_IP:Port>
# param 3: <Bucket>
# param 4: <Path>
# param 5: <RepositoryName>
# param 6: Optional <Elasticsearch major version>
function Main()
{
CheckArgs $1
CreateRepo $1 $2 $3 $4 $5 $6
}
Main $# $1 $2 $3 $4 $5

View File

@@ -1,50 +0,0 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# For more information https://github.com/elastic/elasticsearch-docker/blob/6.5.4/build/elasticsearch/bin/docker-entrypoint.sh
set -e
# Files created by Elasticsearch should always be group writable too
umask 0002
run_as_other_user_if_needed() {
if [[ "$(id -u)" == "0" ]]; then
# If running as root, drop to specified UID and run command
exec chroot --userspec=1000 / "${@}"
else
# Either we are running in Openshift with random uid and are a member of the root group
# or with a custom --user
exec "${@}"
fi
}
#Disabling xpack features
elasticsearch_config_file="/usr/share/elasticsearch/config/elasticsearch.yml"
if grep -Fq "#xpack features" "$elasticsearch_config_file";
then
declare -A CONFIG_MAP=(
[xpack.ml.enabled]=$XPACK_ML
)
for i in "${!CONFIG_MAP[@]}"
do
if [ "${CONFIG_MAP[$i]}" != "" ]; then
sed -i 's/.'"$i"'.*/'"$i"': '"${CONFIG_MAP[$i]}"'/' $elasticsearch_config_file
fi
done
else
echo "
#xpack features
xpack.ml.enabled: $XPACK_ML
" >> $elasticsearch_config_file
fi
# Run load settings script.
./load_settings.sh &
# Execute elasticsearch
run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch

View File

@@ -1,98 +0,0 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
set -e
if [ "x${ELASTICSEARCH_URL}" = "x" ]; then
el_url="http://elasticsearch:9200"
else
el_url="${ELASTICSEARCH_URL}"
fi
if [ "x${WAZUH_API_URL}" = "x" ]; then
wazuh_url="https://wazuh"
else
wazuh_url="${WAZUH_API_URL}"
fi
until curl -XGET $el_url; do
>&2 echo "Elastic is unavailable - sleeping"
sleep 5
done
>&2 echo "Elastic is up - executing command"
if [ $ENABLE_CONFIGURE_S3 ]; then
#Wait for Elasticsearch to be ready to create the repository
sleep 10
IP_PORT="${ELASTICSEARCH_IP}:${ELASTICSEARCH_PORT}"
if [ "x$S3_PATH" != "x" ]; then
if [ "x$S3_ELASTIC_MAJOR" != "x" ]; then
./config/configure_s3.sh $IP_PORT $S3_BUCKET_NAME $S3_PATH $S3_REPOSITORY_NAME $S3_ELASTIC_MAJOR
else
./config/configure_s3.sh $IP_PORT $S3_BUCKET_NAME $S3_PATH $S3_REPOSITORY_NAME
fi
fi
fi
#Insert default templates
sed -i 's| "index.refresh_interval": "5s"| "index.refresh_interval": "5s", "number_of_shards" : '"${ALERTS_SHARDS}"', "number_of_replicas" : '"${ALERTS_REPLICAS}"'|' /usr/share/elasticsearch/config/wazuh-elastic6-template-alerts.json
cat /usr/share/elasticsearch/config/wazuh-elastic6-template-alerts.json | curl -XPUT "$el_url/_template/wazuh" -H 'Content-Type: application/json' -d @-
sleep 5
API_PASS_Q=`echo "$API_PASS" | tr -d '"'`
API_USER_Q=`echo "$API_USER" | tr -d '"'`
API_PASSWORD=`echo -n $API_PASS_Q | base64`
echo "Setting API credentials into Wazuh APP"
CONFIG_CODE=$(curl -s -o /dev/null -w "%{http_code}" -XGET $el_url/.wazuh/wazuh-configuration/1513629884013)
if [ "x$CONFIG_CODE" = "x404" ]; then
curl -s -XPOST $el_url/.wazuh/wazuh-configuration/1513629884013 -H 'Content-Type: application/json' -d'
{
"api_user": "'"$API_USER_Q"'",
"api_password": "'"$API_PASSWORD"'",
"url": "'"$wazuh_url"'",
"api_port": "55000",
"insecure": "true",
"component": "API",
"cluster_info": {
"manager": "wazuh-manager",
"cluster": "Disabled",
"status": "disabled"
},
"extensions": {
"oscap": true,
"audit": true,
"pci": true,
"aws": true,
"virustotal": true,
"gdpr": true,
"ciscat": true
}
}
' > /dev/null
else
echo "Wazuh APP already configured"
fi
sleep 5
curl -XPUT "$el_url/_cluster/settings" -H 'Content-Type: application/json' -d'
{
"persistent": {
"xpack.monitoring.collection.enabled": true
}
}
'
echo "Elasticsearch is ready."

View File

@@ -1,10 +1,12 @@
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
FROM docker.elastic.co/kibana/kibana:6.7.0
ARG WAZUH_APP_VERSION=3.8.2_6.7.0
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
FROM docker.elastic.co/kibana/kibana:6.5.3
ARG WAZUH_APP_VERSION=3.7.2_6.5.3
USER root
ADD https://packages.wazuh.com/wazuhapp/wazuhapp-${WAZUH_APP_VERSION}.zip /tmp
ADD https://raw.githubusercontent.com/wazuh/wazuh/3.7/extensions/elasticsearch/wazuh-elastic6-template-alerts.json /usr/share/kibana/config
RUN NODE_OPTIONS="--max-old-space-size=3072" /usr/share/kibana/bin/kibana-plugin install file:///tmp/wazuhapp-${WAZUH_APP_VERSION}.zip &&\
chown -R kibana:kibana /usr/share/kibana &&\
rm -rf /tmp/*
@@ -14,63 +16,4 @@ RUN chmod 755 /entrypoint.sh
USER kibana
ENV PATTERN="" \
CHECKS_PATTERN="" \
CHECKS_TEMPLATE="" \
CHECKS_API="" \
CHECKS_SETUP="" \
EXTENSIONS_PCI="" \
EXTENSIONS_GDPR="" \
EXTENSIONS_AUDIT="" \
EXTENSIONS_OSCAP="" \
EXTENSIONS_CISCAT="" \
EXTENSIONS_AWS="" \
EXTENSIONS_VIRUSTOTAL="" \
EXTENSIONS_OSQUERY="" \
APP_TIMEOUT="" \
WAZUH_SHARDS="" \
WAZUH_REPLICAS="" \
WAZUH_VERSION_SHARDS="" \
WAZUH_VERSION_REPLICAS="" \
IP_SELECTOR="" \
IP_IGNORE="" \
XPACK_RBAC_ENABLED="" \
WAZUH_MONITORING_ENABLED="" \
WAZUH_MONITORING_FREQUENCY="" \
WAZUH_MONITORING_SHARDS="" \
WAZUH_MONITORING_REPLICAS="" \
ADMIN_PRIVILEGES=""
ARG XPACK_CANVAS="true"
ARG XPACK_LOGS="true"
ARG XPACK_INFRA="true"
ARG XPACK_ML="true"
ARG XPACK_DEVTOOLS="true"
ARG XPACK_MONITORING="true"
ARG XPACK_APM="true"
ARG CHANGE_WELCOME="false"
COPY --chown=kibana:kibana ./config/wazuh_app_config.sh ./
RUN chmod +x ./wazuh_app_config.sh
COPY --chown=kibana:kibana ./config/kibana_settings.sh ./
RUN chmod +x ./kibana_settings.sh
COPY --chown=kibana:kibana ./config/xpack_config.sh ./
RUN chmod +x ./xpack_config.sh
RUN ./xpack_config.sh
COPY --chown=kibana:kibana ./config/welcome_wazuh.sh ./
RUN chmod +x ./welcome_wazuh.sh
RUN ./welcome_wazuh.sh
RUN /usr/local/bin/kibana-docker --optimize
ENTRYPOINT /entrypoint.sh

View File

@@ -1,12 +1,8 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
set -e
##############################################################################
# Waiting for elasticsearch
##############################################################################
if [ "x${ELASTICSEARCH_URL}" = "x" ]; then
el_url="http://elasticsearch:9200"
else
@@ -14,38 +10,47 @@ else
fi
until curl -XGET $el_url; do
>&2 echo "Elastic is unavailable - sleeping."
>&2 echo "Elastic is unavailable - sleeping"
sleep 5
done
sleep 2
>&2 echo "Elastic is up - executing command"
>&2 echo "Elasticsearch is up."
#Insert default templates
cat /usr/share/kibana/config/wazuh-elastic6-template-alerts.json | curl -XPUT "$el_url/_template/wazuh" -H 'Content-Type: application/json' -d @-
sleep 5
##############################################################################
# Waiting for wazuh alerts template
##############################################################################
strlen=0
while [[ $strlen -eq 0 ]]
do
template=$(curl $el_url/_cat/templates/wazuh -s)
strlen=${#template}
>&2 echo "Wazuh alerts template not loaded - sleeping."
sleep 2
done
sleep 2
>&2 echo "Wazuh alerts template is loaded."
./wazuh_app_config.sh
echo "Setting API credentials into Wazuh APP"
CONFIG_CODE=$(curl -s -o /dev/null -w "%{http_code}" -XGET $el_url/.wazuh/wazuh-configuration/1513629884013)
if [ "x$CONFIG_CODE" = "x404" ]; then
curl -s -XPOST $el_url/.wazuh/wazuh-configuration/1513629884013 -H 'Content-Type: application/json' -d'
{
"api_user": "foo",
"api_password": "YmFy",
"url": "https://wazuh",
"api_port": "55000",
"insecure": "true",
"component": "API",
"cluster_info": {
"manager": "wazuh-manager",
"cluster": "Disabled",
"status": "disabled"
},
"extensions": {
"oscap": true,
"audit": true,
"pci": true,
"aws": true,
"virustotal": true,
"gdpr": true,
"ciscat": true
}
}
' > /dev/null
else
echo "Wazuh APP already configured"
fi
sleep 5
./kibana_settings.sh &
/usr/local/bin/kibana-docker

92
kibana/config/kibana.yml Normal file
View File

@@ -0,0 +1,92 @@
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# This setting specifies the IP address of the back end server.
server.host: "0.0.0.0"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This setting
# cannot end in a slash.
# server.basePath: ""
# The maximum payload size in bytes for incoming server requests.
# server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
# server.name: "your-hostname"
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://elasticsearch:9200"
# When this settings value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
# elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesnt already exist.
# kibana.index: ".kibana"
# The default application to load.
# kibana.defaultAppId: "discover"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
# elasticsearch.username: "user"
# elasticsearch.password: "pass"
# Paths to the PEM-format SSL certificate and SSL key files, respectively. These
# files enable SSL for outgoing requests from the Kibana server to the browser.
# server.ssl.cert: /path/to/your/server.crt
# server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
# elasticsearch.ssl.cert: /path/to/your/client.crt
# elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
# elasticsearch.ssl.ca: /path/to/your/CA.pem
# To disregard the validity of SSL certificates, change this settings value to false.
# elasticsearch.ssl.verify: true
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
# elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
# elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
# elasticsearch.requestHeadersWhitelist: [ authorization ]
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
# elasticsearch.shardTimeout: 0
# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
# elasticsearch.startupTimeout: 5000
# Specifies the path where Kibana creates the process ID file.
# pid.file: /var/run/kibana.pid
# Enables you specify a file where Kibana stores log output.
# logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
# logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
logging.quiet: true
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
# logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 10000.
# ops.interval: 10000

View File

@@ -1,63 +0,0 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
WAZUH_MAJOR=3
##############################################################################
# Wait for the Kibana API to start. It is necessary to do it in this container
# because the others are running Elastic Stack and we can not interrupt them.
#
# The following actions are performed:
#
# Add the wazuh alerts index as default.
# Set the Discover time interval to 24 hours instead of 15 minutes.
# Do not ask user to help providing usage statistics to Elastic.
##############################################################################
##############################################################################
# Customize elasticsearch ip
##############################################################################
if [ "$ELASTICSEARCH_KIBANA_IP" != "" ]; then
sed -i "s/elasticsearch:9200/$ELASTICSEARCH_KIBANA_IP:9200/" /usr/share/kibana/config/kibana.yml
fi
if [ "$KIBANA_IP" != "" ]; then
kibana_ip="$KIBANA_IP"
else
kibana_ip="kibana"
fi
while [[ "$(curl -XGET -I -s -o /dev/null -w ''%{http_code}'' $kibana_ip:5601/status)" != "200" ]]; do
echo "Waiting for Kibana API. Sleeping 5 seconds"
sleep 5
done
# Prepare index selection.
echo "Kibana API is running"
default_index="/tmp/default_index.json"
cat > ${default_index} << EOF
{
"changes": {
"defaultIndex": "wazuh-alerts-${WAZUH_MAJOR}.x-*"
}
}
EOF
sleep 5
# Add the wazuh alerts index as default.
curl -POST "http://$kibana_ip:5601/api/kibana/settings" -H "Content-Type: application/json" -H "kbn-xsrf: true" -d@${default_index}
rm -f ${default_index}
sleep 5
# Configuring Kibana TimePicker.
curl -POST "http://$kibana_ip:5601/api/kibana/settings" -H "Content-Type: application/json" -H "kbn-xsrf: true" -d \
'{"changes":{"timepicker:timeDefaults":"{\n \"from\": \"now-24h\",\n \"to\": \"now\",\n \"mode\": \"quick\"}"}}'
sleep 5
# Do not ask user to help providing usage statistics to Elastic
curl -POST "http://$kibana_ip:5601/api/telemetry/v1/optIn" -H "Content-Type: application/json" -H "kbn-xsrf: true" -d '{"enabled":false}'
echo "End settings"

View File

@@ -1,40 +0,0 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
kibana_config_file="/usr/share/kibana/plugins/wazuh/config.yml"
declare -A CONFIG_MAP=(
[pattern]=$PATTERN
[checks.pattern]=$CHECKS_PATTERN
[checks.template]=$CHECKS_TEMPLATE
[checks.api]=$CHECKS_API
[checks.setup]=$CHECKS_SETUP
[extensions.pci]=$EXTENSIONS_PCI
[extensions.gdpr]=$EXTENSIONS_GDPR
[extensions.audit]=$EXTENSIONS_AUDIT
[extensions.oscap]=$EXTENSIONS_OSCAP
[extensions.ciscat]=$EXTENSIONS_CISCAT
[extensions.aws]=$EXTENSIONS_AWS
[extensions.virustotal]=$EXTENSIONS_VIRUSTOTAL
[extensions.osquery]=$EXTENSIONS_OSQUERY
[timeout]=$APP_TIMEOUT
[wazuh.shards]=$WAZUH_SHARDS
[wazuh.replicas]=$WAZUH_REPLICAS
[wazuh-version.shards]=$WAZUH_VERSION_SHARDS
[wazuh-version.replicas]=$WAZUH_VERSION_REPLICAS
[ip.selector]=$IP_SELECTOR
[ip.ignore]=$IP_IGNORE
[xpack.rbac.enabled]=$XPACK_RBAC_ENABLED
[wazuh.monitoring.enabled]=$WAZUH_MONITORING_ENABLED
[wazuh.monitoring.frequency]=$WAZUH_MONITORING_FREQUENCY
[wazuh.monitoring.shards]=$WAZUH_MONITORING_SHARDS
[wazuh.monitoring.replicas]=$WAZUH_MONITORING_REPLICAS
[admin]=$ADMIN_PRIVILEGES
)
for i in "${!CONFIG_MAP[@]}"
do
if [ "${CONFIG_MAP[$i]}" != "" ]; then
sed -i 's/.*#'"$i"'.*/'"$i"': '"${CONFIG_MAP[$i]}"'/' $kibana_config_file
fi
done

View File

@@ -1,23 +0,0 @@
#!/bin/bash
if [[ $CHANGE_WELCOME == "true" ]]
then
rm -rf ./optimize/bundles
kibana_path="/usr/share/kibana"
# Set Wazuh app as the default landing page
echo "Set Wazuh app as the default landing page"
echo "server.defaultRoute: /app/wazuh" >> /usr/share/kibana/config/kibana.yml
# Redirect Kibana welcome screen to Discover
echo "Redirect Kibana welcome screen to Discover"
sed -i "s:'/app/kibana#/home':'/app/wazuh':g" $kibana_path/src/ui/public/chrome/directives/global_nav/global_nav.html
sed -i "s:'/app/kibana#/home':'/app/wazuh':g" $kibana_path/src/ui/public/chrome/directives/header_global_nav/header_global_nav.js
# Redirect Kibana welcome screen to Discover
echo "Hide undesired links"
sed -i 's#visible: true#visible: false#g' $kibana_path/node_modules/x-pack/plugins/rollup/public/crud_app/index.js
sed -i 's#visible: true#visible: false#g' $kibana_path/node_modules/x-pack/plugins/license_management/public/management_section.js
fi

View File

@@ -1,36 +0,0 @@
#!/bin/bash
kibana_config_file="/usr/share/kibana/config/kibana.yml"
if grep -Fq "#xpack features" "$kibana_config_file";
then
declare -A CONFIG_MAP=(
[xpack.apm.ui.enabled]=$XPACK_APM
[xpack.grokdebugger.enabled]=$XPACK_DEVTOOLS
[xpack.searchprofiler.enabled]=$XPACK_DEVTOOLS
[xpack.ml.enabled]=$XPACK_ML
[xpack.canvas.enabled]=$XPACK_CANVAS
[xpack.logstash.enabled]=$XPACK_LOGS
[xpack.infra.enabled]=$XPACK_INFRA
[xpack.monitoring.enabled]=$XPACK_MONITORING
[console.enabled]=$XPACK_DEVTOOLS
)
for i in "${!CONFIG_MAP[@]}"
do
if [ "${CONFIG_MAP[$i]}" != "" ]; then
sed -i 's/.'"$i"'.*/'"$i"': '"${CONFIG_MAP[$i]}"'/' $kibana_config_file
fi
done
else
echo "
#xpack features
xpack.apm.ui.enabled: $XPACK_APM
xpack.grokdebugger.enabled: $XPACK_DEVTOOLS
xpack.searchprofiler.enabled: $XPACK_DEVTOOLS
xpack.ml.enabled: $XPACK_ML
xpack.canvas.enabled: $XPACK_CANVAS
xpack.logstash.enabled: $XPACK_LOGS
xpack.infra.enabled: $XPACK_INFRA
xpack.monitoring.enabled: $XPACK_MONITORING
console.enabled: $XPACK_DEVTOOLS
" >> $kibana_config_file
fi

View File

@@ -1,12 +1,6 @@
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
FROM docker.elastic.co/logstash/logstash:6.7.0
COPY --chown=logstash:logstash config/entrypoint.sh /entrypoint.sh
RUN chmod 755 /entrypoint.sh
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
FROM docker.elastic.co/logstash/logstash:6.5.3
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
COPY config/01-wazuh.conf /usr/share/logstash/pipeline/01-wazuh.conf
ENTRYPOINT /entrypoint.sh

View File

@@ -1,4 +1,4 @@
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
# Wazuh - Logstash configuration file
## Remote Wazuh Manager - Filebeat input
input {

View File

@@ -1,72 +0,0 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
#
# OSSEC container bootstrap. See the README for information of the environment
# variables expected by this script.
#
set -e
##############################################################################
# Waiting for elasticsearch
##############################################################################
if [ "x${ELASTICSEARCH_URL}" = "x" ]; then
el_url="http://elasticsearch:9200"
else
el_url="${ELASTICSEARCH_URL}"
fi
until curl -XGET $el_url; do
>&2 echo "Elastic is unavailable - sleeping."
sleep 5
done
sleep 2
>&2 echo "Elasticsearch is up."
##############################################################################
# Waiting for wazuh alerts template
##############################################################################
strlen=0
while [[ $strlen -eq 0 ]]
do
template=$(curl $el_url/_cat/templates/wazuh -s)
strlen=${#template}
>&2 echo "Wazuh alerts template not loaded - sleeping."
sleep 2
done
sleep 2
>&2 echo "Wazuh alerts template is loaded."
##############################################################################
# Customize logstash output ip
##############################################################################
if [ "$LOGSTASH_OUTPUT" != "" ]; then
>&2 echo "Customize Logstash ouput ip."
sed -i "s/elasticsearch:9200/$LOGSTASH_OUTPUT:9200/" /usr/share/logstash/pipeline/01-wazuh.conf
sed -i "s/elasticsearch:9200/$LOGSTASH_OUTPUT:9200/" /usr/share/logstash/config/logstash.yml
fi
##############################################################################
# Map environment variables to entries in logstash.yml.
# Note that this will mutate logstash.yml in place if any such settings are found.
# This may be undesirable, especially if logstash.yml is bind-mounted from the
# host system.
##############################################################################
env2yaml /usr/share/logstash/config/logstash.yml
export LS_JAVA_OPTS="-Dls.cgroup.cpuacct.path.override=/ -Dls.cgroup.cpu.path.override=/ $LS_JAVA_OPTS"
if [[ -z $1 ]] || [[ ${1:0:1} == '-' ]] ; then
exec logstash "$@"
else
exec "$@"
fi

31
logstash/config/run.sh Normal file
View File

@@ -0,0 +1,31 @@
#!/bin/bash
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
#
# OSSEC container bootstrap. See the README for information of the environment
# variables expected by this script.
#
#
#
# Apply Templates
#
set -e
host="elasticsearch"
until curl -XGET $host:9200; do
>&2 echo "Elastic is unavailable - sleeping"
sleep 1
done
# Add logstash as command if needed
if [ "${1:0:1}" = '-' ]; then
set -- logstash "$@"
fi
# Run as user "logstash" if the command is "logstash"
if [ "$1" = 'logstash' ]; then
set -- gosu logstash "$@"
fi
exec "$@"

View File

@@ -1,4 +1,4 @@
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
FROM nginx:latest
ENV DEBIAN_FRONTEND noninteractive
@@ -13,7 +13,4 @@ RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
VOLUME ["/etc/nginx/conf.d"]
ENV NGINX_NAME="foo" \
NGINX_PWD="bar"
ENTRYPOINT /entrypoint.sh

View File

@@ -1,5 +1,5 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
#!/bin/sh
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
set -e
@@ -12,37 +12,15 @@ else
echo "SSL certificates already present"
fi
# Setting users credentials.
# In order to set NGINX_CREDENTIALS, before "docker-compose up -d" run (a or b):
#
# a) export NGINX_CREDENTIALS="user1:pass1;user2:pass2;" or
# export NGINX_CREDENTIALS="user1:pass1;user2:pass2"
#
# b) Set NGINX_CREDENTIALS in docker-compose.yml:
# NGINX_CREDENTIALS=user1:pass1;user2:pass2; or
# NGINX_CREDENTIALS=user1:pass1;user2:pass2
#
# Configuring default credentiales.
if [ ! -f /etc/nginx/conf.d/kibana.htpasswd ]; then
echo "Setting users credentials"
if [ ! -z "$NGINX_CREDENTIALS" ]; then
IFS=';' read -r -a users <<< "$NGINX_CREDENTIALS"
for index in "${!users[@]}"
do
IFS=':' read -r -a credentials <<< "${users[index]}"
if [ $index -eq 0 ]; then
echo ${credentials[1]}|htpasswd -i -c /etc/nginx/conf.d/kibana.htpasswd ${credentials[0]} >/dev/null
else
echo ${credentials[1]}|htpasswd -i /etc/nginx/conf.d/kibana.htpasswd ${credentials[0]} >/dev/null
fi
done
else
# NGINX_PWD and NGINX_NAME are declared in nginx/Dockerfile
echo $NGINX_PWD|htpasswd -i -c /etc/nginx/conf.d/kibana.htpasswd $NGINX_NAME >/dev/null
fi
echo "Setting Nginx credentials"
echo bar|htpasswd -i -c /etc/nginx/conf.d/kibana.htpasswd foo >/dev/null
else
echo "Kibana credentials already configured"
fi
if [ "x${NGINX_PORT}" = "x" ]; then
NGINX_PORT=443
fi

View File

@@ -1,66 +1,81 @@
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
FROM phusion/baseimage:latest
ARG FILEBEAT_VERSION=6.7.0
ARG WAZUH_VERSION=3.8.2-1
ARG FILEBEAT_VERSION=6.5.3
ARG WAZUH_VERSION=3.7.2-1
ENV API_USER="foo" \
API_PASS="bar"
# Updating image
RUN apt-get update && apt-get upgrade -y -o Dpkg::Options::="--force-confold"
# Set repositories.
RUN set -x && echo "deb https://packages.wazuh.com/3.x/apt/ stable main" | tee /etc/apt/sources.list.d/wazuh.list && \
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add - && \
curl --silent --location https://deb.nodesource.com/setup_8.x | bash - && \
echo "postfix postfix/mailname string wazuh-manager" | debconf-set-selections && \
echo "postfix postfix/main_mailer_type string 'Internet Site'" | debconf-set-selections && \
groupadd -g 1000 ossec && useradd -u 1000 -g 1000 -d /var/ossec ossec
# Set Wazuh repository.
RUN echo "deb https://packages.wazuh.com/3.x/apt/ stable main" | tee /etc/apt/sources.list.d/wazuh.list
RUN curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
RUN add-apt-repository universe && apt-get update && apt-get upgrade -y -o Dpkg::Options::="--force-confold" && \
apt-get --no-install-recommends --no-install-suggests -y install openssl postfix bsd-mailx python-boto python-pip \
apt-transport-https vim expect nodejs python-cryptography mailutils libsasl2-modules wazuh-manager=${WAZUH_VERSION} \
wazuh-api=${WAZUH_VERSION} && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && rm -f \
/var/ossec/logs/alerts/*/*/*.log && rm -f /var/ossec/logs/alerts/*/*/*.json
# Set nodejs repository.
RUN curl --silent --location https://deb.nodesource.com/setup_8.x | bash -
# Adding first run script and entrypoint
COPY config/data_dirs.env /data_dirs.env
COPY config/init.bash /init.bash
COPY config/entrypoint.sh /entrypoint.sh
# Creating ossec user as uid:gid 1000:1000
RUN groupadd -g 1000 ossec
RUN useradd -u 1000 -g 1000 -d /var/ossec ossec
# Configure postfix
RUN echo "postfix postfix/mailname string wazuh-manager" | debconf-set-selections
RUN echo "postfix postfix/main_mailer_type string 'Internet Site'" | debconf-set-selections
# Add universe repository
RUN add-apt-repository universe
# Install packages
RUN apt-get update && apt-get -y install openssl postfix bsd-mailx python-boto python-pip \
apt-transport-https vim expect nodejs python-cryptography wazuh-manager=${WAZUH_VERSION} \
wazuh-api=${WAZUH_VERSION} mailutils libsasl2-modules
# Adding first run script.
ADD config/data_dirs.env /data_dirs.env
ADD config/init.bash /init.bash
# Sync calls are due to https://github.com/docker/docker/issues/9547
RUN chmod 755 /init.bash && \
sync && /init.bash && \
sync && rm /init.bash && \
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-amd64.deb &&\
dpkg -i filebeat-${FILEBEAT_VERSION}-amd64.deb && rm -f filebeat-${FILEBEAT_VERSION}-amd64.deb && \
chmod 755 /entrypoint.sh
RUN chmod 755 /init.bash &&\
sync && /init.bash &&\
sync && rm /init.bash
# Installing and configuring fiebeat
RUN curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-amd64.deb &&\
dpkg -i filebeat-${FILEBEAT_VERSION}-amd64.deb && rm -f filebeat-${FILEBEAT_VERSION}-amd64.deb
COPY config/filebeat.yml /etc/filebeat/
RUN chmod go-w /etc/filebeat/filebeat.yml
RUN chmod go-w /etc/filebeat/filebeat.yml
# Adding entrypoint
ADD config/entrypoint.sh /entrypoint.sh
RUN chmod 755 /entrypoint.sh
# Setting volumes
VOLUME ["/var/ossec/data"]
VOLUME ["/etc/filebeat"]
VOLUME ["/etc/postfix"]
VOLUME ["/var/lib/filebeat"]
# Services ports
EXPOSE 55000/tcp 1514/udp 1515/tcp 514/udp 1516/tcp
# Clean up
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Adding services
RUN mkdir /etc/service/wazuh && \
mkdir /etc/service/wazuh-api && \
mkdir /etc/service/postfix && \
mkdir /etc/service/filebeat
RUN mkdir /etc/service/wazuh
COPY config/wazuh.runit.service /etc/service/wazuh/run
COPY config/wazuh-api.runit.service /etc/service/wazuh-api/run
COPY config/postfix.runit.service /etc/service/postfix/run
COPY config/filebeat.runit.service /etc/service/filebeat/run
RUN chmod +x /etc/service/wazuh/run
RUN mkdir /etc/service/wazuh-api
COPY config/wazuh-api.runit.service /etc/service/wazuh-api/run
RUN chmod +x /etc/service/wazuh-api/run
RUN mkdir /etc/service/postfix
COPY config/postfix.runit.service /etc/service/postfix/run
RUN chmod +x /etc/service/postfix/run
RUN mkdir /etc/service/filebeat
COPY config/filebeat.runit.service /etc/service/filebeat/run
RUN chmod +x /etc/service/filebeat/run
RUN chmod +x /etc/service/wazuh-api/run && \
chmod +x /etc/service/wazuh/run && \
chmod +x /etc/service/postfix/run && \
chmod +x /etc/service/filebeat/run
# Run all services
ENTRYPOINT ["/entrypoint.sh"]

View File

@@ -1,5 +1,5 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
#
# OSSEC container bootstrap. See the README for information of the environment
@@ -107,6 +107,9 @@ else
print "No Wazuh configuration files to mount..."
fi
# Enabling ossec-authd.
exec_cmd "/var/ossec/bin/ossec-control enable auth"
function ossec_shutdown(){
${WAZUH_INSTALL_PATH}/bin/ossec-control stop;
}
@@ -129,28 +132,4 @@ do
exec_cmd_stdout "${CUSTOM_COMMAND}"
done
##############################################################################
# Change Wazuh API user credentials.
##############################################################################
pushd /var/ossec/api/configuration/auth/
echo "Change Wazuh API user credentials"
change_user="node htpasswd -b -c user $API_USER $API_PASS"
eval $change_user
popd
##############################################################################
# Customize filebeat output ip
##############################################################################
if [ "$FILEBEAT_OUTPUT" != "" ]; then
sed -i "s/logstash:5000/$FILEBEAT_OUTPUT:5000/" /etc/filebeat/filebeat.yml
fi
##############################################################################
# Start Wazuh Server.
##############################################################################
/sbin/my_init

View File

@@ -1,14 +1,14 @@
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
filebeat:
prospectors:
inputs:
- type: log
paths:
- "/var/ossec/logs/alerts/alerts.json"
document_type: json
- "/var/ossec/data/logs/alerts/alerts.json"
fields:
document_type: wazuh-alerts
json.message_key: log
json.keys_under_root: true
json.overwrite_keys: true
tail_files: true
output:
logstash:

View File

@@ -1,5 +1,5 @@
#!/bin/bash
# Wazuh App Copyright (C) 2019 Wazuh Inc. (License GPLv2)
# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
#
# Initialize the custom data directory layout