timebeat.yml has its default output templated for an elastic database however you may find that you want to ship it to Logstash first (another elastic product - we would only recommend this is you are familiar with the Logstash pipelining system). Or you may wish to simply log it to a file as we mention in our before you begin guide.
It is advised that you consult Timebeat support with questions on this area if you are unsure.
For a breakdown of each section continue reading:
# ================================== General =================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging # ================================= Dashboards ================================= # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here or by using the `setup` command. setup.dashboards.enabled: false # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the artifacts.elastic.co # website. #setup.dashboards.url: #windows directory path: #setup.dashboards.directory: ${path.home}\kibana #Linux directory path: #setup.dashboards.directory: ${path.home}/kibana # ================================== Outputs =================================== # Configure what output to use when sending the data collected by the beat. # ---------------------------- Elasticsearch Output ---------------------------- output.elasticsearch: # Array of hosts to connect to. hosts: ['localhost:9200'] # insert ip address of your elastic server # hosts: ['elastic.customer.timebeat.app:9200'] # host format when using timebeat cloud # Protocol - either `http` (default) or `https` (https is required for timebeat cloud). #protocol: 'https' # Authentication credentials - either API key or username/password. Must match your elastic details, only applicable if configured in elastic. #api_key: "id:api_key" #username: 'elastic' #password: 'changeme' # Timebeat Cloud PKI connectivity details # ssl.certificate_authorities: ['/etc/timebeat/pki/ca.crt'] # ssl.certificate: '/etc/timebeat/pki/timebeat.crt' # ssl.key: '/etc/timebeat/pki/timebeat.key' # ssl.verification_mode: 'certificate' # setup.ilm.rollover_alias: 'timebeat' # ================================== Logging =================================== # There are four options for the log output: file, stderr, syslog, eventlog # The file output is the default. # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: info # If enabled, {{.BeatName | title}} periodically logs its internal metrics that have changed # in the last period. For each metric that changed, the delta from the value at # the beginning of the period is logged. Also, the total values for # all non-zero internal metrics are logged on shutdown. The default is true. #logging.metrics.enabled: true # The period after which to log the internal metrics. The default is 30s. #logging.metrics.period: 30s # Logging to rotating files. Set logging.to_files to false to disable logging to # files. logging.to_files: true logging.files: # Configure the path where the logs are written. The default is the logs directory # under the home path (the binary location). path: /var/log/{{.BeatName}} # The name of the files where the logs are written to. name: {{.BeatName}} # Configure log file size limit. If limit is reached, log file will be # automatically rotated rotateeverybytes: 10485760 # = 10MB # Number of rotated log files to keep. Oldest files will be deleted first. keepfiles: 7 # The permissions mask to apply when rotating log files. The default value is 0600. # Must be a valid Unix-style file permissions mask expressed in octal notation. permissions: 0600 # Enable log file rotation on time intervals in addition to size-based rotation. # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h # are boundary-aligned with minutes, hours, days, weeks, months, and years as # reported by the local system clock. All other intervals are calculated from the # Unix epoch. Defaults to disabled. #interval: 0 # Rotate existing logs on startup rather than appending to the existing # file. Defaults to true. # rotateonstartup: true # ============================== Process Security ============================== # Disable the filtering of syscalls as we require system clock access seccomp.enabled: false # ============================= X-Pack Monitoring ============================== # Timebeat can export internal metrics to a central Elasticsearch monitoring # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The # reporting is disabled by default. # Set to true to enable the monitoring reporter. monitoring.enabled: true # Sets the UUID of the Elasticsearch cluster under which monitoring data for this # Timebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch. #monitoring.cluster_uuid: # Uncomment to send the metrics to Elasticsearch. Most settings from the # Elasticsearch output are accepted here as well. # Note that the settings should point to your Elasticsearch *monitoring* cluster. # Any setting that is not set is automatically inherited from the Elasticsearch # output configuration, so if you have the Elasticsearch output configured such # that it is pointing to your Elasticsearch monitoring cluster, you can simply # uncomment the following line. monitoring.elasticsearch:
General:
# ================================== General =================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging
This is where you can customise what the reported hostname is for the device running Timebeat. By default, Timebeat will log with the hostname configured to your device, but if you wish to log an alternative name you can amend as below:
name: alternative device name
Tags are an important part of the Timebeat platform when it comes to reporting, monitoring, and alerting. Tags allow you to give specific labels to devices (as many as you like) this allows you to customise how your visualisations appear in your front end. As an example you may only want to report on a subsection of your server estate, let's say by location in this case London and New York. You can tag the respective servers in London with a London tag and New York a New York tag, then within your front end, you can view the various groups of servers by location.
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
Optional fields are rarely used, these are in place in the event you want to reduce or enrich the reported data from Timebeat. There are limitations as to what this can do but a guide to this can be found on the elastic website (link here). We don't believe this setting is ever required as we believe all reported data is important and holds the level of enrichment required, but feel free to experiment.
Dashboards:
# ================================= Dashboards ================================= # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here or by using the `setup` command. setup.dashboards.enabled: false # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the artifacts.elastic.co # website. #setup.dashboards.url: #windows directory path: #setup.dashboards.directory: ${path.home}\kibana #Linux directory path: #setup.dashboards.directory: ${path.home}/kibana
This element of the config file is specifically related to the Kibana Dashboard which ships for Free with all versions of Timebeat.
By default the dashboard setup, it configured to false. When set to true Timebeat will attempt to set up a default dashboard within Kibana. If a dashboard already exists in Kibana the application will overwrite the dashboard of matching name.
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
setup.dashboards.enabled: false
It is recommended that only one Timebeat application is configured to set up dashboards as they will overwrite the dashboard in Kibana at startup if it has the same dashboard name.
We recommend only configuring this to true if you are using Timebeat FREE, for Timebeat Enterprise users we recommend using the more detailed Grafana dashboards.
You should have been provided the Grafana dashboards upon order however if not you can contact Timebeat technical support who will be able to assist with this.
#setup.dashboards.url:
#windows directory path:
#setup.dashboards.directory: ${path.home}\kibana
#Linux directory path:
#setup.dashboards.directory: ${path.home}/kibana
The above snippet is where you will specify the path location to the dashboard you are looking to load into Kibana. It is unlikely you will need to use the top config line setup.dashbaords.url: because this relates to dashboards stored online which currently Timebeat has no online repository for.
The important path directories to note are the Windows and Linux paths. Provided you have installed Timebeat as it should by default (I.e. you haven't moved it from its default install paths) all you need to do is uncomment out the config line that matches your Operating System and the Timebeat app will do all the rest.
Outputs:
# ================================== Outputs =================================== # Configure what output to use when sending the data collected by the beat. # ---------------------------- Elasticsearch Output ---------------------------- output.elasticsearch: # Array of hosts to connect to. hosts: ['localhost:9200'] # insert ip address of your elastic server # hosts: ['elastic.customer.timebeat.app:9200'] # host format when using timebeat cloud # Protocol - either `http` (default) or `https` (https is required for timebeat cloud). #protocol: 'https' # Authentication credentials - either API key or username/password. Must match your elastic details, only applicable if configured in elastic. #api_key: 'id:api_key' #username: 'elastic' #password: 'changeme' # Timebeat Cloud PKI connectivity details # ssl.certificate_authorities: ['/etc/timebeat/pki/ca.crt'] # ssl.certificate: '/etc/timebeat/pki/timebeat.crt' # ssl.key: '/etc/timebeat/pki/timebeat.key' # ssl.verification_mode: 'certificate' # setup.ilm.rollover_alias: 'timebeat'
Elasticsearch is the default logging output of Timebeat. When running your own installation of elastic and not utilising Timebeat Cloud you should modify the hosts: setting to the IP addresses of your elastic search installation. As an example:
output.elasticsearch: # Array of hosts to connect to. hosts: ['localhost:9200'] # insert ip address of your elastic server
Multiple IP addresses can be inserted into the above config line, each ip in quotes and comma separated for example: ['1.2.3.4:9200', '2.3.4.5:9200']
To direct your Timebeat reporting to Timebeat Cloud the below config snippet is used. A member of the Timebeat team will always be able to assist in this area of the config as they will have all the required information for your unique installation of Timebeat Cloud. That said the below information may help with any quick questions.
# hosts: ['elastic.customer.timebeat.app:9200'] # host format when using timebeat cloud # Protocol - either `http` (default) or `https` (https is required for timebeat cloud). #protocol: 'https' # Authentication credentials - either API key or username/password. Must match your elastic details, only applicable if configured in elastic. #api_key: 'id:api_key' #username: 'elastic' #password: 'changeme' # Timebeat Cloud PKI connectivity details # ssl.certificate_authorities: ['/etc/timebeat/pki/ca.crt'] # ssl.certificate: '/etc/timebeat/pki/timebeat.crt' # ssl.key: '/etc/timebeat/pki/timebeat.key' # ssl.verification_mode: 'certificate' # setup.ilm.rollover_alias: 'timebeat'
First you need to modify the host address, typically this means replacing customer with your company name however if you are unsure check with Timebeat support.
Timebeat Cloud requires https to be uncommented out so that it registers as using the https protocol.
For security your Timebeat application will require a Username and Password to access the database. This will be provided to you upon initial deployment and the Timebeat Support team can provide this again if required.
Upon signing up to Timebeat Cloud you will be issued with 3 files for enabling your system to access our Cloud securely and privately. If you require these to be re-issued get in touch with Timebeat Support. For more information on connecting to Timebeat Cloud check out the guide here
It is possible you may wish to configure Timebeat to output to file which is briefly mentioned in our Before you Begin guide and is a method for getting Timebeat operating in a very raw form with no dependencies.
To do this comment out all of the elastic output section and instead insert the below configuration:
output.file: path: "/tmp/filebeat" filename: filebeat #rotate_every_kb: 10000 #number_of_files: 7 #permissions: 0600
Logging
# There are four options for the log output: file, stderr, syslog, eventlog # The file output is the default. # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: info # If enabled, {{.BeatName | title}} periodically logs its internal metrics that have changed # in the last period. For each metric that changed, the delta from the value at # the beginning of the period is logged. Also, the total values for # all non-zero internal metrics are logged on shutdown. The default is true. #logging.metrics.enabled: true # The period after which to log the internal metrics. The default is 30s. #logging.metrics.period: 30s # Logging to rotating files. Set logging.to_files to false to disable logging to # files. logging.to_files: true logging.files: # Configure the path where the logs are written. The default is the logs directory # under the home path (the binary location). path: /var/log/{{.BeatName}} # The name of the files where the logs are written to. name: {{.BeatName}} # Configure log file size limit. If limit is reached, log file will be # automatically rotated rotateeverybytes: 10485760 # = 10MB # Number of rotated log files to keep. Oldest files will be deleted first. keepfiles: 7 # The permissions mask to apply when rotating log files. The default value is 0600. # Must be a valid Unix-style file permissions mask expressed in octal notation. permissions: 0600 # Enable log file rotation on time intervals in addition to size-based rotation. # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h # are boundary-aligned with minutes, hours, days, weeks, months, and years as # reported by the local system clock. All other intervals are calculated from the # Unix epoch. Defaults to disabled. #interval: 0 # Rotate existing logs on startup rather than appending to the existing # file. Defaults to true. # rotateonstartup: true
It is recommended that you do not modify the above sections however it provides the necessary configuration lines for you to carry out your own log file management. It should be noted that this only effects the log file and not the reporting of Timebeat to your elastic installation.
Process Security & X-Pack Monitoring:
# ============================== Process Security ============================== # Disable the filtering of syscalls as we require system clock access seccomp.enabled: false # ============================= X-Pack Monitoring ============================== # Timebeat can export internal metrics to a central Elasticsearch monitoring # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The # reporting is disabled by default. # Set to true to enable the monitoring reporter. monitoring.enabled: true # Sets the UUID of the Elasticsearch cluster under which monitoring data for this # Timebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch. #monitoring.cluster_uuid: # Uncomment to send the metrics to Elasticsearch. Most settings from the # Elasticsearch output are accepted here as well. # Note that the settings should point to your Elasticsearch *monitoring* cluster. # Any setting that is not set is automatically inherited from the Elasticsearch # output configuration, so if you have the Elasticsearch output configured such # that it is pointing to your Elasticsearch monitoring cluster, you can simply # uncomment the following line. monitoring.elasticsearch:
Process security by default sets seccomp.enabled to false. This is critical to the operation of the Timebeat application. If this is not configured to false then Timebeat will be unable to modify the system clock.
X-Pack monitoring is useful when you are running your own installation and when utilising Timebeat Cloud. If using Timebeat Cloud this allows us to have more monitoring metrics of the health to your database system in respect of the individual indices (reporting Timebeat clients) which allows us to support you better and even pro-actively resolve problems so that you don't even notice.
Similarly, this can work to your favour when running your own elasticsearch instance as you will then receive this additional monitoring information to give you a greater understanding of your database health.
By default, this is set to true. To disable simply comment out the below lines:
# ============================= X-Pack Monitoring ============================== # Timebeat can export internal metrics to a central Elasticsearch monitoring # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The # reporting is disabled by default. # Set to true to enable the monitoring reporter. monitoring.enabled: true # Sets the UUID of the Elasticsearch cluster under which monitoring data for this # Timebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch. #monitoring.cluster_uuid: # Uncomment to send the metrics to Elasticsearch. Most settings from the # Elasticsearch output are accepted here as well. # Note that the settings should point to your Elasticsearch *monitoring* cluster. # Any setting that is not set is automatically inherited from the Elasticsearch # output configuration, so if you have the Elasticsearch output configured such # that it is pointing to your Elasticsearch monitoring cluster, you can simply # uncomment the following line. monitoring.elasticsearch: