Once we configure the input file, we need to filter the input based on our needs so that we can identify which fields we need, and process them as per the required analysis.
A
filter
plugin will perform the intermediary processing on the input event. We can apply the filter conditionally based on certain fields.
Since our input file is a CSV file, we will use the csv
filter for the same. The csv
filter takes an event field that contains CSV formatted data, parses it, and stores it as individual fields. It can also parse data with any separator other than commas. A typical csv
filter is as follows:
filter { csv { columns => #Array of column names. separator => #String ; default -"," } }
The attribute
columns take the name of fields in our CSV file, which is optional. By default, the columns will be named as column 1
, column 2
, and so on.
The attribute
separator defines what character is used to separate the different columns in the...