3 Commits
v0.3.2 ... v2.x

Author SHA1 Message Date
Karl
efa84fb63f Update README.md 2017-04-01 12:20:41 +01:00
Karl
a9acb33adf Update README.md
Removes incorrect performance information on unsafe_statement. No longer relevant for current releases.
2016-10-05 11:48:18 +01:00
Karl Southern
1788c81f91 Backport travisci fix 2016-09-15 22:13:53 +01:00
2 changed files with 39 additions and 19 deletions

View File

@@ -2,6 +2,8 @@
[![Build Status](https://travis-ci.org/theangryangel/logstash-output-jdbc.svg?branch=v2.x)](https://travis-ci.org/theangryangel/logstash-output-jdbc) [![Build Status](https://travis-ci.org/theangryangel/logstash-output-jdbc.svg?branch=v2.x)](https://travis-ci.org/theangryangel/logstash-output-jdbc)
⚠️ The logstash v2 version of the plugin does not contain all fixes covered by the v5 version. If you find an issue is resolved under v5, but require the same issue fixed under v2, please raise an issue and I will do my best to find the time to backport the fix. At this time I recommend Logstash v5 where possible.
This plugin is provided as an external plugin and is not part of the Logstash project. This plugin is provided as an external plugin and is not part of the Logstash project.
This plugin allows you to output to SQL databases, using JDBC adapters. This plugin allows you to output to SQL databases, using JDBC adapters.
@@ -21,7 +23,7 @@ See CHANGELOG.md
Released versions are available via rubygems, and typically tagged. Released versions are available via rubygems, and typically tagged.
For development: For development:
- See master branch for logstash v5 (currently **development only**) - See master branch for logstash v5
- See v2.x branch for logstash v2 - See v2.x branch for logstash v2
- See v1.5 branch for logstash v1.5 - See v1.5 branch for logstash v1.5
- See v1.4 branch for logstash 1.4 - See v1.4 branch for logstash 1.4
@@ -47,7 +49,7 @@ For development:
| username | String | JDBC username - this is optional as it may be included in the connection string, for many drivers | No | | | username | String | JDBC username - this is optional as it may be included in the connection string, for many drivers | No | |
| password | String | JDBC password - this is optional as it may be included in the connection string, for many drivers | No | | | password | String | JDBC password - this is optional as it may be included in the connection string, for many drivers | No | |
| statement | Array | An array of strings representing the SQL statement to run. Index 0 is the SQL statement that is prepared, all other array entries are passed in as parameters (in order). A parameter may either be a property of the event (i.e. "@timestamp", or "host") or a formatted string (i.e. "%{host} - %{message}" or "%{message}"). If a key is passed then it will be automatically converted as required for insertion into SQL. If it's a formatted string then it will be passed in verbatim. | Yes | | | statement | Array | An array of strings representing the SQL statement to run. Index 0 is the SQL statement that is prepared, all other array entries are passed in as parameters (in order). A parameter may either be a property of the event (i.e. "@timestamp", or "host") or a formatted string (i.e. "%{host} - %{message}" or "%{message}"). If a key is passed then it will be automatically converted as required for insertion into SQL. If it's a formatted string then it will be passed in verbatim. | Yes | |
| unsafe_statement | Boolean | If yes, the statement is evaluated for event fields - this allows you to use dynamic table names, etc. **This is highly dangerous** and you should **not** use this unless you are 100% sure that the field(s) you are passing in are 100% safe. Failure to do so will result in possible SQL injections. Please be aware that there is also a potential performance penalty as each event must be evaluated and inserted into SQL one at a time, where as when this is false multiple events are inserted at once. Example statement: [ "insert into %{table_name_field} (column) values(?)", "fieldname" ] | No | False | | unsafe_statement | Boolean | If yes, the statement is evaluated for event fields - this allows you to use dynamic table names, etc. **This is highly dangerous** and you should **not** use this unless you are 100% sure that the field(s) you are passing in are 100% safe. Failure to do so will result in possible SQL injections. Example statement: [ "insert into %{table_name_field} (column) values(?)", "fieldname" ] | No | False |
| max_pool_size | Number | Maximum number of connections to open to the SQL server at any 1 time. Default set to same as Logstash default number of workers | No | 24 | | max_pool_size | Number | Maximum number of connections to open to the SQL server at any 1 time. Default set to same as Logstash default number of workers | No | 24 |
| connection_timeout | Number | Number of seconds before a SQL connection is closed | No | 2800 | | connection_timeout | Number | Number of seconds before a SQL connection is closed | No | 2800 |
| flush_size | Number | Maximum number of entries to buffer before sending to SQL - if this is reached before idle_flush_time | No | 1000 | | flush_size | Number | Maximum number of entries to buffer before sending to SQL - if this is reached before idle_flush_time | No | 1000 |

View File

@@ -4,6 +4,32 @@ require 'stud/temporary'
require 'java' require 'java'
require 'securerandom' require 'securerandom'
RSpec.configure do |c|
def start_service(name)
cmd = "sudo /etc/init.d/#{name}* start"
`which systemctl`
if $?.success?
cmd = "sudo systemctl start #{name}"
end
`#{cmd}`
end
def stop_service(name)
cmd = "sudo /etc/init.d/#{name}* stop"
`which systemctl`
if $?.success?
cmd = "sudo systemctl stop #{name}"
end
`#{cmd}`
end
end
RSpec.shared_context 'rspec setup' do RSpec.shared_context 'rspec setup' do
it 'ensure jar is available' do it 'ensure jar is available' do
expect(ENV[jdbc_jar_env]).not_to be_nil, "#{jdbc_jar_env} not defined, required to run tests" expect(ENV[jdbc_jar_env]).not_to be_nil, "#{jdbc_jar_env} not defined, required to run tests"
@@ -107,29 +133,21 @@ RSpec.shared_context 'when outputting messages' do
# Check that everything is fine right now # Check that everything is fine right now
expect { p.multi_receive([event]) }.not_to raise_error expect { p.multi_receive([event]) }.not_to raise_error
# Start a thread to stop and restart the service. stop_service(systemd_database_service)
# Start a thread to restart the service after the fact.
t = Thread.new(systemd_database_service) { |systemd_database_service| t = Thread.new(systemd_database_service) { |systemd_database_service|
start_stop_cmd = 'sudo /etc/init.d/%<service>s* %<action>s' sleep 20
`which systemctl` start_service(systemd_database_service)
if $?.success?
start_stop_cmd = 'sudo systemctl %<action>s %<service>s'
end
cmd = start_stop_cmd % { action: 'stop', service: systemd_database_service }
`#{cmd}`
sleep 10
cmd = start_stop_cmd % { action: 'start', service: systemd_database_service }
`#{cmd}`
} }
# Wait a few seconds to the service to stop t.run
sleep 5
expect(logger).to receive(:warn).at_least(:once).with(/JDBC - Exception. Retrying/, Hash) expect(logger).to receive(:warn).at_least(:once).with(/JDBC - Exception. Retrying/, Hash)
expect { p.multi_receive([event]) }.to_not raise_error expect { p.multi_receive([event]) }.to_not raise_error
# Wait for the thread to finish
t.join t.join
end end
end end