Update README.md

Removes incorrect performance information on unsafe_statement. No longer relevant for current releases.

CHANGELOG.md

@@ -1,6 +1,13 @@
 # Change Log
 All notable changes to this project will be documented in this file, from 0.2.0.
 
+## [0.3.2] - 2016-09-15
+  - Adds long/bigint support to address https://github.com/theangryangel/logstash-output-jdbc/issues/61
+
+## [0.3.1] - 2016-08-28
+  - Adds connection_test configuration option, to prevent the connection test from occuring, allowing the error to be suppressed.
+    Useful for cockroachdb deployments. https://github.com/theangryangel/logstash-output-jdbc/issues/53 
+
 ## [0.3.0] - 2016-07-24
   - Brings tests from v5 branch, providing greater coverage
   - Removes bulk update support, due to inconsistent behaviour

README.md

@@ -1,6 +1,8 @@
 # logstash-output-jdbc 
 
-[![Build Status](https://travis-ci.org/theangryangel/logstash-output-jdbc.svg?branch=master)](https://travis-ci.org/theangryangel/logstash-output-jdbc)
+[![Build Status](https://travis-ci.org/theangryangel/logstash-output-jdbc.svg?branch=v2.x)](https://travis-ci.org/theangryangel/logstash-output-jdbc)
+
+⚠️ The logstash v2 version of the plugin does not contain all fixes covered by the v5 version. If you find an issue is resolved under v5, but require the same issue fixed under v2, please raise an issue and I will do my best to find the time to backport the fix. At this time I recommend Logstash v5 where possible.
 
 This plugin is provided as an external plugin and is not part of the Logstash project.
 
@@ -21,7 +23,7 @@ See CHANGELOG.md
 Released versions are available via rubygems, and typically tagged.
 
 For development:
-  - See master branch for logstash v5 (currently **development only**)
+  - See master branch for logstash v5 
   - See v2.x branch for logstash v2
   - See v1.5 branch for logstash v1.5 
   - See v1.4 branch for logstash 1.4
@@ -43,10 +45,11 @@ For development:
 | driver_auto_commit | Boolean | If the driver does not support auto commit, you should set this to false | No | True |
 | driver_jar_path | String | File path to jar file containing your JDBC driver. This is optional, and all JDBC jars may be placed in $LOGSTASH_HOME/vendor/jar/jdbc instead. | No | |
 | connection_string | String | JDBC connection URL | Yes | |
+| connection_test | Boolean | Run a JDBC connection test. Some drivers do not function correctly, and you may need to disable the connection test to supress an error. Cockroach with the postgres JDBC driver is such an example. | No | Yes |
 | username | String | JDBC username - this is optional as it may be included in the connection string, for many drivers | No | |
 | password | String | JDBC password - this is optional as it may be included in the connection string, for many drivers | No | |
 | statement | Array | An array of strings representing the SQL statement to run. Index 0 is the SQL statement that is prepared, all other array entries are passed in as parameters (in order). A parameter may either be a property of the event (i.e. "@timestamp", or "host") or a formatted string (i.e. "%{host} - %{message}" or "%{message}"). If a key is passed then it will be automatically converted as required for insertion into SQL. If it's a formatted string then it will be passed in verbatim. | Yes |  |
-| unsafe_statement | Boolean | If yes, the statement is evaluated for event fields - this allows you to use dynamic table names, etc. **This is highly dangerous** and you should **not** use this unless you are 100% sure that the field(s) you are passing in are 100% safe. Failure to do so will result in possible SQL injections. Please be aware that there is also a potential performance penalty as each event must be evaluated and inserted into SQL one at a time, where as when this is false multiple events are inserted at once. Example statement: [ "insert into %{table_name_field} (column) values(?)", "fieldname" ] | No | False |
+| unsafe_statement | Boolean | If yes, the statement is evaluated for event fields - this allows you to use dynamic table names, etc. **This is highly dangerous** and you should **not** use this unless you are 100% sure that the field(s) you are passing in are 100% safe. Failure to do so will result in possible SQL injections. Example statement: [ "insert into %{table_name_field} (column) values(?)", "fieldname" ] | No | False |
 | max_pool_size | Number | Maximum number of connections to open to the SQL server at any 1 time. Default set to same as Logstash default number of workers | No | 24 |
 | connection_timeout | Number | Number of seconds before a SQL connection is closed | No | 2800 |
 | flush_size | Number | Maximum number of entries to buffer before sending to SQL - if this is reached before idle_flush_time | No | 1000 |

lib/logstash/outputs/jdbc.rb

@@ -83,6 +83,9 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
   # Suitable for configuring retryable custom JDBC SQL state codes.
   config :retry_sql_states, validate: :array, default: []
 
+  # Run a connection test on start.
+  config :connection_test, validate: :boolean, default: true
+
   # Maximum number of sequential failed attempts, before we stop retrying.
   # If set to < 1, then it will infinitely retry.
   # At the default values this is a little over 10 minutes
@@ -148,10 +151,12 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
 
     validate_connection_timeout = (@connection_timeout / 1000) / 2
 
+    return unless @connection_test
+
     # Test connection
     test_connection = @pool.getConnection
     unless test_connection.isValid(validate_connection_timeout)
-      @logger.error('JDBC - Connection is not valid. Please check connection string or that your JDBC endpoint is available.')
+      @logger.error('JDBC - Connection is not reporting as validate. Either connection is invalid, or driver is not getting the appropriate response.')
     end
     test_connection.close
   end
@@ -275,7 +280,13 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
         # strftime appears to be the most reliable across drivers.
         statement.setString(idx + 1, value.time.strftime(STRFTIME_FMT))
       when Fixnum, Integer
-        statement.setInt(idx + 1, value)
+        # bit_length doesn't exist in the current version of ruby/jruby logstash targets
+        # and this seems quicker than doing some Math.log2(value < 0 ? -value : value+1).ceil shit
+        if value > 2147483647 or value < -2147483648
+          statement.setLong(idx + 1, value)
+        else
+          statement.setInt(idx + 1, value)
+        end
       when Float
         statement.setFloat(idx + 1, value)
       when String

logstash-output-jdbc.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name = 'logstash-output-jdbc'
-  s.version = "0.3.0"
+  s.version = "0.3.2"
   s.licenses = [ "Apache License (2.0)" ]
   s.summary = "This plugin allows you to output to SQL, via JDBC"
   s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
@@ -34,5 +34,5 @@ Gem::Specification.new do |s|
 
   s.add_development_dependency 'logstash-devutils'
 
-  s.add_development_dependency 'rubocop'
+  s.add_development_dependency 'rubocop', '0.41.2'
 end

spec/jdbc_spec_helper.rb

@@ -4,6 +4,32 @@ require 'stud/temporary'
 require 'java'
 require 'securerandom'
 
+RSpec.configure do |c|
+
+  def start_service(name)
+    cmd = "sudo /etc/init.d/#{name}* start"
+
+    `which systemctl`
+    if $?.success?
+      cmd = "sudo systemctl start #{name}"
+    end
+
+    `#{cmd}`
+  end
+
+  def stop_service(name)
+    cmd = "sudo /etc/init.d/#{name}* stop"
+
+    `which systemctl`
+    if $?.success?
+      cmd = "sudo systemctl stop #{name}"
+    end
+
+    `#{cmd}`
+  end
+
+end
+
 RSpec.shared_context 'rspec setup' do
   it 'ensure jar is available' do
     expect(ENV[jdbc_jar_env]).not_to be_nil, "#{jdbc_jar_env} not defined, required to run tests"
@@ -31,11 +57,11 @@ RSpec.shared_context 'when outputting messages' do
   end
 
   let(:jdbc_create_table) do
-    "CREATE table #{jdbc_test_table} (created_at datetime not null, message varchar(512) not null, message_sprintf varchar(512) not null, static_int int not null, static_bit bit not null)"
+    "CREATE table #{jdbc_test_table} (created_at datetime not null, message varchar(512) not null, message_sprintf varchar(512) not null, static_int int not null, static_bit bit not null, static_bigint bigint not null)"
   end
 
   let(:jdbc_statement) do
-    ["insert into #{jdbc_test_table} (created_at, message, message_sprintf, static_int, static_bit) values(?, ?, ?, ?, ?)", '@timestamp', 'message', 'sprintf-%{message}', 1, true]
+    ["insert into #{jdbc_test_table} (created_at, message, message_sprintf, static_int, static_bit, static_bigint) values(?, ?, ?, ?, ?, ?)", '@timestamp', 'message', 'sprintf-%{message}', 1, true, 4000881632477184]
   end
 
   let(:systemd_database_service) do
@@ -107,29 +133,21 @@ RSpec.shared_context 'when outputting messages' do
     # Check that everything is fine right now
     expect { p.multi_receive([event]) }.not_to raise_error
 
-    # Start a thread to stop and restart the service.
+    stop_service(systemd_database_service)
+
+    # Start a thread to restart the service after the fact.
     t = Thread.new(systemd_database_service) { |systemd_database_service|
-      start_stop_cmd = 'sudo /etc/init.d/%<service>s* %<action>s'
+      sleep 20
     
-      `which systemctl`
-      if $?.success?
-        start_stop_cmd = 'sudo systemctl %<action>s %<service>s'
-      end
-
-      cmd = start_stop_cmd % { action: 'stop', service: systemd_database_service }
-      `#{cmd}`
-      sleep 10
-
-      cmd = start_stop_cmd % { action: 'start', service: systemd_database_service }
-      `#{cmd}`
+      start_service(systemd_database_service)
     }
 
-    # Wait a few seconds to the service to stop
-    sleep 5
+    t.run
     
     expect(logger).to receive(:warn).at_least(:once).with(/JDBC - Exception. Retrying/, Hash)
     expect { p.multi_receive([event]) }.to_not raise_error
 
+    # Wait for the thread to finish
     t.join
   end
 end

spec/outputs/jdbc_derby_spec.rb

@@ -10,7 +10,7 @@ describe 'logstash-output-jdbc: derby', if: ENV['JDBC_DERBY_JAR'] do
   end
 
   let(:jdbc_create_table) do
-    "CREATE table #{jdbc_test_table} (created_at timestamp not null, message varchar(512) not null, message_sprintf varchar(512) not null, static_int int not null, static_bit boolean not null)"
+    "CREATE table #{jdbc_test_table} (created_at timestamp not null, message varchar(512) not null, message_sprintf varchar(512) not null, static_int int not null, static_bit boolean not null, static_bigint bigint not null)"
   end
  
   let(:jdbc_settings) do