Merge pull request #34 from Lytro/aws_ha_release_ruby

Aws ha release ruby

.gitignore

@@ -1,3 +1,5 @@
 # OS generated files #
 ######################
-.DS_Store
+.DS_Store
+tmp
+*.gem

Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in aws-missing-tools.gemspec
+gemspec

Gemfile.lock

@@ -0,0 +1,32 @@
+PATH
+  remote: .
+  specs:
+    aws-missing-tools (0.0.1)
+      aws-sdk
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    aws-sdk (1.11.0)
+      json (~> 1.4)
+      nokogiri (>= 1.4.4)
+      uuidtools (~> 2.1)
+    diff-lcs (1.1.3)
+    json (1.8.0)
+    nokogiri (1.5.9)
+    rspec (2.12.0)
+      rspec-core (~> 2.12.0)
+      rspec-expectations (~> 2.12.0)
+      rspec-mocks (~> 2.12.0)
+    rspec-core (2.12.2)
+    rspec-expectations (2.12.1)
+      diff-lcs (~> 1.1.3)
+    rspec-mocks (2.12.2)
+    uuidtools (2.1.4)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  aws-missing-tools!
+  rspec

README.md

@@ -0,0 +1,33 @@
+# AWS HA Release
+
+## Introduction:
+aws-ha-release allows the high-availability / no downtime replacement of all EC2 Instances in an Auto Scaling Group that is behind an Elastic Load Balancer.
+#
+## Potential Use:
+Some potential uses for aws-ha-release are listed below:
+
+1. Delivery of new code - if your deployment scheme utilizes the termination of EC2 instances in order to release new code aws-ha-release provides an automated way to do this without incurring any downtime.
+
+2. Return of all EC2 instances to "pristine" or "vanilla" state - all older EC2 instances can be replaced with newer EC2 instances.
+
+## Directions For Use:
+### Example of Use:
+`aws-ha-release.sh -a my-scaling-group`
+
+the above example would terminate and replace each EC2 Instance in the Auto Scaling group "my-scaling-group" with a new EC2 Instance.
+### Required Options:
+aws-ha-release.sh requires the following option:
+`-a, --as-group-name GROUP_NAME` - the name of the Auto Scaling Group for which you wish to perform a high availability release.
+### Optional Parameters:
+`-r, --region REGION` - allows you specify the region in which your Auto Scaling Group is in. By default aws-ha-release assumes the "us-east-1" region.
+`-t, --elb-timeout TIME` - time, in seconds, in which an EC2 instance should be given to complete request processing prior to being terminated. Set this value high enough so that any requests sent through an ELB would have time to be completed by an EC2 instance. For example: if the ELB allows connections to stay open for 120 seconds then setting this value to 120 seconds allows an instance behind an ELB 120 seconds to complete all processing before being terminated. By default both an AWS ELB and aws-ha-release.sh utilize 60 seconds timeout period.
+`-i, --inservice-time-allowed TIME` - allows you to specify the number of seconds an EC2 instance is provided to come into service. By default EC2 instances are given 300 seconds to come into service - if aws-ha-release.sh notices that an instance has not come into service in 300 seconds it will exit and return an exit status of 79. If an EC2 instance and application combination requires more than 300 seconds to come "InService" from the perspective of an ELB then this value should be set to a greater number.
+`-m, --min-inservice-time TIME` - Minimum time an instance must be in service before it is considered healthy (seconds). Default is 30 seconds. See [issue 32](https://github.com/colinbjohnson/aws-missing-tools/issues/32).
+`-o, --aws_access_key AWS_ACCESS_KEY` - your AWS Access Key. If not specified, must be available as an environment variable called AWS_ACCESS_KEY. If you specify the secret key, you must also specify the access key.
+`-s, --aws_secret_key AWS_SECRET_KEY` - your AWS Secret Key. If not specified, must be available as an environment variable called AWS_SECRET_KEY. If you specify the access key, you must also specify the secret key.
+
+# Additional Information:
+- Authors: Colin Johnson / colin@cloudavail.com, Anuj Biyani / abiyani@lytro.com
+- Date: 2013-05-30
+- Version 0.0.1
+- License Type: GNU GENERAL PUBLIC LICENSE, Version 3

aws-ha-release/README.md

@@ -1,27 +0,0 @@
-# Introduction:
-aws-ha-release allows the high-availability / no downtime replacement of all EC2 Instances in an Auto Scaling Group that is behind an Elastic Load Balancer.
-#
-# Potential Use:
-Some potential uses for aws-ha-release are listed below:
-
-1. Delivery of new code - if your deployment scheme utilizes the termination of EC2 instances in order to release new code aws-ha-release provides an automated way to do this without incurring any downtime.
-
-2. Return of all EC2 instances to "pristine" or "vanilla" state - all older EC2 instances can be replaced with newer EC2 instances.
-
-# Directions For Use:
-## Example of Use:
-`aws-ha-release.sh -a my-scaling-group`
-
-the above example would terminate and replace each EC2 Instance in the Auto Scaling group "my-scaling-group" with a new EC2 Instance.
-## Required Options:
-aws-ha-release.sh requires the following option:
-`-a <auto-scaling-group-name>` - the name of the Auto Scaling Group for which you wish to perform a high availability release.
-## Optional Parameters:
-`-r <region>` - allows you specify the region in which your Auto Scaling Group is in. By default aws-ha-release assumes the "us-east-1" region.
-`-t <elb_timeout>` - time, in seconds, in which an EC2 instance should be given to complete request processing prior to being terminated. Set this value high enough so that any requests sent through an ELB would have time to be completed by an EC2 instance. For example: if the ELB allows connections to stay open for 120 seconds then setting this value to 120 seconds allows an instance behind an ELB 120 seconds to complete all processing before being terminated. By default both an AWS ELB and aws-ha-release.sh utilize 60 seconds timeout period.
-`-i <inservice_time_allowed>` - allows you to specify the number of seconds an EC2 instance is provided to come into service. By default EC2 instances are given 300 seconds to come into service - if aws-ha-release.sh notices that an instance has not come into service in 300 seconds it will exit and return an exit status of 79. If an EC2 instance and application combination requires more than 300 seconds to come "InService" from the perspective of an ELB then this value should be set to a greater number.
-# Additional Information:
-- Author: Colin Johnson / colin@cloudavail.com
-- Date: 2012-12-09
-- Version 0.1
-- License Type: GNU GENERAL PUBLIC LICENSE, Version 3

aws-missing-tools.gemspec

@@ -0,0 +1,21 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path('../lib/aws-missing-tools/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.authors       = ['Colin Johnson', 'Anuj Biyani']
+  gem.email         = %w(colin@cloudavail.com abiyani@lytro.com)
+  gem.description   = %q{Extensions to Amazon's AWS CLI Tools.}
+  gem.summary       = %q{A collection of useful tools to supplement the AWS CLI Tools. Many of these tools depend on official AWS tools to function.}
+  gem.homepage      = 'https://github.com/colinbjohnson/aws-missing-tools/'
+
+  gem.files         = `git ls-files`.split("\n")
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.name          = 'aws-missing-tools'
+  gem.require_paths = %w(lib)
+  gem.version       = AwsMissingTools::VERSION
+
+  gem.add_dependency 'aws-sdk'
+
+  gem.add_development_dependency 'rspec'
+end

bin/aws-ha-release.rb

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+
+require 'aws-missing-tools'
+AwsMissingTools::AwsHaRelease.new(ARGV.dup).execute!

lib/aws-missing-tools.rb

@@ -0,0 +1,6 @@
+require 'aws-missing-tools/version'
+
+module AwsMissingTools
+  require 'aws-sdk'
+  require 'aws-missing-tools/aws-ha-release/aws-ha-release'
+end

lib/aws-missing-tools/aws-ha-release/aws-ha-release.rb

@@ -0,0 +1,193 @@
+require 'timeout'
+require 'optparse'
+
+module AwsMissingTools
+  class AwsHaRelease
+    attr_reader :group
+
+    PROCESSES_TO_SUSPEND = %w(ReplaceUnhealthy AlarmNotification ScheduledActions AZRebalance)
+    INSERVICE_POLLING_TIME = 10
+
+    def initialize(argv)
+      STDOUT.sync = true
+
+      @opts = AwsHaRelease.parse_options(argv)
+
+      AWS.config(access_key_id: @opts[:aws_access_key], secret_access_key: @opts[:aws_secret_key], region: @opts[:region])
+
+      @as = AWS::AutoScaling.new
+      @group = @as.groups[@opts[:as_group_name]]
+
+      if @group.nil?
+        raise ArgumentError, "The Auto Scaling Group named #{@opts[:as_group_name]} does not exist in #{@opts[:region]}."
+      end
+
+      @max_size_change = 0
+      @time_spent_inservice = 0
+    end
+
+    def self.parse_options(arguments)
+      options = {
+        region: 'us-east-1',
+        elb_timeout: 60,
+        inservice_time_allowed: 300,
+        min_inservice_time: 30
+      }
+
+      OptionParser.new('Usage: aws-ha-release.rb -a <group name> [options]', 50) do |opts|
+        opts.on('-a', '--as-group-name GROUP_NAME', 'AutoScaling Group Name') do |v|
+          options[:as_group_name] = v
+        end
+
+        opts.on('-r', '--region REGION', 'Region') do |v|
+          options[:region] = v
+        end
+
+        opts.on('-t', '--elb-timeout TIME', 'ELB Timeout (seconds)') do |v|
+          options[:elb_timeout] = v.to_i
+        end
+
+        opts.on('-i', '--inservice-time-allowed TIME', 'Time allowed for instance to come in service (seconds)') do |v|
+          options[:inservice_time_allowed] = v.to_i
+        end
+
+        opts.on('-m', '--min-inservice-time TIME', 'Minimum time an instance must be in service before it is considered healthy (seconds)') do |v|
+          options[:min_inservice_time] = v.to_i
+        end
+
+        opts.on('-o', '--aws_access_key AWS_ACCESS_KEY', 'AWS Access Key') do |v|
+          options[:aws_access_key] = v
+        end
+
+        opts.on('-s', '--aws_secret_key AWS_SECRET_KEY', 'AWS Secret Key') do |v|
+          options[:aws_secret_key] = v
+        end
+      end.parse!(arguments)
+
+      raise OptionParser::MissingArgument, 'You must specify the AutoScaling Group Name: aws-ha-release.rb -a <group name>' if options[:as_group_name].nil?
+
+      if options[:aws_secret_key] && options[:aws_access_key].nil? || options[:aws_access_key] && options[:aws_secret_key].nil?
+        raise OptionParser::MissingArgument, 'If specifying either the AWS Access or Secret Key, then the other must also be specified. aws-ha-release.rb -a <group name> -o access_key -s secret_key'
+      elsif options[:aws_secret_key].nil? && options[:aws_access_key].nil?
+        options[:aws_access_key] = ENV['AWS_ACCESS_KEY']
+        options[:aws_secret_key] = ENV['AWS_SECRET_KEY']
+      end
+
+      options
+    end
+
+    def execute!
+      %w(RemoveFromLoadBalancerLowPriority Terminate Launch HealthCheck AddToLoadBalancer).each do |process|
+        if @group.suspended_processes.keys.include? process
+          raise "AutoScaling process #{process} is currently suspended on #{@group.name} but is necessary for this script."
+        end
+      end
+
+      @group.suspend_processes PROCESSES_TO_SUSPEND
+
+      if @group.max_size == @group.desired_capacity
+        puts "#{@group.name} has a max-size of #{@group.max_size}. In order to recycle instances max-size will be temporarily increased by 1."
+        @group.update(max_size: @group.max_size + 1)
+        @max_size_change = 1
+      end
+
+      @group.update(desired_capacity: @group.desired_capacity + 1)
+
+      puts "The list of instances in Auto Scaling Group #{@group.name} that will be terminated is:\n#{@group.auto_scaling_instances.map{ |i| i.ec2_instance.id }.to_ary}"
+      @group.auto_scaling_instances.each do |instance|
+        time_taken = 0
+
+        begin
+          Timeout::timeout(@opts[:inservice_time_allowed]) do
+
+            until all_instances_inservice_for_time_period?(@group.load_balancers, INSERVICE_POLLING_TIME)
+              puts "#{time_taken} seconds have elapsed while waiting for all instances to be InService for a minimum of #{@opts[:min_inservice_time]} seconds."
+
+              time_taken += INSERVICE_POLLING_TIME
+              sleep INSERVICE_POLLING_TIME
+            end
+
+            puts "\nThe new instance was found to be healthy; one old instance will now be removed from the load balancers."
+            deregister_instance instance.ec2_instance, @group.load_balancers
+          end
+        rescue Timeout::Error => e
+          puts "\nDuring the last #{time_taken} seconds, a new AutoScaling instance failed to become healthy."
+          puts "The following settings were changed and will not be changed back by this script:\n"
+
+          puts "AutoScaling processes #{PROCESSES_TO_SUSPEND} were suspended."
+          puts "The desired capacity was changed from #{@group.desired_capacity - 1} to #{@group.desired_capacity}."
+
+          if @max_size_change > 0
+            puts "The maximum size was changed from #{@group.max_size - @max_size_change} to #{@group.max_size}"
+          end
+
+          raise
+        end
+
+        puts "Sleeping for the ELB Timeout period of #{@opts[:elb_timeout]}"
+        sleep @opts[:elb_timeout]
+
+        puts "\nInstance #{instance.id} will now be terminated. By terminating this instance, the actual capacity will be decreased to 1 under desired-capacity."
+        instance.terminate false
+      end
+
+      puts "\n#{@group.name} had its desired-capacity increased temporarily by 1 to a desired-capacity of #{@group.desired_capacity}."
+      puts "The desired-capacity of #{@group.name} will now be returned to its original desired-capacity of #{@group.desired_capacity - 1}."
+      @group.update(desired_capacity: @group.desired_capacity - 1)
+
+      if @max_size_change > 0
+        puts "\n#{@group.name} had its max_size increased temporarily by #{@max_size_change} to a max_size of #{@group.max_size}."
+        puts "The max_size of #{@group.name} will now be returned to its original max_size of #{@group.max_size - @max_size_change}."
+
+        @group.update(max_size: @group.max_size - @max_size_change)
+        @max_size_change = 0
+      end
+
+      @group.resume_all_processes
+    end
+
+    def deregister_instance(instance, load_balancers)
+      load_balancers.each do |load_balancer|
+        load_balancer.instances.deregister instance
+      end
+    end
+
+    def instances_inservice?(load_balancer)
+      return false if load_balancer.instances.count != @group.desired_capacity
+
+      load_balancer.instances.health.each do |instance_health|
+        unless instance_health[:state] == 'InService'
+          puts "\nInstance #{instance_health[:instance].id} is currently #{instance_health[:state]} on load balancer #{load_balancer.name}."
+
+          return false
+        end
+      end
+
+      true
+    end
+
+    def all_instances_inservice?(load_balancers)
+      load_balancers.each do |load_balancer|
+        return false unless instances_inservice?(load_balancer)
+      end
+
+      true
+    end
+
+    def all_instances_inservice_for_time_period?(load_balancers, change_in_time)
+      if all_instances_inservice?(load_balancers)
+        if @time_spent_inservice >= @opts[:min_inservice_time]
+          return true
+        else
+          puts "\nAll instances have been InService for #{@time_spent_inservice} seconds."
+
+          @time_spent_inservice += change_in_time
+          return false
+        end
+      else
+        @time_spent_inservice = 0
+        return false
+      end
+    end
+  end
+end

lib/aws-missing-tools/version.rb

@@ -0,0 +1,3 @@
+module AwsMissingTools
+  VERSION = '0.0.1'
+end

spec/aws-missing-tools/aws-ha-release/aws-ha-release_spec.rb

@@ -0,0 +1,267 @@
+require 'spec_helper'
+
+describe 'aws-ha-release' do
+  let(:opts) { %w(-a test_group -o testaccesskey -s testsecretkey -r test_region -i 1 -t 0 -m 5) }
+
+  let(:as) { AWS::FakeAutoScaling.new }
+
+  let(:instance_one) { AWS::FakeAutoScaling::Instance.new(@group) }
+
+  let(:instance_two) { AWS::FakeAutoScaling::Instance.new(@group) }
+
+  before do
+    AWS::AutoScaling.stub(:new).and_return(as)
+    IO.any_instance.stub(:puts)
+  end
+
+  describe '#initialize' do
+    it 'initializes the AWS connection' do
+      as.groups.create opts[1]
+
+      AWS.should_receive(:config).with(access_key_id: 'testaccesskey', secret_access_key: 'testsecretkey', region: 'test_region')
+      AwsMissingTools::AwsHaRelease.new(opts)
+    end
+
+    it 'ensures the as group exists' do
+      lambda {
+        opts[1] = 'fake_group'
+        AwsMissingTools::AwsHaRelease.new(opts)
+      }.should raise_error
+    end
+  end
+
+  describe '#parse_options' do
+    it 'requires the autoscaling group name to be passed in' do
+      expect{ AwsMissingTools::AwsHaRelease.parse_options([]) }.to raise_error OptionParser::MissingArgument
+      expect(AwsMissingTools::AwsHaRelease.parse_options(%w(-a test_group))[:as_group_name]).to eq 'test_group'
+      expect(AwsMissingTools::AwsHaRelease.parse_options(%w(--as-group-name test_group))[:as_group_name]).to eq 'test_group'
+    end
+
+    it 'sets default options' do
+      options = AwsMissingTools::AwsHaRelease.parse_options(%w(-a test_group))
+      expect(options[:elb_timeout]).not_to be_nil
+      expect(options[:region]).not_to be_nil
+      expect(options[:inservice_time_allowed]).not_to be_nil
+      expect(options[:aws_access_key]).not_to be_nil
+      expect(options[:aws_secret_key]).not_to be_nil
+      expect(options[:min_inservice_time]).not_to be_nil
+    end
+
+    context 'optional params' do
+      it 'ELB timeout' do
+        [%w(-a test_group -t 10), %w(-a test_group --elb-timeout 10)].each do |options|
+          expect(AwsMissingTools::AwsHaRelease.parse_options(options)[:elb_timeout]).to eq 10
+        end
+      end
+
+      it 'region' do
+        [%w(-a test_group -r test_region), %w(-a test_group --region test_region)].each do |options|
+          expect(AwsMissingTools::AwsHaRelease.parse_options(options)[:region]).to eq 'test_region'
+        end
+      end
+
+      it 'inservice time allowed' do
+        [%w(-a test_group -i 300), %w(-a test_group --inservice-time-allowed 300)].each do |options|
+          expect(AwsMissingTools::AwsHaRelease.parse_options(options)[:inservice_time_allowed]).to eq 300
+        end
+      end
+
+      it 'aws_access_key and aws_secret_key' do
+        expect{ AwsMissingTools::AwsHaRelease.parse_options(%w(-a test_group -o testkey)) }.to raise_error OptionParser::MissingArgument
+        expect{ AwsMissingTools::AwsHaRelease.parse_options(%w(-a test_group -s testsecretkey)) }.to raise_error OptionParser::MissingArgument
+
+        options = AwsMissingTools::AwsHaRelease.parse_options(%w(-a test_group -o testkey -s testsecretkey))
+        expect(options[:aws_access_key]).to eq 'testkey'
+        expect(options[:aws_secret_key]).to eq 'testsecretkey'
+      end
+
+      it 'minimum inservice time' do
+        [%w(-a test_group -m 30), %w(-a test_group --min-inservice-time 30)].each do |options|
+          expect(AwsMissingTools::AwsHaRelease.parse_options(options)[:min_inservice_time]).to eq 30
+        end
+      end
+    end
+  end
+
+  describe '#execute!' do
+    before do
+      @group = as.groups.create opts[1]
+      @aws_ha_release = AwsMissingTools::AwsHaRelease.new(opts)
+      @aws_ha_release.stub!(:all_instances_inservice_for_time_period?).and_return(true)
+    end
+
+    it 'suspends certain autoscaling processes' do
+      AWS::FakeAutoScaling::Group.any_instance.should_receive(:suspend_processes)
+          .with(%w(ReplaceUnhealthy AlarmNotification ScheduledActions AZRebalance))
+      @aws_ha_release.execute!
+    end
+
+    it 'requires certain autoscaling processes to not be suspended' do
+      @aws_ha_release.group.suspend_processes %w(RemoveFromLoadBalancerLowPriority Terminate Launch HealthCheck AddToLoadBalancer)
+      expect{ @aws_ha_release.execute! }.to raise_error
+    end
+
+    it 'adjusts the max size as well as the desired capacity if the desired capacity is equal to it' do
+      @group.update(max_size: 1, desired_capacity: 1)
+
+      @aws_ha_release.group.should_receive(:update).with(max_size: 2).ordered.and_call_original
+      @aws_ha_release.group.should_receive(:update).with(desired_capacity: 2).ordered.and_call_original
+      @aws_ha_release.group.should_receive(:update).with(desired_capacity: 1).ordered.and_call_original
+      @aws_ha_release.group.should_receive(:update).with(max_size: 1).ordered.and_call_original
+      @aws_ha_release.execute!
+    end
+
+    it 'only adjusts the desired capacity if max size does not equal desired capacity' do
+      @aws_ha_release.group.should_receive(:update).with(desired_capacity: 2).ordered.and_call_original
+      @aws_ha_release.group.should_receive(:update).with(desired_capacity: 1).ordered.and_call_original
+      @aws_ha_release.execute!
+    end
+  end
+
+  describe 'determining if instances are in service' do
+    before do
+      @group = as.groups.create opts[1]
+      @group.update(desired_capacity: 2)
+      @aws_ha_release = AwsMissingTools::AwsHaRelease.new(opts)
+    end
+
+    it 'checks all instances across a given load balancer' do
+      load_balancer = AWS::FakeELB::LoadBalancer.new 'test_load_balancer_01', [
+        {
+          instance: instance_one,
+          healthy: true
+        },
+        {
+          instance: instance_two,
+          healthy: false
+        }
+      ]
+
+      expect(@aws_ha_release.instances_inservice?(load_balancer)).to eq false
+
+      load_balancer.instances.make_instance_healthy(instance_two)
+      expect(@aws_ha_release.instances_inservice?(load_balancer)).to eq true
+    end
+
+    it 'checks all instances across an array of load balancers' do
+      load_balancers = [
+        AWS::FakeELB::LoadBalancer.new('test_load_balancer_01', [
+          {
+            instance: instance_one,
+            healthy: true
+          },
+          {
+            instance: instance_two,
+            healthy: false
+          }
+        ]), AWS::FakeELB::LoadBalancer.new('test_load_balancer_02', [
+          {
+            instance: instance_one,
+            healthy: true
+          },
+          {
+            instance: instance_two,
+            healthy: false
+          }
+        ])
+      ]
+
+      expect(@aws_ha_release.all_instances_inservice?(load_balancers)).to eq false
+
+      load_balancers[0].instances.make_instance_healthy(instance_two)
+      expect(@aws_ha_release.all_instances_inservice?(load_balancers)).to eq false
+
+      load_balancers[1].instances.make_instance_healthy(instance_two)
+      expect(@aws_ha_release.all_instances_inservice?(load_balancers)).to eq true
+    end
+
+    it 'requires the number of inservice instances to match the desired capacity' do
+      load_balancer = AWS::FakeELB::LoadBalancer.new 'test_load_balancer_01', [
+        {
+          instance: instance_one,
+          healthy: true
+        },
+        {
+          instance: instance_two,
+          healthy: true
+        }
+      ]
+
+      @group.update(desired_capacity: 3)
+
+      expect(@aws_ha_release.instances_inservice?(load_balancer)).to eq false
+
+      instance_three = AWS::FakeAutoScaling::Instance.new(@group)
+      load_balancer.instances.register instance_three
+      load_balancer.instances.make_instance_healthy(instance_three)
+
+      expect(@aws_ha_release.instances_inservice?(load_balancer)).to eq true
+    end
+
+    # ELB health checks seems to be reporting the EC2 health status for a short period of time before switching to the
+    # ELB check. This is a false positive and, until Amazon implements a fix, we must work around it
+    # see https://forums.aws.amazon.com/message.jspa?messageID=455646
+    it 'ensures that an instance has been in service for a period of time before considering it healthy' do
+      load_balancers = [
+        AWS::FakeELB::LoadBalancer.new('test_load_balancer_01', [
+          {
+            instance: instance_one,
+            healthy: true
+          },
+          {
+            instance: instance_two,
+            healthy: false
+          }
+        ])
+      ]
+
+      expect(@aws_ha_release.all_instances_inservice_for_time_period?(load_balancers, 5)).to eq false
+
+      load_balancers[0].instances.make_instance_healthy instance_two
+      expect(@aws_ha_release.all_instances_inservice_for_time_period?(load_balancers, 5)).to eq false
+
+      expect(@aws_ha_release.all_instances_inservice_for_time_period?(load_balancers, 5)).to eq true
+    end
+  end
+
+  describe '#deregister_instance' do
+    before do
+      @group = as.groups.create opts[1]
+      @aws_ha_release = AwsMissingTools::AwsHaRelease.new(opts)
+    end
+
+    it 'deregisters an instance across all load balancers' do
+      elb_one = AWS::FakeELB::LoadBalancer.new 'test_load_balancer_01', [
+        {
+          instance: instance_one,
+          healthy: true
+        },
+        {
+          instance: instance_two,
+          healthy: true
+        }
+      ]
+      elb_two = AWS::FakeELB::LoadBalancer.new 'test_load_balancer_02', [
+        {
+          instance: instance_one,
+          healthy: true
+        },
+        {
+          instance: instance_two,
+          healthy: true
+        }
+      ]
+
+      elb_one.instances.register instance_one
+      elb_one.instances.register instance_two
+
+      elb_two.instances.register instance_one
+      elb_two.instances.register instance_two
+
+      @aws_ha_release.deregister_instance instance_one, [elb_one, elb_two]
+
+      expect(elb_one.instances).not_to include instance_one
+      expect(elb_two.instances).not_to include instance_one
+    end
+  end
+end

spec/spec_helper.rb

@@ -0,0 +1,12 @@
+require 'rspec'
+require 'aws-missing-tools'
+
+Dir['spec/support/**/*.rb'].each { |f| require File.expand_path(f) }
+
+RSpec.configure do |config|
+  config.order = 'random'
+
+  config.before do
+    AWS.stub!
+  end
+end

spec/support/fake_auto_scaling.rb

@@ -0,0 +1,81 @@
+module AWS
+  class FakeAutoScaling
+    def initialize
+    end
+
+    def groups
+      @groups ||= GroupCollection.new
+    end
+
+    class GroupCollection
+      def initialize
+        @groups = {}
+      end
+
+      def [](name)
+        @groups[name]
+      end
+
+      def create(name, options = {})
+        @groups[name] = Group.new name
+      end
+    end
+
+    class Group
+      attr_reader :name, :max_size, :desired_capacity, :suspended_processes
+
+      def initialize(name)
+        @name = name
+        @suspended_processes = {}
+        @max_size = 2
+        @desired_capacity = 1
+      end
+
+      def suspend_processes(processes)
+        processes.each do |process|
+          @suspended_processes[process] = 'test'
+        end
+      end
+
+      def resume_all_processes
+        @suspended_processes.clear
+      end
+
+      def update(options = {})
+        options.each do |key, value|
+          self.instance_variable_set "@#{key}", value
+        end
+      end
+
+      def auto_scaling_instances
+        @auto_scaling_instances ||= [AWS::FakeAutoScaling::Instance.new(self), AWS::FakeAutoScaling::Instance.new(self)]
+      end
+
+      def load_balancers
+        @load_balancers ||= AWS::FakeELB::LoadBalancerCollection.new
+      end
+
+      def ec2_instances
+        auto_scaling_instances.map(&:ec2_instance)
+      end
+    end
+
+    class Instance
+      def initialize(group)
+        @group = group
+      end
+
+      def terminate(decrement_desired_capacity)
+        @group.update(desired_capacity: @group.desired_capacity - 1) if decrement_desired_capacity
+      end
+
+      def id
+        'i-test'
+      end
+
+      def ec2_instance
+        AWS::FakeEC2::Instance.new(self.id)
+      end
+    end
+  end
+end

spec/support/fake_ec2.rb

@@ -0,0 +1,11 @@
+module AWS
+  class FakeEC2
+    class Instance
+      attr_reader :id
+
+      def initialize(id)
+        @id = id
+      end
+    end
+  end
+end

spec/support/fake_elb.rb

@@ -0,0 +1,79 @@
+module AWS
+  class FakeELB
+    def initialize
+    end
+
+    class LoadBalancer
+      attr_reader :name, :instances
+
+      def initialize(name, instances_and_healths)
+        @name = name
+        @instances ||= InstanceCollection.new(instances_and_healths)
+      end
+    end
+
+    class LoadBalancerCollection < Array
+      def initialize
+      end
+    end
+
+    class InstanceCollection < Array
+      attr_reader :health
+
+      def initialize(instances_and_healths)
+        @health = []
+
+        instances_and_healths.each do |instance_and_health|
+          self << instance_and_health[:instance]
+          instance_and_health[:healthy] ? make_instance_healthy(instance_and_health[:instance]) : make_instance_unhealthy(instance_and_health[:instance])
+        end
+      end
+
+      def register(*instances)
+        self.concat instances
+      end
+
+      def deregister(*instances)
+        instances.each do |i|
+          self.delete i
+        end
+      end
+
+      def make_instance_healthy(instance)
+        opts = {
+          instance: instance,
+          description: 'N/A',
+          state: 'InService',
+          reason_code: 'N/A'
+        }
+
+        @health.each_with_index do |health, i|
+          if health[:instance] == instance
+            @health[i] = opts
+            return
+          end
+        end
+
+        @health << opts
+      end
+
+      def make_instance_unhealthy(instance)
+        opts = {
+          instance: instance,
+          description: 'Instance has failed at least the UnhealthyThreshold number of health checks consecutively.',
+          state: 'OutOfService',
+          reason_code: 'Instance'
+        }
+
+        @health.each_with_index do |health, i|
+          if health[:instance] == instance
+            @health[i] = opts
+            return
+          end
+        end
+
+        @health << opts
+      end
+    end
+  end
+end