Merge branch 'master' of https://github.com/colinbjohnson/aws-missing-tools
This commit is contained in:
Colin Johnson
commit
6bd5066efd
|
|
@ -93,7 +93,7 @@ fi
|
|||
|
||||
|
||||
#gets Auto Scaling Group max-size
|
||||
asg_initial_max_size=`echo $asg_result | grep ^AUTO-SCALING-GROUP | cut -d "$delimiter" -f 10`
|
||||
asg_initial_max_size=`echo $asg_result | grep ^AUTO-SCALING-GROUP | cut -d "$delimiter" -f 9`
|
||||
asg_temporary_max_size=$(($asg_initial_max_size+1))
|
||||
#gets Auto Scaling Group desired-capacity
|
||||
asg_initial_desired_capacity=`echo "$asg_result" | grep ^AUTO-SCALING-GROUP | cut -d "$delimiter" -f 10`
|
||||
|
|
|
|||
|
|
@ -22,12 +22,14 @@ the above example would provide a single backup of the EBS volumeid vol-6d6a0527
|
|||
ec2-automate-backup requires one of the following two parameters be provided:
|
||||
|
||||
`-v <volumeid>` - the "volumeid" parameter is required to select EBS volumes for snapshot if ec2-automate-backup is run using the "volumeid" selection method - the "volumeid" selection method is the default selection method.
|
||||
|
||||
|
||||
`-t <tag>` - the "tag" parameter is required if the "method" of selecting EBS volumes for snapshot is by tag (-s tag). The format for tag is key=value (example: Backup=true) and the correct method for running ec2-automate-backup in this manner is ec2-automate-backup -s tag -t Backup=true.
|
||||
|
||||
`-d <destination region(s)>` - the "destination region" parameter is required if the "method" of selecting EBS snapshots for copy to another region is by region (-s regioncopy). The format for `-d` is space-delimited region names such as ec2-automate-backup.sh -s regioncopy -d "us-west-1 us-west-2". This selection method only operates on snapshots.
|
||||
## Optional Parameters:
|
||||
`-r <region>` - the region that contains the EBS volumes for which you wish to have a snapshot created.
|
||||
|
||||
`-s <selection_method>` - the selection method by which EBS volumes will be selected. Currently supported selection methods are "volumeid" and "tag." The selection method "volumeid" identifies EBS volumes for which a snapshot should be taken by volume id whereas the selection method "tag" identifies EBS volumes for which a snapshot should be taken by a key=value format tag.
|
||||
`-s <selection_method>` - the selection method by which EBS volumes will be selected. Currently supported selection methods are "volumeid", "tag" and "regioncopy." The selection method "volumeid" identifies EBS volumes for which a snapshot should be taken by volume id whereas the selection method "tag" identifies EBS volumes for which a snapshot should be taken by a key=value format tag. The selection method "regioncopy" identifies regions as destinations for EBS volumes that have been scheduled for copy to that region.
|
||||
|
||||
`-c <cron_primer_file>` - running with the -c option and a providing a file will cause ec2-automate-backup to source a file for environmental configuration - ideal for running ec2-automate-backup under cron. An example cron primer file is located in the "Resources" directory and is called cron-primer.sh.
|
||||
|
||||
|
|
@ -36,14 +38,18 @@ ec2-automate-backup requires one of the following two parameters be provided:
|
|||
`-k <purge_after_period>` - the period after which a snapshot can be purged. For example, running "ec2-automate-backup.sh -v "vol-6d6a0527 vol-636a0112" -k 31" would allow snapshots to be removed after 31 days. purge_after_period creates two tags for each volume that was backed up - a PurgeAllow tag which is set to PurgeAllow=true and a PurgeAfterFE tag which is set to the number of minutes from epoch after which a snapshot can be purged. Values can also be entered as -k 60m (this would purge snapshots after 60 minutes or -k 2h (this would purge snapshots after 2 hours) or -k 7d (this would purge snapshots after 7 days). With no trailing characters the purge after period defaults to days.
|
||||
|
||||
`-p` - the -p flag will purge (meaning delete) all snapshots that where the current date is passed the volumes "PurgeAfterFE" tag. ec2-automate-backup looks at two tags to determine which snapshots should be deleted - the PurgeAllow and PurgeAfterFE tags. The tags must be set as follows: PurgeAllow=true and PurgeAfterFE=xxxxxxxxxx where xxxxxxxxxx is a UNIX time that is before the current date.
|
||||
|
||||
`-g <scheduled destination regions>` - Space-delimited destination regions for the volumes selected. This is added when the `-v` or `-t` selection method is used. Note that this only adds a tag that schedules the snapshot to be copied to the destinations. On subsequent calls, the `-d` parameter will intiate copies to the specified regions.
|
||||
# Potential Uses and Methods of Use:
|
||||
* To backup multiple EBS volumes use ec2-automate-backup as follows: `ec2-automate-backup -v "vol-6d6a0527 vol-636a0112"`
|
||||
* To backup multiple EBS volumes and schedule them for copy to two other regions: `ec2-automate-backup -v "vol-6d6a0527 vol-636a0112" -g "us-west-1 us-west-2"`
|
||||
* To backup a selected group of EBS volumes on a daily schedule tag each volume you wish to backup with the tag "Backup=true" and run ec2-automate-backup using cron as follows: `0 0 * * 0 ec2-automate-backup -s tag -t "Backup=true"`
|
||||
* To backup a selected group of EBS volumes on a daily and/or monthly schedule tag each volume you wish to backup with the tag "Backup-Daily=true" and/or "Backup-Monthly=true" and run ec2-automate-backup using cron as follows:
|
||||
- `0 0 * * 0 ec2-user /home/ec2-user/ec2-automate-backup.sh -s tag -t "Backup-Daily=true"`
|
||||
- `0 0 1 * * ec2-user /home/ec2-user/ec2-automate-backup.sh -s tag -t "Backup-Monthly=true"`
|
||||
* To perform daily backup using cron and to load environment configuration with a "cron-primer" file:
|
||||
- `0 0 * * 0 ec2-user /home/ec2-user/ec2-automate-backup.sh -c /home/ec2-user/cron-primer.sh -s tag -t "Backup=True"`
|
||||
* To initiate the copy of scheduled snapshots to their destination regions: `ec2-automate-backup.sh -s regioncopy -d "us-west-1 us-west-2"`
|
||||
|
||||
`-u` - the -u flag will tag snapshots with additional data so that snapshots can be more easily located. Currently the two user tags created are Volume="ebs_volume" and Created="date." These can be easily modified in code.
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,14 @@
|
|||
#confirms that executables required for succesful script execution are available
|
||||
prerequisite_check()
|
||||
{
|
||||
for prerequisite in basename ec2-create-snapshot ec2-create-tags ec2-describe-snapshots ec2-delete-snapshot date
|
||||
check_values=$1
|
||||
|
||||
if [[ -z $check_values ]]
|
||||
then
|
||||
check_values='ec2-create-snapshot ec2-create-tags ec2-describe-snapshots ec2-delete-snapshot date'
|
||||
fi
|
||||
|
||||
for prerequisite in basename $check_values
|
||||
do
|
||||
#use of "hash" chosen as it is a shell builtin and will add programs to hash table, possibly speeding execution. Use of type also considered - open to suggestions.
|
||||
hash $prerequisite &> /dev/null
|
||||
|
|
@ -27,27 +34,38 @@ get_EBS_List()
|
|||
fi
|
||||
ebs_selection_string="$volumeid"
|
||||
;;
|
||||
tag)
|
||||
tag)
|
||||
if [[ -z $tag ]]
|
||||
then echo "The selected selection_method \"tag\" (-s tag) requires a valid tag (-t key=value) for operation. Correct usage is as follows: \"-s tag -t backup=true\" or \"-s tag -t Name=my_tag.\"" 1>&2 ; exit 64
|
||||
fi
|
||||
ebs_selection_string="--filter tag:$tag"
|
||||
;;
|
||||
regioncopy)
|
||||
if [[ -z $region_copy_destinations ]]
|
||||
then echo "The selected selection_method \"regioncopy\" (-s regioncopy) requires valid region names (-d '[region name(s)]') or 'all' for operation. Correct usage is as follows: \"-s regioncopy -d 'us-west-1'\" or \"-s regioncopy -d 'us-west-1 us-west-2'.\"" 1>&2 ; exit 64
|
||||
fi
|
||||
;;
|
||||
*) echo "If you specify a selection_method (-s selection_method) for selecting EBS volumes you must select either \"volumeid\" (-s volumeid) or \"tag\" (-s tag)." 1>&2 ; exit 64 ;;
|
||||
esac
|
||||
#creates a list of all ebs volumes that match the selection string from above
|
||||
ebs_backup_list_complete=`ec2-describe-volumes --show-empty-fields --region $region $ebs_selection_string 2>&1`
|
||||
#takes the output of the previous command
|
||||
ebs_backup_list_result=`echo $?`
|
||||
if [[ $ebs_backup_list_result -gt 0 ]]
|
||||
then echo -e "An error occured when running ec2-describe-volumes. The error returned is below:\n$ebs_backup_list_complete" 1>&2 ; exit 70
|
||||
|
||||
if [[ -n $ebs_selection_string ]]
|
||||
then
|
||||
#creates a list of all ebs volumes that match the selection string from above
|
||||
ebs_backup_list_complete=`ec2-describe-volumes --show-empty-fields --region $region $ebs_selection_string 2>&1`
|
||||
#takes the output of the previous command
|
||||
ebs_backup_list_result=`echo $?`
|
||||
if [[ $ebs_backup_list_result -gt 0 ]]
|
||||
then echo -e "An error occured when running ec2-describe-volumes. The error returned is below:\n$ebs_backup_list_complete" 1>&2 ; exit 70
|
||||
fi
|
||||
ebs_backup_list=`echo "$ebs_backup_list_complete" | grep ^VOLUME | cut -f 2`
|
||||
#code to right will output list of EBS volumes to be backed up: echo -e "Now outputting ebs_backup_list:\n$ebs_backup_list"
|
||||
fi
|
||||
ebs_backup_list=`echo "$ebs_backup_list_complete" | grep ^VOLUME | cut -f 2`
|
||||
#code to right will output list of EBS volumes to be backed up: echo -e "Now outputting ebs_backup_list:\n$ebs_backup_list"
|
||||
}
|
||||
|
||||
create_EBS_Snapshot_Tags()
|
||||
{
|
||||
echo "create_EBS_Snapshot_Tags region_copy_scheduled_destinations=$region_copy_scheduled_destinations"
|
||||
|
||||
#snapshot tags holds all tags that need to be applied to a given snapshot - by aggregating tags we ensure that ec2-create-tags is called only onece
|
||||
snapshot_tags=""
|
||||
#if $name_tag_create is true then append ec2ab_${ebs_selected}_$current_date to the variable $snapshot_tags
|
||||
|
|
@ -68,6 +86,15 @@ create_EBS_Snapshot_Tags()
|
|||
snapshot_tags="$snapshot_tags --tag Volume=${ebs_selected} --tag Created=$current_date"
|
||||
fi
|
||||
|
||||
#if $name_tag_create is true then append RegionCopy=[region]:scheduled,[region]:scheduled to the variable $snapshot_tags
|
||||
if [[ -n $region_copy_scheduled_destinations ]]
|
||||
then
|
||||
ec2_snapshot_resource_id=`echo "$ec2_create_snapshot_result" | cut -f 2`
|
||||
regions=$(echo $region_copy_scheduled_destinations | sed 's/ */:scheduled,/g')
|
||||
snapshot_tags="$snapshot_tags --tag RegionCopy=${regions}:scheduled"
|
||||
echo "tagging $ec2_snapshot_resource_id with $snapshot_tags"
|
||||
fi
|
||||
|
||||
#if $snapshot_tags is not zero length then set the tag on the snapshot using ec2-create-tags
|
||||
if [[ -n $snapshot_tags ]]
|
||||
then echo "Tagging Snapshot $ec2_snapshot_resource_id with the following Tags:"
|
||||
|
|
@ -113,7 +140,7 @@ purge_EBS_Snapshots()
|
|||
snapshot_tag_list=`ec2-describe-tags --show-empty-fields --region $region --filter resource-type=snapshot --filter key=PurgeAllow,PurgeAfterFE`
|
||||
#snapshot_purge_allowed is a list of all snapshot_ids with PurgeAllow=true
|
||||
snapshot_purge_allowed=`echo "$snapshot_tag_list" | grep .*PurgeAllow'\s'true | cut -f 3`
|
||||
|
||||
|
||||
for snapshot_id_evaluated in $snapshot_purge_allowed
|
||||
do
|
||||
#gets the "PurgeAfterFE" date which is in UTC with UNIX Time format (or xxxxxxxxxx / %s)
|
||||
|
|
@ -133,6 +160,50 @@ purge_EBS_Snapshots()
|
|||
done
|
||||
}
|
||||
|
||||
region_copy_EBS_Snapshots()
|
||||
{
|
||||
#snapshots_to_region_copy is a list of all snapshot_ids with RegionCopy=.*:scheduled.*
|
||||
# tr and $'n' substitution used so $IFS doesn't have to be
|
||||
# manipulated with for loops and the cut command
|
||||
snapshot_tag_list=$(ec2-describe-tags --show-empty-fields --region $region --filter resource-type=snapshot --filter key=RegionCopy | grep ':scheduled' | cut -f3,5- | tr '[\t ]' '~')
|
||||
|
||||
for snapshot_data in $(echo ${snapshot_tag_list//$'\n'/ } | cut -f1-)
|
||||
do
|
||||
snapshot_id=$(echo $snapshot_data | cut -d'~' -f1)
|
||||
regions=$(echo $snapshot_data | cut -d'~' -f2-)
|
||||
original_regions=$regions
|
||||
region_copy_tag=''
|
||||
for region_data in ${regions//,/ }
|
||||
do
|
||||
destination_region=$(echo $region_data | cut -d: -f1)
|
||||
status=$(echo $region_data | cut -d: -f2-)
|
||||
region_copy_tag_append="$destination_region:$status"
|
||||
if [[ "$region_copy_destinations" = "all" || -n $(echo "$region_copy_destinations"| grep $destination_region) ]]
|
||||
then
|
||||
current_datetime=$(date +%Y-%m-%d_%H:%M:%S)
|
||||
if [ "$status" = "scheduled" ]
|
||||
then
|
||||
ec2_copy_snapshot_complete=$(ec2-copy-snapshot -r $region -s $snapshot_id --region $destination_region)
|
||||
ec2_copy_snapshot_result=`echo $?`
|
||||
if [[ $ec2_copy_snapshot_result -gt 0 ]]
|
||||
then
|
||||
echo -e "An error occured when running ec2-copy-snapshot. The error returned is below:\n$ec2_copy_snapshot_complete" 1>&2 ; exit 70
|
||||
else
|
||||
region_copy_tag_append="$destination_region:$current_datetime"
|
||||
echo "The snapshot \"$snapshot_id\" is being copied to $destination_region."
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
region_copy_tag="${region_copy_tag},$region_copy_tag_append"
|
||||
done
|
||||
|
||||
if [[ "${region_copy_tag#,}" != "$original_regions" ]]
|
||||
then
|
||||
ec2-create-tags $snapshot_id --region $region --tag RegionCopy=${region_copy_tag#,}
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
app_name=`basename $0`
|
||||
|
||||
#sets defaults
|
||||
|
|
@ -148,7 +219,7 @@ user_tags=false
|
|||
#sets the Purge Snapshot feature to false - this feature will eventually allow the removal of snapshots that have a "PurgeAfterFE" tag that is earlier than current date
|
||||
purge_snapshots=false
|
||||
#handles options processing
|
||||
while getopts :s:c:r:v:t:k:pnu opt
|
||||
while getopts :s:c:r:v:t:k:g:d:l:pnu opt
|
||||
do
|
||||
case $opt in
|
||||
s) selection_method="$OPTARG";;
|
||||
|
|
@ -157,6 +228,8 @@ while getopts :s:c:r:v:t:k:pnu opt
|
|||
v) volumeid="$OPTARG";;
|
||||
t) tag="$OPTARG";;
|
||||
k) purge_after_input="$OPTARG";;
|
||||
g) region_copy_scheduled_destinations="$OPTARG";;
|
||||
d) region_copy_destinations="$OPTARG";;
|
||||
n) name_tag_create=true;;
|
||||
p) purge_snapshots=true;;
|
||||
u) user_tags=true;;
|
||||
|
|
@ -166,7 +239,7 @@ while getopts :s:c:r:v:t:k:pnu opt
|
|||
|
||||
#sources "cron_primer" file for running under cron or other restricted environments - this file should contain the variables and environment configuration required for ec2-automate-backup to run correctly
|
||||
if [[ -n $cron_primer ]]
|
||||
then if [[ -f $cron_primer ]]
|
||||
then if [[ -f $cron_primer ]]
|
||||
then source $cron_primer
|
||||
else
|
||||
echo "Cron Primer File \"$cron_primer\" Could Not Be Found." 1>&2 ; exit 70
|
||||
|
|
@ -177,7 +250,7 @@ fi
|
|||
if [[ -z $region ]]
|
||||
#if the environment variable $EC2_REGION is not set set to us-east-1
|
||||
then if [[ -z $EC2_REGION ]]
|
||||
#if both
|
||||
#if both
|
||||
then region="us-east-1"
|
||||
else
|
||||
region=$EC2_REGION
|
||||
|
|
@ -213,7 +286,7 @@ do
|
|||
then echo -e "An error occured when running ec2-create-snapshot. The error returned is below:\n$ec2_create_snapshot_result" 1>&2 ; exit 70
|
||||
else
|
||||
ec2_snapshot_resource_id=`echo "$ec2_create_snapshot_result" | cut -f 2`
|
||||
fi
|
||||
fi
|
||||
create_EBS_Snapshot_Tags
|
||||
done
|
||||
|
||||
|
|
@ -221,4 +294,12 @@ done
|
|||
if $purge_snapshots
|
||||
then echo "Snapshot Purging is Starting Now."
|
||||
purge_EBS_Snapshots
|
||||
fi
|
||||
fi
|
||||
|
||||
#if region_copy_destinations is true, then run region_copy_EBS_Snapshots function
|
||||
if [[ -n $region_copy_destinations ]]
|
||||
then echo "Snapshot Copying to regions $region_copy_destinations is Starting Now."
|
||||
prerequisite_check ec2-copy-snapshot
|
||||
region_copy_EBS_Snapshots
|
||||
fi
|
||||
|
||||
|
|
|
|||
|
|
@ -6,7 +6,6 @@
|
|||
"ec2:CreateSnapshot",
|
||||
"ec2:DeleteSnapshot",
|
||||
"ec2:CreateTags",
|
||||
"ec2:Deletenapshot",
|
||||
"ec2:DescribeTags"
|
||||
],
|
||||
"Effect": "Allow",
|
||||
|
|
@ -15,4 +14,4 @@
|
|||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
#!/bin/bash -
|
||||
#get instance id - used for putting metric
|
||||
INSTANCE_ID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`
|
||||
INSTANCE_AZ=`curl -s curl http://169.254.169.254/latest/meta-data/placement/availability-zone/`
|
||||
INSTANCE_AZ=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone/`
|
||||
INSTANCE_REGION=${INSTANCE_AZ%?}
|
||||
|
||||
#could be done using "free" or "vmstat" - use of less and grep is believed to provide widest compatibility - CJ 2011-11-24
|
||||
|
|
@ -15,4 +15,4 @@ mon-put-data --region $INSTANCE_REGION --metric-name MemoryFree --namespace EC2/
|
|||
mon-put-data --region $INSTANCE_REGION --metric-name SwapUsed --namespace EC2/Memory --value $swapused --unit Kilobytes --dimensions "InstanceId=$INSTANCE_ID"
|
||||
|
||||
#to run in cron every 5 minutes - note that you must first provide credentials for mon-put-data
|
||||
#echo "*/5 * * * * ec2-user /usr/local/bin/ec2-write-memory-metrics.sh" > /etc/cron.d/ec2-write-memory-metrics
|
||||
#echo "*/5 * * * * ec2-user /usr/local/bin/ec2-write-memory-metrics.sh" > /etc/cron.d/ec2-write-memory-metrics
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
#!/bin/bash -
|
||||
#get instance id - used for putting metric
|
||||
INSTANCE_ID=`GET http://169.254.169.254/latest/meta-data/instance-id`
|
||||
INSTANCE_AZ=`curl -s curl http://169.254.169.254/latest/meta-data/placement/availability-zone/`
|
||||
INSTANCE_AZ=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone/`
|
||||
INSTANCE_REGION=${INSTANCE_AZ%?}
|
||||
|
||||
#belowshould be changed to grep - get only everything after % space slash
|
||||
|
|
@ -14,4 +14,4 @@ for filesystemmountpoint in $filesystemlist
|
|||
done
|
||||
|
||||
#to run in cron every 5 minutes - note that you must first provide credentials for mon-put-data
|
||||
#echo "*/5 * * * * ec2-user /usr/local/bin/ec2-write-storage-used.sh" > /etc/cron.d/ec2-write-storage-used
|
||||
#echo "*/5 * * * * ec2-user /usr/local/bin/ec2-write-storage-used.sh" > /etc/cron.d/ec2-write-storage-used
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user