scripts
Script: Yum Check Update
Submitted by jmarki on 8 October 2009 - 2:50amI have been using this script to check for updates on my Redhat systems for quite some time. Put this into your cron.daily, and you have a daily nag to update your system. 
#!/bin/bash
#########
## Yum Check Update Script
##
## This script checks for system updates and sends email
## to sysmin team if there are any updates.
##
## Changelog
## ---------
## 24 Oct 2008 (Junhao)
## - Initial commit
##
#########
_CAT="/bin/cat"
_DATE="/bin/date"
_HOSTNAME="/bin/hostname"
_MAILX="/bin/mailx"
_RM="/bin/rm"
_TOUCH="/bin/touch"
_YUM="/usr/bin/yum"
HOSTNAME=`${_HOSTNAME}`
DATESTAMP=`${_DATE} +%Y%b%d-%H:%M:%S`
EMAIL=root
MAILSUB="RHEL Update Available for ${HOSTNAME} on ${DATESTAMP}"
TEMPLOG=/tmp/yum-check-update.tmp
${_TOUCH} ${TEMPLOG}
${_YUM} check-update 1> ${TEMPLOG} 2>&1
if [[ $? != 0 ]]; then
${_CAT} ${TEMPLOG} | ${_MAILX} -s "${MAILSUB}" ${EMAIL}
fi
${_RM} ${TEMPLOG}
- jmarki's blog
- Add new comment
- 217 reads
Script: Check No Missing Files After Reorganisation of Directory Trees
Submitted by jmarki on 1 August 2009 - 4:43pmHere's another script I did when I had to reorganised a folder hierarchy of years of data. Basically to ensure files are not missing, or corrupted.
#!/bin/bash
#########################
#
# checkNoMissingFiles
# ===================
#
# This script checks that no files are missing after folders are reorganised.
# Basic algorithm is to checksum all files in both old and new folders, then
# checking through both lists of checksums to ensure all checksums are present
# in both lists.
#
# Changelog
# =========
#
# 18 Oct 2007 - Junhao
# * Initial commit
#
# 11 Dec 2007 - Junhao
# * Tidied style
# * Fixed bug with spaces in filenames
# * added option to save generated checksums
# * changed md5sum to sha1sum
# * changed checksum to general algorithm
#########################
PATH=/bin:/usr/bin
export PATH
## Program Locations
awk=/bin/awk
cat=/bin/cat
echo=/bin/echo
find=/usr/bin/find
grep=/bin/grep
checksum=/usr/bin/sha1sum
mktemp=/bin/mktemp
rm=/bin/rm
tee="/usr/bin/tee -a"
touch=/bin/touch
## End Program Locations
## Start Script
## Script parameters
f_logFile=/dev/null
d_orgLoc=/dev/null
d_newLoc=/dev/null
v_oldFileName=
v_oldFileChksum=
f_oldChksumLog=
f_newChksumLog=
v_missingFilesCount=0
v_missingFiles=""
v_output=
v_f1flag=1
v_f2flag=1
## End Script parameters
function print_usage () {
${echo} "
$0
Usage: $0 [-L logfile] [-f1 filename] [-f2 filename] [oldDir] [newDir]
or $0 -h
Description: Checks that there are no missing files after reorganising a directory.
Options:
-L logfile (Optional) Path to log file
-h (Optional) This help text
-1 (Optional) Filename to save checksum for old directory
-2 (Optional) Filename to save checksum for new directory
oldDir Location of old directory
newDir Location of new directory
"
}
if [ $# -lt 2 ]; then
print_usage
exit 1
else
while getopts hL:1:2: options; do
case "${options}" in
h) print_usage
exit 1
;;
L) f_logFile=${OPTARG}
;;
1) f_oldChksumLog=${OPTARG}
v_f1flag=0
;;
2) f_newChksumLog=${OPTARG}
v_f2flag=0
;;
*) f_logFile=/dev/null
;;
esac
done
shift $((${OPTIND} - 1))
if [ -d "$1" ]; then
d_orgLoc="$1"
else
${echo} "Error: Original directory does not exist!"
print_usage
exit 1
fi
if [ -d "$2" ]; then
d_newLoc="$2"
else
${echo} "Error: New directory does not exist!"
print_usage
exit 1
fi
if [ -z ${f_oldChksumLog} ]; then
f_oldChksumLog=`${mktemp}`
elif [ -f ${f_oldChksumLog} ]; then
${echo} "Error: File ${f_oldChksumLog} exists! Please give another filename."
exit 2
else
${touch} ${f_oldChksumLog}
if [ ! -f ${f_oldChksumLog} ]; then
${echo} "Error: ${f_oldChksumLog} cannot be created!"
exit 4
fi
fi
if [ -z ${f_newChksumLog} ]; then
f_newChksumLog=`${mktemp}`
elif [ -f ${f_newChksumLog} ]; then
${echo} "Error: File ${f_newChksumLog} exists! Please give another filename."
exit 3
else
${touch} ${f_newChksumLog}
if [ ! -f ${f_newChksumLog} ]; then
${echo} "Error: File ${f_newChksumLog} cannot be created!"
exit 5
fi
fi
fi
${echo} "${find} \"${d_orgLoc}\" -type f -exec ${checksum} \"\\{\\}\" \\;" | ${tee} ${f_logFile}
${find} "${d_orgLoc}" -type f -exec ${checksum} "{}" \; | ${tee} ${f_oldChksumLog}
${echo} "${find} \"${d_newLoc}\" -type f -exec ${checksum} \"\\{\\}\" \\;" | ${tee} ${f_logFile}
${find} "${d_newLoc}" -type f -exec ${checksum} "{}" \; | ${tee} ${f_newChksumLog}
while read -r v_oldFileChksum v_oldFileName; do
if [[ `${grep} ${v_oldFileChksum} ${f_newChksumLog}` ]]; then
v_output="Okay: ${v_oldFileName} -> "
v_output="${v_output} `${grep} \"${v_oldFileChksum}\" \"${f_newChksumLog}\" | ${awk} '{print $2}'`"
else
v_output="ERROR: ${v_oldFileName} is missing"
v_missingFiles="${v_missingFiles} ${v_oldFileName}"
v_missingFilesCount=$((v_missingFilesCount+1))
fi
${echo} "${v_output}" | ${tee} ${f_logFile}
done < ${f_oldChksumLog}
#### cleanup ####
if [ "1" == ${v_f1flag} ]; then
${rm} ${f_oldChksumLog}
fi
if [ "1" == ${v_f2flag} ]; then
${rm} ${f_newChksumLog}
fi
if [ ${v_missingFilesCount} -gt 0 ]; then
${echo} "ERROR: ${v_missingFilesCount} files are missing:" | ${tee} ${f_logFile}
${echo} "ERROR: ${v_missingFiles}" | ${tee} ${f_logFile}
exit 99
else
${echo} "Success: ${v_missingFilesCount} files are missing" | ${tee} ${f_logFile}
exit 0
fi
- 2 comments
- 207 reads
Script: Zimbra Backup Script
Submitted by jmarki on 1 August 2009 - 4:29pmI am currently migrating out of Zimbra to a 3rd-party host. Just for archival, here's my Zimbra backup script. Just run this script using a cronjob every day. There will be a short downtime where Zimbra is shutdown to synchronise the last bit of emails, but that should be okay if you have a backup MX server.
This script creates a working live copy of the Zimbra directory, then shutdown Zimbra to sync the directory. The directory is then passed through star, into a small(er) file.
#!/bin/bash
########
## Zimbra backup script
##
## See
##
## Requires star, rsync, bash, gzip
## Does full backups of /opt/zimbra only. Tries to
## minimise zimbra shutdown time with a live rsync,
## then a offline rsync
##
## Changelog
## ---------
## 15 Feb 2009 (Junhao)
## - BUGFIX: deletes leftover archive.tgz tarball before
## creating new tar
## 21 Oct 2008 (Junhao)
## - added disk size to log
## - added tarball -t test
## 19 Oct 2008 (Junhao)
## - Initial commit
##
########
## config
_AWK=`which awk`
_CAT=`which cat`
_CD=cd #bash builtin
_CHECKSUM=`which sha1sum`
_DATE=`which date`
_DF=`which df`
_DU=`which du`
_ECHO=`which echo`
_MV=`which mv`
_MAIL=`which mail`
_RSYNC=`which rsync`
_RM=`which rm`
_SLEEP=`which sleep`
_SU=`which su`
_TAR=`which star`
_TOUCH=`which touch`
DATESTAMP=`date +%Y%b%d-%T`
RELEASE=`${_SU} - zimbra -c"zmcontrol -v"`
RELEASE=`${_ECHO} ${RELEASE} | ${_AWK} '{ print $2"-"$3"-"$4 }'`
TARBALL=zimbra-${RELEASE}-backup-full-${DATESTAMP}.tgz
LOGFILE=zimbra-${RELEASE}-backup-full-${DATESTAMP}.log
EMAIL=user@domain.co.m
MAILSUB="ZCS Backup Report on ${DATESTAMP}"
BKUPRETRIES=5
RESTARTRETRIES=100
ZIMBRADIR=/opt/zimbra
BASEDIR=/opt/zimbra-backups
WORKDIR=${BASEDIR}/working
LOGDIR=${BASEDIR}/logs
SAVEDIR=${BASEDIR}/saved
LOG=${LOGDIR}/${LOGFILE}
CHKSUMLOG=${SAVEDIR}/checksum
TARLOG=${WORKDIR}/tar.log
LOCK=${BASEDIR}/zimbra-backup.lock
TEMPARCHIVE=archive.tgz
function calc_downtime() {
if [ -z "${STARTTIME3}" ]; then
STARTTIME3=`date +%s`
fi
if [ -z "${ENDTIME}" ]; then
ENDTIME=`date +%s`
fi
TOTAL=$((${ENDTIME} - ${STARTTIME1}))
OFFLINE=$((${STARTTIME3} - ${STARTTIME2}))
${_ECHO} "Time taken: $((${TOTAL} / 3600)) hours $((${TOTAL} % 3600 / 60)) minutes $((${TOTAL} % 3600 % 60)) seconds" >> ${LOG}
${_ECHO} "Zimbra Offline: $((${OFFLINE} / 3600)) hours $((${OFFLINE} % 3600 / 60)) minutes $((${OFFLINE} % 3600 % 60)) seconds" >> ${LOG}
return 0
}
function force_restart() {
for (( i=0; i<=${RESTARTRETRIES} ; i=$(($i+1)) )); do
${_SU} - zimbra -c"zmcontrol stop"
${_SLEEP} 10
${_SU} - zimbra -c"zmcontrol start"
${_SLEEP} 10
${_SU} - zimbra -c"zmcontrol status"
if [[ $? == 0 ]]; then
${_ECHO} "Sucessfully restarted zimbra after $((${i}+1)) tries" >> ${LOG}
return 0
else
${_SLEEP} 10
fi
done
${_ECHO} "Could not restart zimbra after ${RESTARTRETRIES} tries" >> ${LOG}
return 254
}
function lock_set() {
${_TOUCH} ${LOCK}
}
function lock_remove() {
${_RM} ${LOCK}
}
function synchronise() {
for (( i=0; i<=${BKUPRETRIES} ; i=$(($i+1)) )); do
${_RSYNC} -avHK --delete --exclude=*.pid ${ZIMBRADIR} ${WORKDIR}
if [[ $? == 0 || $? == 24 ]]; then
return $?
fi
${_SLEEP} 10
done
return 254
}
function send_mail() {
calc_downtime
${_ECHO} "" >> ${LOG}
${_SU} - zimbra -c"zmcontrol status" >> ${LOG}
${_ECHO} "" >> ${LOG}
${_CAT} ${TARLOG} >> ${LOG}
${_RM} ${TARLOG}
${_CAT} ${LOG} | ${_MAIL} -s "${MAILSUB}" ${EMAIL}
}
function send_error() {
MAILSUB="[Failed] ${MAILSUB}"
send_mail
}
function send_success() {
MAILSUB="[Success] ${MAILSUB}"
send_mail
}
${_TOUCH} ${LOG}
lock_set
${_ECHO} "Server: `${_SU} - zimbra -c"zmhostname"`" >>${LOG}
${_ECHO} "Tarball: ${TARBALL}" >> ${LOG}
${_ECHO} "Logfile: ${LOG}" >> ${LOG}
${_ECHO} "Backup started at ${DATESTAMP}" >> ${LOG}
${_ECHO} "" >> ${LOG}
## Outputs time backup started for logging
STARTTIME1=`${_DATE} +%s`
## Online sync to working directory
${_ECHO} "Starting online rsync" >> ${LOG}
synchronise
if [[ $? == 254 ]]; then
${_ECHO} "Error in creating live copy" >> ${LOG}
send_error
lock_remove
exit 1
fi
STARTTIME2=`${_DATE} +%s`
## Shut down zimbra
${_SU} - zimbra -c"zmcontrol stop"
if [[ $? != 0 ]]; then
${_ECHO} "Error stopping zimbra" >> ${LOG}
${_SU} - zimbra -c"zmcontrol status" >>${LOG}
${_ECHO} "Aborting backup, force restarting zimbra" >> ${LOG}
force_restart
if [[ $? == 255 ]]; then
${_ECHO} "Trying again" >> ${LOG}
force_restart
fi
send_error
lock_remove
exit 2
fi
${_SLEEP} 10
## Offline sync to working directory
${_ECHO} "Starting offline rsync" >>${LOG}
synchronise
if [[ $? == 0 ]]; then
${_ECHO} "Offline rsync completed successfully" >> ${LOG}
elif [[ $? == 24 ]]; then
## some files disappeared, meaning some open process running
## wait a while, rerun rsync, and assume okay
${_SLEEP} 60
synchronise
else
${_ECHO} "Error in creating offline copy" >> ${LOG}
${_ECHO} "Aborting backup, force restarting zimbra" >> ${LOG}
force_restart
send_error
lock_remove
exit 3
fi
## Start zimbra
force_restart
if [[ $? == 254 ]]; then
${_ECHO} "Trying again" >> ${LOG}
force_restart
send_error
lock_remove
exit 4
fi
STARTTIME3=`date +%s`
## Synchronization sucessful, create archive
${_CD} ${WORKDIR}
if [ -f ${TARLOG} ]; then
${_RM} ${TARLOG}
else
${_TOUCH} ${TARLOG}
fi
if [ -f ${TEMPARCHIVE} ]; then
${_RM} ${TEMPARCHIVE}
fi
${_TAR} czf ${TEMPARCHIVE} ./ 1>> ${TARLOG} 2>&1
if [[ $? != 0 && $? != 254 ]]; then
${_ECHO} "Error creating tarball. Error Code: $?" >> ${LOG}
${_ECHO} "Aborting backup" >> ${LOG}
${_ECHO} "Tarball location: ${WORKDIR}/${TEMPARCHIVE}" >> ${LOG}
send_error
lock_remove
exit 5
fi
${_TAR} ztf ${TEMPARCHIVE}
if [[ $? == 0 ]]; then
${_MV} ${TEMPARCHIVE} ${SAVEDIR}/${TARBALL}
${_ECHO} "" >> ${LOG}
${_DU} -sh ${SAVEDIR}/${TARBALL} >> ${LOG}
${_DF} -h ${SAVEDIR}/${TARBALL} >> ${LOG}
else
${_ECHO} "Error validating tarball. Error Code: $?" >> ${LOG}
${_ECHO} "Aborting backup" >> ${LOG}
${_ECHO} "Tarball location: ${WORKDIR}/${TEMPARCHIVE}" >> ${LOG}
send_error
lock_remove
exit 6
fi
## Creating checksum
${_CD} ${SAVEDIR}
CHKSUM=`${_CHECKSUM} ${TARBALL}`
${_ECHO} "${CHKSUM}" >> ${CHKSUMLOG}
${_ECHO} "" >> ${LOG}
${_ECHO} "Checksum using ${_CHECKSUM}">> ${LOG}
${_ECHO} "${CHKSUM}" >> ${LOG}
## Backup done!
ENDTIME=`date +%s`
send_success
lock_remove
exit 0
- Add new comment
- 188 reads
Puppet - Centralised Configuration Management
Submitted by jmarki on 28 April 2009 - 8:21amRecently, I have started to migrate my scripts to use Puppet. Everything from initial system provisioning to manual failover systems had been converted. Wee~
The idea behind Puppet is to consolidate and standardise configuration across multiple servers. By centralising configuration, a standard security and provisioning baseline is maintained. Configuration for each service can be standardised and reused across an entire infrastructure. Even better, puppet ensures the system remains as configured. Locally configured files are reverted, services are restarted, etc. The end result? Less headache and easier knowledge sharing.
Someone once commented about me using a "commandline webmin". I don't think Puppet is like webmin at all. Webmin pre-defines the fields for configuration. Puppet is, well, blank. It simply provides an API for defining my systems, and then helps me push/maintain it across the infrastructure.
Who says system administrators can't code? 
Okay, back to coding...
- jmarki's blog
- Add new comment
- 318 reads
Bash Scripting Tips
Submitted by jmarki on 12 January 2008 - 12:43amI went looking around for bash scripting tips, especially secure coding of bash. Can't find much information, so decided to consolidate whatever I found here.
- Salt string comparisons of variables to increase security
if [[ "a$?" == "a4" ]]; then
- Use the full paths to any binaries, either by hardcoding them into the script or use variable substitution. This prevents the script from executing incorrect/rogue binaries in the path.
/bin/grep "hardcoding the full path" * echo=/bin/echo ${echo} "From bash manpage under EXPANSION: The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion." - Change the environment path at the start of the script to ensure no rouge directories are in the PATH
#!/bin/bash # comments PATH=/bin:/usr/bin
- Write a function to explain the usage of the script
function print_usage () { ${echo} " $0 Usage: $0 [-a opts] [arguments] or $0 -h Description: Something fishy Options: -a opts (Optional) Options -h (Optional) Help arguments Smelly smelly fish " } - Here's a sample code snippet to process script options
if [ $# -lt 2 ]; then print_usage exit 1 else while getopts ha:b: options; do case "${options}" in h) print_usage exit 1 ;; a) flag=${options} ;; b) flag=${options} ;; *) echo "default case, everything else fits here" ;; esac done shift $((${OPTIND} - 1)) - Variables should be enclosed in parenthesis when used, to indicate exactly which variable you are using. Of course, this can prevent an exploit involving longer variable names.
a=erie ab=were if [[ "${a}b" == "erieb" ]]; then
- Add new comment
- 785 reads
Script: check for missing files in a directory after reorganisation
Submitted by jmarki on 11 December 2007 - 7:50pm*Updated: 11 Dec 2007
I'm wondering where I should store the scripts I'm writing. Out of pure laziness, I'll just dump them as my blog entry for now.
Here's a script to check for missing files after a directory has been re-organised. Basically, it compares the md5sum of the files in the old directory and the new directory.
Please let me know if there are any bugs.
#!/bin/bash
#########################
#
# checkNoMissingFiles
# ===================
#
# This script checks that no files are missing after folders are reorganised.
# Basic algorithm is to checksum all files in both old and new folders, then
# checking through both lists of checksums to ensure all checksums are present
# in both lists.
#
# Changelog
# =========
#
# 18 Oct 2007 - Junhao
# * Initial commit
#
# 11 Dec 2007 - Junhao
# * Tidied style
# * Fixed bug with spaces in filenames
# * added option to save generated checksums
# * changed md5sum to sha1sum
# * changed checksum to general algorithm
#########################
PATH=/bin:/usr/bin;
## Program Locations
awk=/usr/bin/awk
cat=/usr/bin/cat
echo=/usr/bin/echo
find=/usr/bin/find
grep=/bin/grep
checksum="/usr/bin/sha1sum"
mktemp=/bin/mktemp
rm=/usr/bin/rm
tee="/usr/bin/tee -a"
touch="/bin/touch"
## End Program Locations
## Start Script
## Script parameters
f_logFile=/dev/null
d_orgLoc=/dev/null
d_newLoc=/dev/null
v_oldFileName=
v_oldFileChksum=
f_oldChksumLog=
f_newChksumLog=
v_missingFilesCount=0
v_missingFiles=""
v_output=
v_f1flag=1
v_f2flag=1
## End Script parameters
function print_usage () {
${echo} "
$0
Usage: $0 [-L logfile] [-f1 filename] [-f2 filename] [oldDir] [newDir]
or $0 -h
Description: Checks that there are no missing files after reorganising a directory.
Options:
-L logfile (Optional) Path to log file
-h (Optional) This help text
-1 (Optional) Filename to save checksum for old directory
-2 (OPtional) Filename to save checksum for new directory
oldDir Location of old directory
newDir Location of new directory
"
}
if [ $# -lt 2 ]; then
print_usage
exit 1
else
while getopts hL:1:2: options; do
case "${options}" in
h) print_usage
exit 1
;;
L) f_logFile=${OPTARG}
;;
1) f_oldChksumLog=${OPTARG}
v_f1flag=0
;;
2) f_newChksumLog=${OPTARG}
v_f2flag=0
;;
*) f_logFile=/dev/null
;;
esac
done
shift $((${OPTIND} - 1))
if [ -d $1 ]; then
d_orgLoc=$1
else
${echo} "Error: Original directory does not exist!"
print_usage
exit 1
fi
if [ -d $2 ]; then
d_newLoc=$2
else
${echo} "Error: New directory does not exist!"
print_usage
exit 1
fi
if [ -z ${f_oldChksumLog} ]; then
f_oldChksumLog=${mktemp}
elif [ -f ${f_oldChksumLog} ]; then
${echo} "Error: File ${f_oldChksumLog} exists! Please give another filename."
exit 2
else
${touch} ${f_oldChksumLog}
if [ ! -f ${f_oldChksumLog} ]; then
${echo} "Error: ${f_oldChksumLog} cannot be created!"
exit 4
fi
fi
if [ -z ${f_newChksumLog} ]; then
f_oldChksumLog=${mktemp}
elif [ -f ${f_newChksumLog} ]; then
${echo} "Error: File ${f_newChksumLog} exists! Please give another filename."
exit 3
else
${touch} ${f_newChksumLog}
if [ ! -f ${f_newChksumLog} ]; then
${echo} "Error: File ${f_newChksumLog} cannot be created!"
exit 5
fi
fi
fi
${echo} "${find} \"${d_orgLoc}\" -type f -exec ${checksum} \\"\{\}\\" \;" | ${tee} ${f_logFile}
${find} "${d_orgLoc}" -type f -exec ${checksum} \"\{\}\" \; | ${tee} ${f_oldChksumLog}
${find} "${find} \"${d_newLoc}\" -type f -exec ${checksum} \\"\{\}\\" \;" | ${tee} ${f_logFile}
${find} "${d_newLoc}" -type f -exec ${checksum} \"\{\}\" \; | ${tee} ${f_newChksumLog}
while read -r v_oldFileChksum v_oldFileName; do
if [[ `${grep} ${v_oldFileChksum} ${f_newChksumLog}` ]]; then
v_output="Okay: ${v_oldFileName} -> "
v_output="${v_output} `${grep} \"${v_oldFileChksum}\" \"${f_newChksumLog}\" | ${awk} '{print $2}'`"
else
v_output="ERROR: ${v_oldFileName} is missing"
v_missingFiles="${v_missingFiles} ${v_oldFileName}"
v_missingFilesCount=$((v_missingFilesCount+1))
fi
${echo} "${v_output}" | ${tee} ${f_logFile}
done < ${f_oldChksumLog}
#### cleanup ####
if [ "1" == ${v_f1flag} ]; then
${rm} ${f_oldChksumLot}
fi
if [ "1" == ${v_f2flag} ]; then
${rm} ${f_newChksumLog}
fi
if [ ${v_missingFilesCount} -gt 0 ]; then
${echo} "ERROR: ${v_missingFilesCount} files are missing:" | ${tee} ${f_logFile}
${echo} "ERROR: ${v_missingFiles}" | ${tee} ${f_logFile}
exit 99
else
${echo} "Success: ${v_missingFilesCount} files are missing" | ${tee} ${f_logFile}
exit 0
fi
- Add new comment
- 562 reads
Code Repository
Submitted by jmarki on 19 October 2007 - 8:45pmI often have to code many many scripts for my daily work as a system administrator. In the (vain) hopes these might be useful to someone else, maybe I should release these into the public domain.
My style of coding hasn't really stablised; still trying to find a style that allows secure coding and easy readability. If you have suggestions, please let me know.
Of course, if there a bugs, please let me know. Thanks!
- Add new comment
- 926 reads
