Script: Check No Missing Files After Reorganisation of Directory Trees

Here's another script I did when I had to reorganised a folder hierarchy of years of data. Basically to ensure files are not missing, or corrupted.


#!/bin/bash                                                                                                                              

#########################
#                        
# checkNoMissingFiles    
# ===================    
#                        
# This script checks that no files are missing after folders are reorganised.
# Basic algorithm is to checksum all files in both old and new folders, then 
# checking through both lists of checksums to ensure all checksums are present
# in both lists.                                                              
#                                                                             
# Changelog                                                                   
# =========                                                                   
#                                                                             
# 18 Oct 2007 - Junhao                                                        
# * Initial commit                                                            
#                                                                             
# 11 Dec 2007 - Junhao                                                        
# * Tidied style                                                              
# * Fixed bug with spaces in filenames                                        
# * added option to save generated checksums                                  
# * changed md5sum to sha1sum                                                 
# * changed checksum to general algorithm                                     
#########################                                                     

PATH=/bin:/usr/bin
export PATH       

## Program Locations
awk=/bin/awk        
cat=/bin/cat        
echo=/bin/echo      
find=/usr/bin/find  
grep=/bin/grep      
checksum=/usr/bin/sha1sum
mktemp=/bin/mktemp       
rm=/bin/rm               
tee="/usr/bin/tee -a"    
touch=/bin/touch         
## End Program Locations 

## Start Script

## Script parameters
f_logFile=/dev/null 
d_orgLoc=/dev/null  
d_newLoc=/dev/null  
v_oldFileName=      
v_oldFileChksum=    
f_oldChksumLog=     
f_newChksumLog=     
v_missingFilesCount=0
v_missingFiles=""    
v_output=            
v_f1flag=1           
v_f2flag=1           
## End Script parameters

function print_usage () {
    ${echo} "            
$0                       
Usage: $0 [-L logfile] [-f1 filename] [-f2 filename] [oldDir] [newDir]
 or    $0 -h                                                          
Description: Checks that there are no missing files after reorganising a directory.
Options:                                                                           
  -L logfile    (Optional) Path to log file                                        
  -h            (Optional) This help text                                          
  -1            (Optional) Filename to save checksum for old directory             
  -2            (Optional) Filename to save checksum for new directory             
  oldDir        Location of old directory                                          
  newDir        Location of new directory                                          
"                                                                                  
}                                                                                  

if [ $# -lt 2 ]; then
    print_usage      
    exit 1           
else                 
    while getopts hL:1:2: options; do
        case "${options}" in         
            h)  print_usage          
                exit 1               
                ;;                   
            L)  f_logFile=${OPTARG}  
                ;;                   
            1)  f_oldChksumLog=${OPTARG}
                v_f1flag=0              
                ;;                      
            2)  f_newChksumLog=${OPTARG}
                v_f2flag=0              
                ;;                      
            *)  f_logFile=/dev/null     
                ;;                      
        esac                            
    done                                
    shift $((${OPTIND} - 1))            

    if [ -d "$1" ]; then
        d_orgLoc="$1"   
    else                
        ${echo} "Error: Original directory does not exist!"
        print_usage                                        
        exit 1                                             
    fi                                                     
                                                           
    if [ -d "$2" ]; then                                   
        d_newLoc="$2"                                      
    else                                                   
        ${echo} "Error: New directory does not exist!"     
        print_usage                                        
        exit 1                                             
    fi                                                     

    if [ -z ${f_oldChksumLog} ]; then
        f_oldChksumLog=`${mktemp}`   
    elif [ -f ${f_oldChksumLog} ]; then
        ${echo} "Error: File ${f_oldChksumLog} exists! Please give another filename."
        exit 2                                                                       
    else                                                                             
        ${touch} ${f_oldChksumLog}                                                   
        if [ ! -f ${f_oldChksumLog} ]; then                                          
            ${echo} "Error: ${f_oldChksumLog} cannot be created!"                    
            exit 4                                                                   
        fi                                                                           
    fi                                                                               

    if [ -z ${f_newChksumLog} ]; then
        f_newChksumLog=`${mktemp}`   
    elif [ -f ${f_newChksumLog} ]; then
        ${echo} "Error: File ${f_newChksumLog} exists! Please give another filename."
        exit 3                                                                       
    else                                                                             
        ${touch} ${f_newChksumLog}                                                   
        if [ ! -f ${f_newChksumLog} ]; then                                          
            ${echo} "Error: File ${f_newChksumLog} cannot be created!"               
            exit 5                                                                   
        fi                                                                           
    fi                                                                               
fi                                                                                   

${echo} "${find} \"${d_orgLoc}\" -type f -exec ${checksum} \"\\{\\}\" \\;" | ${tee} ${f_logFile}
${find} "${d_orgLoc}" -type f -exec ${checksum} "{}" \; | ${tee} ${f_oldChksumLog}              
${echo} "${find} \"${d_newLoc}\" -type f -exec ${checksum} \"\\{\\}\" \\;" | ${tee} ${f_logFile}
${find} "${d_newLoc}" -type f -exec ${checksum} "{}" \; | ${tee} ${f_newChksumLog}              


while read -r v_oldFileChksum v_oldFileName; do
    if [[ `${grep} ${v_oldFileChksum} ${f_newChksumLog}` ]]; then
        v_output="Okay:  ${v_oldFileName} -> "                   
        v_output="${v_output} `${grep} \"${v_oldFileChksum}\" \"${f_newChksumLog}\" | ${awk} '{print $2}'`"
    else                                                                                                   
        v_output="ERROR: ${v_oldFileName} is missing"                                                      
        v_missingFiles="${v_missingFiles} ${v_oldFileName}"                                                
        v_missingFilesCount=$((v_missingFilesCount+1))                                                     
    fi                                                                                                     
    ${echo} "${v_output}" | ${tee} ${f_logFile}
done < ${f_oldChksumLog}

#### cleanup ####
if [ "1" == ${v_f1flag} ]; then
    ${rm} ${f_oldChksumLog}
fi
if [ "1" == ${v_f2flag} ]; then
    ${rm} ${f_newChksumLog}
fi


if [ ${v_missingFilesCount} -gt 0 ]; then
    ${echo} "ERROR: ${v_missingFilesCount} files are missing:" | ${tee} ${f_logFile}
    ${echo} "ERROR:   ${v_missingFiles}" | ${tee} ${f_logFile}
    exit 99
else
    ${echo} "Success: ${v_missingFilesCount} files are missing" | ${tee} ${f_logFile}
    exit 0
fi

Comments

I am going to try it, so if I have any question, can I ask you?

Yeah. Smiling

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <pre> <img> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • Textual smileys will be replaced with graphical ones.

More information about formatting options

Smileys

Smileys
SmileyAcronyms
=)=) :) :-) :smile:
;);) ;-) :wink:
=(=( :( :-( :sad:
=D=D :D :-D :lol:
}:)}:) }:-) :evil:
=p=p =P :P :-P :tongue:
O.oO.o :O :-O :shocked:
:?:? :-? :puzzled:
8)8) 8-) :cool:
:jawdrop::jawdrop:
:sick::sick: :barf: