Portfolio: File Insurance

By

Download All

file-insurance.tar.gz

Individual Files

README

View raw: README

About File Insurance
--------------------
These scripts were written in response to a serious bug in a piece of software I
once used extensively. The program would occasionally crash while saving. The
crashes would irrecoverably corrupt the files being saved, resulting in total
data loss. Bug reports to the developer weren't addressed. At that time, I
didn't have a continuous backup system, so after losing important data one time
too many, I whipped up these scripts.

file-insurance monitors a PID and one or more files. Every 5 minutes until the
PID terminates, it makes a copy of the files it is monitoring and timestamps
them. If the copy is identical to the most recent one, it is discarded. Call it
with --help for usage instructions.

file-insurance-cleanup is intended to be run manually once it is known that all
is well with the data. It deletes the backups created by file-insurance except
the most recent one. Call it with --help for usage instructions.

start-program-with-file-insurance launches the program specified on the command
line, passing it all filename arguments given. Then, it starts file-insurance
and arranges for it to monitor those files.


Known Bug
---------
The file-insurance scripts frequently parse the output of ls. However, ls
was designed for human readability and not for parsing. It is impossible to
write code to parse the output of ls that works for all valid filenames. For
practical purposes, though, these scripts' use of ls is OK; it isn't common to
encounter filenames containing newlines or other "unusual" characters. These
scripts can cope with spaces in filenames. Other "weird" characters might
cause these scripts to blow up.

This bug isn't deemed worth fixing at this time. Filenames that would break
the scripts are uncommon. The proper fix would be to replace ls with shell
globs. Even then, that wouldn't solve all issues. A complete solution would
most likely involve a rewrite in another language, and such a rewrite simply
isn't worth the time it would take.

file-insurance

View raw: file-insurance

#!/bin/bash
# -*- coding: utf-8 -*-
# 
# Copyright by Scott Severance
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

# Auto-backup script
#
# Ensures that too much data doesn't get lost if $program
#   crashes during a save.

prog="$(basename $0)"
timeout="300" # 5 minutes
timeoutStep="5" #seconds
pid=
watchFiles=

# Color codes for an easier-to-read output
FILE_COLOR="\033[4;34m"
ERROR_COLOR="\033[0;31m"
MSG_COLOR="\033[0;32m"
NO_COLOR="\033[0m" #Transparent - don't change
TITLEBAR_START="\033]0;"
TITLEBAR_END="\007"

E_USAGE=65
E_NORMAL=0
E_BAD_ARGUMENT=66

function usage {
    echo -e "${MSG_COLOR}Usage:"
    echo -e "$prog program_pid filename [filename [...]]${NO_COLOR}"
    echo
    echo "$prog defends against files being ruined by misbehaving programs. To"
    echo "accomplish this, $prog makes a compressed copy of each file listed on"
    echo "the command line every $timeout seconds until the program with the specified pid"
    echo "terminates."
    echo
    echo "$prog keeps its backups in ~/.file-insurance/\$programName/\$filename."
    [ -z "$1" ] && exit $E_USAGE
}

function programAlive {
    local pid="$1"
    ps --pid "$pid" > /dev/null 2>&1
    return $?
}

function getNewestFile {
    ls -1t "$1" | while read "crud"; do
                      if [ -n "$crud" ]; then
                          echo "$crud"
                          break
                      fi
                  done
}

function makeBackup {
    local programName="$HOME/.file-insurance/$1"
    local filename="$2"
    
    echo -ne "$ERROR_COLOR"
    
    if [ ! -f "$filename" ]; then
        echo -e "Error: the file $filename doesn't exist. Aborting...${NO_COLOR}"
        exit "$E_BAD_ARGUMENT"
    fi
    
    [ -d "$programName" ] || mkdir -p "$programName"
    local dir="$programName/$filename"
    [ -d "$dir" ] || mkdir -p "$dir"
    
    local currentTime="$(date +'%Y-%m-%d_%H-%M-%S')"
    local backupFile="$dir/file-insurance.$currentTime"
    local lastBackupFile=
    if [ "$(ls -1 "$dir" | wc | awk '{print $1}')" != "0" ]; then
        globalTmp="$(getNewestFile "$dir")"
        lastBackupFile="${dir}/${globalTmp}"
    fi
    
    cp "$filename" "$backupFile"
    bzip2 "$backupFile"
    backupFile="${backupFile}.bz2"
    
    if [ -n "$lastBackupFile" ]; then # no sense in making multiple copies of identical files
        if $(diff "$lastBackupFile" "$backupFile" >/dev/null); then # diff exits 0 if there are no differences
            rm -f "$backupFile"
        fi
    fi
    
    echo -ne "$NO_COLOR"
}

function sleepCounter { # args: sleep time, time step, callback function, [callback function args...]
     local remaining="$1"; shift
     local step="$1";      shift
     local callback="$1";  shift
     
     while [[ $remaining > 0 ]]; do
         $callback "$@"
         (( remaining -= step ))
         sleep $step
     done
}

function exitIfProgramDead {
    if ! $(programAlive "$pid"); then
        echo -e "${MSG_COLOR}$pidName ($pid) is no longer running. Exiting.${NO_COLOR}"
        echo -e "\n\nConsider running file-insurance-cleanup to purge unnecessary files."
        exit "$E_NORMAL"
    fi
}

function initThis {
    if [ "z$1" = "z" -o "z$2" = "z" ]; then
        usage
    else
        pid="$1"; shift
        watchFiles=("$@")
    fi
    echo -ne "$ERROR_COLOR"
    if ! $(programAlive "$pid"); then
        echo "Error: invalid pid"
        echo
        usage "false"
        exit "$E_BAD_ARGUMENT"
    fi
    if [ -z "${watchFiles[0]}" ]; then
        echo "Error: no file to watch"
        echo
        usage false
        exit "$E_BAD_ARGUMENT"
    fi
    echo -ne "$NO_COLOR"
}

function main {
    [ "$1" = "--help" ] && usage
    
    initThis "$@"
    shift
    
    local pidName="$(ps --pid "$pid" --no-headers | awk '{print $4}' | sed 's/\//_/g')"
    
    echo -ne "${MSG_COLOR}Monitoring $pidName ($pid) and the file(s)"
    for i in "$@"; do
        echo -en " \"${FILE_COLOR}${i}${MSG_COLOR}\""
    done
    echo -e "...${NO_COLOR}"
    
    echo -ne "${TITLEBAR_START}file-insurance monitoring ${pidName} (${pid})${TITLEBAR_END}"
    
    while : ; do
        for i in "${watchFiles[@]}"; do
            makeBackup "$pidName" "$i"
        done
        
        sleepCounter "$timeout" "$timeoutStep" "exitIfProgramDead"
    done
}

main "$@"

file-insurance-cleanup

View raw: file-insurance-cleanup

#!/bin/bash
# -*- coding: utf-8 -*-
# 
# Copyright by Scott Severance
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

# file-insurance-cleanup
# Deletes all but the most recent backup files created by file-insurance.

prog="$(basename $0)"
backupDir="$HOME/.file-insurance"
escDir="$(echo "$backupDir" | sed 's/\//\\\//g')\/"
counter="0"
quiet=

FILE_COLOR="\033[4;34m"
ERROR_COLOR="\033[0;31m"
MSG_COLOR="\033[0;32m"
NO_COLOR="\033[0m" #Transparent - don't change
TITLEBAR_START="\033]0;"
TITLEBAR_END="\007"

E_USAGE=65
E_NORMAL=0
E_CANCEL=66

function cleanDir {
    local dir="$1"
    local oldPwd="$(pwd)"
    local bakFile=
    if [[ "$dir" =~ "$backupDir" ]]; then
        local fullDir="$dir"
    else
        local fullDir="$oldPwd/$dir"
    fi
    counter="0"
    
    builtin cd "$dir" 2> /dev/null
    IFS="\n" ls -1 --reverse | while read bakFile; do
                                   cleanFiles "$bakFile"
                               done
    builtin cd "$oldPwd" 2> /dev/null
    rmdir "$dir" 2> /dev/null
}

function cleanFiles {
    local bakFile="$1"
    
    if [ -d "$bakFile" ]; then
        cleanDir "$bakFile"
        return
    fi
    
    local fullFile="$(pwd)/$bakFile"
    local abbrFile="$(echo "$fullFile" | sed "s/${escDir}//g")"
    
    if [[ ! "$bakFile" =~ '^file-insurance\.[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}-[0-9]{2}-[0-9]{2}\.bz2$' ]]; then
        echo -e "${ERROR_COLOR}WARNING: Skipping file with unexpected name: ${FILE_COLOR}$abbrFile${NO_COLOR}" 1>&2
        return
    fi
    if [ "$counter" = "0" ]; then
        (( counter++ ))
        [ -z "$quiet" ] && echo -e "${MSG_COLOR}Preserving ${FILE_COLOR}$abbrFile${MSG_COLOR}...${NO_COLOR}"
    else
        [ -z "$quiet" ] && echo -e "Removing ${FILE_COLOR}$abbrFile${NO_COLOR}..."
        rm "$bakFile"
        (( counter++ ))
    fi
}

function usage {
    echo -e "${MSG_COLOR}Usage:"
    echo -e "$prog [--quiet|--help]${NO_COLOR}"
    echo -e "\n$prog cleans up after file-insurance by deleting all backup"
    echo -e "files it created except the most recent.\n"
    echo "With the --quiet option, $prog is silent except for errors."
}

function main {
    local i=
    
    case "$1" in
        "--help")
            usage
            exit $E_USAGE
            ;;
        "--quiet")
            quiet="true"
            FILE_COLOR=
            ERROR_COLOR=
            ;;
    esac
    
    if [ -z "$quiet" ]; then
        backupDirAbbr="$(echo "$backupDir" | sed "s/^$(echo "$HOME" | sed 's/[\/[{?.+*]/\\&/g')/~/g")"
        echo -n "Do you want to prune file-insurance's backup directory ($backupDirAbbr)? [y/N] "
        read answer
        if [[ ! "$answer" =~ '[Yy]' ]]; then
            echo "Exiting..."
            exit $E_CANCEL
        fi
    fi
    
    cleanDir "$backupDir"
    
    exit $E_NORMAL 
}

main "$@"

start-program-with-file-insurance

View raw: start-program-with-file-insurance

#! /bin/bash
# -*- coding: utf-8 -*-
# 
# Copyright by Scott Severance
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

"$@" &
pid=$!
echo "$@"
shift
echo "file-insurance $pid $@"
file-insurance "$pid" "$@"