Portfolio: File Insurance
Download All
file-insurance.tar.gzIndividual Files
README
View raw: README
About File Insurance
--------------------
These scripts were written in response to a serious bug in a piece of software I
once used extensively. The program would occasionally crash while saving. The
crashes would irrecoverably corrupt the files being saved, resulting in total
data loss. Bug reports to the developer weren't addressed. At that time, I
didn't have a continuous backup system, so after losing important data one time
too many, I whipped up these scripts.
file-insurance monitors a PID and one or more files. Every 5 minutes until the
PID terminates, it makes a copy of the files it is monitoring and timestamps
them. If the copy is identical to the most recent one, it is discarded. Call it
with --help for usage instructions.
file-insurance-cleanup is intended to be run manually once it is known that all
is well with the data. It deletes the backups created by file-insurance except
the most recent one. Call it with --help for usage instructions.
start-program-with-file-insurance launches the program specified on the command
line, passing it all filename arguments given. Then, it starts file-insurance
and arranges for it to monitor those files.
Known Bug
---------
The file-insurance scripts frequently parse the output of ls. However, ls
was designed for human readability and not for parsing. It is impossible to
write code to parse the output of ls that works for all valid filenames. For
practical purposes, though, these scripts' use of ls is OK; it isn't common to
encounter filenames containing newlines or other "unusual" characters. These
scripts can cope with spaces in filenames. Other "weird" characters might
cause these scripts to blow up.
This bug isn't deemed worth fixing at this time. Filenames that would break
the scripts are uncommon. The proper fix would be to replace ls with shell
globs. Even then, that wouldn't solve all issues. A complete solution would
most likely involve a rewrite in another language, and such a rewrite simply
isn't worth the time it would take.file-insurance
View raw: file-insurance
#!/bin/bash
# -*- coding: utf-8 -*-
#
# Copyright by Scott Severance
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
# Auto-backup script
#
# Ensures that too much data doesn't get lost if $program
# crashes during a save.
prog="$(basename $0)"
timeout="300" # 5 minutes
timeoutStep="5" #seconds
pid=
watchFiles=
# Color codes for an easier-to-read output
FILE_COLOR="\033[4;34m"
ERROR_COLOR="\033[0;31m"
MSG_COLOR="\033[0;32m"
NO_COLOR="\033[0m" #Transparent - don't change
TITLEBAR_START="\033]0;"
TITLEBAR_END="\007"
E_USAGE=65
E_NORMAL=0
E_BAD_ARGUMENT=66
function usage {
echo -e "${MSG_COLOR}Usage:"
echo -e "$prog program_pid filename [filename [...]]${NO_COLOR}"
echo
echo "$prog defends against files being ruined by misbehaving programs. To"
echo "accomplish this, $prog makes a compressed copy of each file listed on"
echo "the command line every $timeout seconds until the program with the specified pid"
echo "terminates."
echo
echo "$prog keeps its backups in ~/.file-insurance/\$programName/\$filename."
[ -z "$1" ] && exit $E_USAGE
}
function programAlive {
local pid="$1"
ps --pid "$pid" > /dev/null 2>&1
return $?
}
function getNewestFile {
ls -1t "$1" | while read "crud"; do
if [ -n "$crud" ]; then
echo "$crud"
break
fi
done
}
function makeBackup {
local programName="$HOME/.file-insurance/$1"
local filename="$2"
echo -ne "$ERROR_COLOR"
if [ ! -f "$filename" ]; then
echo -e "Error: the file $filename doesn't exist. Aborting...${NO_COLOR}"
exit "$E_BAD_ARGUMENT"
fi
[ -d "$programName" ] || mkdir -p "$programName"
local dir="$programName/$filename"
[ -d "$dir" ] || mkdir -p "$dir"
local currentTime="$(date +'%Y-%m-%d_%H-%M-%S')"
local backupFile="$dir/file-insurance.$currentTime"
local lastBackupFile=
if [ "$(ls -1 "$dir" | wc | awk '{print $1}')" != "0" ]; then
globalTmp="$(getNewestFile "$dir")"
lastBackupFile="${dir}/${globalTmp}"
fi
cp "$filename" "$backupFile"
bzip2 "$backupFile"
backupFile="${backupFile}.bz2"
if [ -n "$lastBackupFile" ]; then # no sense in making multiple copies of identical files
if $(diff "$lastBackupFile" "$backupFile" >/dev/null); then # diff exits 0 if there are no differences
rm -f "$backupFile"
fi
fi
echo -ne "$NO_COLOR"
}
function sleepCounter { # args: sleep time, time step, callback function, [callback function args...]
local remaining="$1"; shift
local step="$1"; shift
local callback="$1"; shift
while [[ $remaining > 0 ]]; do
$callback "$@"
(( remaining -= step ))
sleep $step
done
}
function exitIfProgramDead {
if ! $(programAlive "$pid"); then
echo -e "${MSG_COLOR}$pidName ($pid) is no longer running. Exiting.${NO_COLOR}"
echo -e "\n\nConsider running file-insurance-cleanup to purge unnecessary files."
exit "$E_NORMAL"
fi
}
function initThis {
if [ "z$1" = "z" -o "z$2" = "z" ]; then
usage
else
pid="$1"; shift
watchFiles=("$@")
fi
echo -ne "$ERROR_COLOR"
if ! $(programAlive "$pid"); then
echo "Error: invalid pid"
echo
usage "false"
exit "$E_BAD_ARGUMENT"
fi
if [ -z "${watchFiles[0]}" ]; then
echo "Error: no file to watch"
echo
usage false
exit "$E_BAD_ARGUMENT"
fi
echo -ne "$NO_COLOR"
}
function main {
[ "$1" = "--help" ] && usage
initThis "$@"
shift
local pidName="$(ps --pid "$pid" --no-headers | awk '{print $4}' | sed 's/\//_/g')"
echo -ne "${MSG_COLOR}Monitoring $pidName ($pid) and the file(s)"
for i in "$@"; do
echo -en " \"${FILE_COLOR}${i}${MSG_COLOR}\""
done
echo -e "...${NO_COLOR}"
echo -ne "${TITLEBAR_START}file-insurance monitoring ${pidName} (${pid})${TITLEBAR_END}"
while : ; do
for i in "${watchFiles[@]}"; do
makeBackup "$pidName" "$i"
done
sleepCounter "$timeout" "$timeoutStep" "exitIfProgramDead"
done
}
main "$@"file-insurance-cleanup
View raw: file-insurance-cleanup
#!/bin/bash
# -*- coding: utf-8 -*-
#
# Copyright by Scott Severance
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
# file-insurance-cleanup
# Deletes all but the most recent backup files created by file-insurance.
prog="$(basename $0)"
backupDir="$HOME/.file-insurance"
escDir="$(echo "$backupDir" | sed 's/\//\\\//g')\/"
counter="0"
quiet=
FILE_COLOR="\033[4;34m"
ERROR_COLOR="\033[0;31m"
MSG_COLOR="\033[0;32m"
NO_COLOR="\033[0m" #Transparent - don't change
TITLEBAR_START="\033]0;"
TITLEBAR_END="\007"
E_USAGE=65
E_NORMAL=0
E_CANCEL=66
function cleanDir {
local dir="$1"
local oldPwd="$(pwd)"
local bakFile=
if [[ "$dir" =~ "$backupDir" ]]; then
local fullDir="$dir"
else
local fullDir="$oldPwd/$dir"
fi
counter="0"
builtin cd "$dir" 2> /dev/null
IFS="\n" ls -1 --reverse | while read bakFile; do
cleanFiles "$bakFile"
done
builtin cd "$oldPwd" 2> /dev/null
rmdir "$dir" 2> /dev/null
}
function cleanFiles {
local bakFile="$1"
if [ -d "$bakFile" ]; then
cleanDir "$bakFile"
return
fi
local fullFile="$(pwd)/$bakFile"
local abbrFile="$(echo "$fullFile" | sed "s/${escDir}//g")"
if [[ ! "$bakFile" =~ '^file-insurance\.[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}-[0-9]{2}-[0-9]{2}\.bz2$' ]]; then
echo -e "${ERROR_COLOR}WARNING: Skipping file with unexpected name: ${FILE_COLOR}$abbrFile${NO_COLOR}" 1>&2
return
fi
if [ "$counter" = "0" ]; then
(( counter++ ))
[ -z "$quiet" ] && echo -e "${MSG_COLOR}Preserving ${FILE_COLOR}$abbrFile${MSG_COLOR}...${NO_COLOR}"
else
[ -z "$quiet" ] && echo -e "Removing ${FILE_COLOR}$abbrFile${NO_COLOR}..."
rm "$bakFile"
(( counter++ ))
fi
}
function usage {
echo -e "${MSG_COLOR}Usage:"
echo -e "$prog [--quiet|--help]${NO_COLOR}"
echo -e "\n$prog cleans up after file-insurance by deleting all backup"
echo -e "files it created except the most recent.\n"
echo "With the --quiet option, $prog is silent except for errors."
}
function main {
local i=
case "$1" in
"--help")
usage
exit $E_USAGE
;;
"--quiet")
quiet="true"
FILE_COLOR=
ERROR_COLOR=
;;
esac
if [ -z "$quiet" ]; then
backupDirAbbr="$(echo "$backupDir" | sed "s/^$(echo "$HOME" | sed 's/[\/[{?.+*]/\\&/g')/~/g")"
echo -n "Do you want to prune file-insurance's backup directory ($backupDirAbbr)? [y/N] "
read answer
if [[ ! "$answer" =~ '[Yy]' ]]; then
echo "Exiting..."
exit $E_CANCEL
fi
fi
cleanDir "$backupDir"
exit $E_NORMAL
}
main "$@"start-program-with-file-insurance
View raw: start-program-with-file-insurance
#! /bin/bash
# -*- coding: utf-8 -*-
#
# Copyright by Scott Severance
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
"$@" &
pid=$!
echo "$@"
shift
echo "file-insurance $pid $@"
file-insurance "$pid" "$@"