Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OSUG
RESIF
datadump
Commits
b4655aa6
Commit
b4655aa6
authored
Jul 08, 2019
by
Jonathan Schaeffer
Browse files
Checksum compute
parent
7db8906f
Changes
1
Hide whitespace changes
Inline
Side-by-side
src/datadump.sh
View file @
b4655aa6
...
...
@@ -62,6 +62,8 @@ cd $SNAPSHOT_DIR
log
"Starting dump of validated data"
for
dir
in
$(
find
.
-maxdepth
3
-type
d
)
;
do
# First clean workspace
rm
$RESIFDD_WORKDIR
/
*
.tar
IFS
=
'/'
read
-r
-a
YNS
<<<
$dir
[
!
${#
YNS
[@]
}
-eq
4
]
&&
continue
YEAR
=
${
YNS
[1]
}
...
...
@@ -77,36 +79,47 @@ for dir in $(find . -maxdepth 3 -type d); do
fi
SIZE
=
$((
$(
stat
-c
%s
$RESIFDD_WORKDIR
/
$KEY
.tar
)
/
1024
/
1024
/
1024
))
# Check if file exists on irods server
ils
-l
$KEY
/latest.tar
>
/dev/null 2>&1
irods_latest
=
$(
ils
-L
$KEY
/latest.tar
)
log
"[
$KEY
] irods latest file is :
$irods_latest
"
"debug"
if
[[
$?
-eq
0
]]
;
then
log
"[
$KEY
] latest.tar already exists on iRODS server. Let's compare hashes"
# Check hash on irods server
local_sha
=
$(
sha256sum
$RESIFDD_WORKDIR
/
$KEY
.tar |
awk
'{print $1}'
| xxd
-r
-p
|
base64
)
irods_sha
=
$(
ichksum
$KEY
/latest.tar |
awk
-F
':'
'/sha2:/ {print $2; exit;}'
)
# Compare to local hash
# If the hashes differs, then move distant file and push this one
if
[[
"
$local_
ha
sh
"
=
"
$irods_
ha
sh
"
]]
;
then
if
[[
"
$local_sh
a
"
=
"
$irods_sh
a
"
]]
;
then
log
"[
$KEY
] The archive on irods is the same as our version. Skipping."
continue
fi
log
"[
$KEY
] Local checksum differs from last dump. Let's dump it"
irm
$KEY
/previous.tar
>
/dev/null 2>&1
# Silently remove previous.tar, ignore errors
imv
$KEY
/latest.tar
$KEY
/previous.tar
>
/dev/null 2>&1
# Rename latest distant dump tp "previous"
imv
$KEY
/latest.tar
$KEY
/previous.tar
# Rename latest distant dump tp "previous"
if
[[
$?
-ne
0
]]
;
then
# Something wrong moving remote file arround
log
"[
$KEY
] There where something wrong moving
$KEY
/previous.tar, cancel iput"
"error"
# zabbixsnd $KEY
continue
fi
fi
# Send file to IRODS
log
"[
$KEY
] Sending file to iRODS (
$SIZE
GB)"
imkdir
-p
$KEY
# Décommenter le bloc et les redirections pour traiter stdout et stderr différemment
# {
iput
\
--lfrestart
$RESIFDD_WORKDIR
/
$KEY
.lfrestart
\
-X
$RESIFDD_WORKDIR
/
$KEY
.restart
\
--retries
5
-T
-b
-K
\
$RESIFDD_WORKDIR
/
$KEY
.tar
$KEY
/latest.tar
--retries
5
-T
-K
\
$RESIFDD_WORKDIR
/
$KEY
.tar
$KEY
/latest.tar
# 2>&1 1>&3 3>&- | logger -t 'resifdd' -p local3.error; } 3>&1 1>&2 | logger -t 'resifdd' -p local3.info
RC
=
$?
if
[[
$RC
-ne
0
]]
;
then
log
"[
$KEY
] error sending file to irods. The command was : iput --lfrestart
$RESIFDD_WORKDIR
/
$KEY
.lfrestart -X
$RESIFDD_WORKDIR
/
$KEY
.restart --retries 5 -T -b -K
$RESIFDD_WORKDIR
/
$KEY
.tar
$KEY
/latest.tar"
error
log
"[
$KEY
] Error sending file to irods. The command was : iput --lfrestart
$RESIFDD_WORKDIR
/
$KEY
.lfrestart --retries 5 -T -K
$RESIFDD_WORKDIR
/
$KEY
.tar
$KEY
/latest.tar"
'error'
log
"[
$KEY
] Rolling back from
$KEY
/previous.tar to
$KEY
/latest.tar"
# Roll back
irm
-f
$KEY
/latest.tar
imv
$KEY
/previous.tar
$KEY
/latest.tar
exit
1
fi
log
"[
$KEY
] Sent successfully"
log
"Cleaning generated tar
$RESIFDD_WORKDIR
/
$YEAR_$NETWORK_$STATION
.tar"
rm
$RESIFDD_WORKDIR
/
$KEY
.tar
done
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment