No good server would be complete without a system to back up your files and folders. For this purpose we use bacula.\u00a0 In this instance we already have postgresql installed so all we need to do is create a user<\/p>\n
#createuser bacula -DRSP -U postgres<\/pre>\nOnce we have a user we can begin installing bacula<\/p>\n
#apt-get install bacula-server bacula-director-pgsql bacula-common-pgsql bacula-console<\/pre>\nYou will be prompted to set up a database for bacula. Select \u201cyes\u201d to allow dbconfig-common to automatically configure the database.<\/p>\n
Next, we need to create some directories to act as our backup and restore points. Using the \u201c-p\u201d flag with the \u201cmkdir\u201d command creates any necessary parent directories along with the target directory:<\/p>\n
#mkdir -p \/bacula\/backup \/bacula\/restore \/etc\/bacula\/conf.d\/clients \/etc\/bacula\/conf.d\/filesets \/etc\/bacula\/conf.d\/jobs<\/pre>\nWe need to change the file permissions so that only the bacula process can access these locations.<\/p>\n
#chown -R bacula \/etc\/bacula\r\n#chmod -R 700 \/etc\/bacula<\/pre>\n<\/p>\n
Configuring bacula<\/strong><\/p>\n
If you are using a tape drive then the first thing that you are probably going to want to know is how to access it. If it is a scsi tape drive then you will need to install lsscsi<\/p>\n
#apt-get install lsscsi<\/pre>\nOnce installed you can run lsscsi to retrieve a list of devices, in all of the installs so far the tape has been on \/dev\/st0, but that cannot be guaranteed. Running lsscsi should return something like<\/p>\n
[5:0:0:0]\u00a0\u00a0 tape\u00a0\u00a0 IBM\u00a0\u00a0\u00a0\u00a0 ULTRIUM-TD4\u00a0\u00a0\u00a0\u00a0 8371 \/dev\/st0<\/pre>\nYou can now use this information in your bacula-sd.conf file<\/p>\n
<\/p>\n
bacula-sd.conf<\/strong><\/p>\n
bacula-sd.conf is where you configure the devices (be they directories or hardware) that you will be backing up on to. In the example we are using we have an LTO-4 drive so we will start by uncommenting the LTO-4 stanza. It should now look something like this<\/p>\n
Device {\r\n Name = LTO-4\r\n Media Type = LTO-4\r\n Archive Device = \/dev\/nst0\r\n AutomaticMount = yes;\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # when device opened, read it\r\n AlwaysOpen = yes;\r\n RemovableMedia = yes;\r\n RandomAccess = no;\r\n Maximum File Size = 5GB\r\n # Changer Command = \"\/etc\/bacula\/scripts\/mtx-changer %c %o %S %a %d\"\r\n # Changer Device = \/dev\/sg0\r\n # AutoChanger = yes\r\n # Enable the Alert command only if you have the mtx package loaded\r\n # Alert Command = \"sh -c 'tapeinfo -f %c |grep TapeAlert|cat'\"\r\n # If you have smartctl, enable this, it has more info than tapeinfo\r\n # Alert Command = \"sh -c 'smartctl -H -l error %c'\"\r\n}<\/pre>\nNow we need to change the Archive Device directive to \/dev\/st0 to reflect our configuration that was retrieved using lsscsi<\/p>\n
Archive Device = \/dev\/st0<\/pre>\nIf we are connecting to clients then we need to exit the Storage stanza by commenting out the SDAddress directive<\/p>\n
# SDAddress = 127.0.0.1<\/pre>\nIf we are backing up to a file then we need to edit the FileStorage declaration to match our chosen configuration. This is fairly self explanatory and configurations are so varying that I am not going to go into too much detail, but at the very least you need to set the \u201cArchive Device\u201d directive correctly.<\/p>\n
Device {\r\n Name = FileStorage\r\n Media Type = File\r\n Archive Device = \/nonexistant\/path\/to\/file\/archive\/dir\r\n LabelMedia = yes;\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # lets Bacula label unlabeled media\r\n Random Access = Yes;\r\n AutomaticMount = yes;\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # when device opened, read it\r\n RemovableMedia = no;\r\n AlwaysOpen = no;\r\n}<\/pre>\nTo check the syntax run after you have saved your file run<\/p>\n
#bacula-sd -tc \/etc\/bacula\/bacula-sd.conf<\/pre>\nno output signifies that our configuration file is valid (not necessarily correct, but at least valid).<\/p>\n
<\/p>\n
Mail Notifications<\/strong><\/p>\n
Bacula has its own built in program called bsmtp<\/strong>, but this does not work on our system because port 25 is already being used by postfix. We also have the problem that we need to authenticate. The workaround for this is to use msmtp<\/strong> and mutt<\/strong>.<\/p>\n
I initially tried to use sendemail<\/strong>, but could only get it working on a localhost email server that was using the snakeoil certificate (changing the certificate to a self signed one resulted in certificate errors).<\/p>\n
<\/p>\n
MSMTP<\/strong><\/p>\n
My latest attempt at getting email notifcations working is msmpt. To install it run<\/p>\n
#apt-get install msmtp\r\n#touch \/etc\/msmtprc<\/pre>\nThen add the following lines to the newly created file:<\/p>\n
host hostname.fqdn.suffix\r\nauth on\r\nport 587\r\nuser emailuser@hostname.fqdn.suffix\r\npassword myemailloginpassword\r\ntls on<\/pre>\n<\/p>\n
Option 1 (CA signed certificates)<\/strong><\/p>\n
If postfix is using a certificate certified by a certificate authority then you need to add the following line:<\/p>\n
tls_trust_file \/etc\/ssl\/certs\/ca-certificates.crt<\/pre>\nThis will allow the SHA fingerprint to be updated automatically and is particularly useful when your server is using letsencrypt for certification and needs to regularly renew your certificates.<\/p>\n
<\/p>\n
Option 2 (Self signed certificates)<\/strong><\/p>\n
If postfix is using a self signed certificate then you will need to do a little extra configuring to get the SHA fingerprint.\u00a0 Save the file and run the following from the command line:<\/p>\n
#msmtp --serverinfo --tls-certcheck=off<\/pre>\nThis should return the SHA1 finterprint. Copy the fingerprint and reopen \/etc\/msmtp and add the following to the end of the file<\/p>\n
tls_fingerprint sd:sf:24:df:........<\/pre>\nusing your the fingerprint that you got from the previous command. Save the file and you are now ready to test the configuration.<\/p>\n
NOTE:<\/strong> From here on in the set up is identical regardless of which option you used above.<\/p>\n
Next run:<\/p>\n
#msmtp --serverinfo<\/pre>\nIf all is well this should provide the same information as before, but without needing to turn off tls certificate checking.<\/p>\n
The next thing to do is install and configure mutt (although it is usually installed by this point)<\/p>\n
#apt-get install mutt<\/pre>\nNow you need to make a bacula home directory and grant permissions to the bacula user and group<\/p>\n
#mkdir \/home\/bacula\r\n#chown bacula:bacula \/home\/bacula<\/pre>\nNow you can create a file called .muttrc<\/p>\n
#touch \/home\/bacula\/.muttrc\r\n#chown bacula:bacula \/home\/bacula\/.muttrc<\/pre>\nand edit it by adding the following<\/p>\n
set sendmail=\"\/usr\/bin\/msmtp\"\r\nset use_from=yes\r\nset realname=\"WhateverNameYouLike\"\r\nset from= emailuser@hostname.fqdn.suffix\r\nset envelope_from=yes<\/pre>\nNow you can test that it is working by running mutt<\/p>\n
#su bacula\r\n#mutt<\/pre>\nand doing the following<\/p>\n
Press \u2018m\u2019 and type the email address that you want to send the test mail to;<\/p>\n
Press enter and give it a subject;<\/p>\n
Pressing enter again will bring up an editor where you can type the body of the message;<\/p>\n
Type something in the messge and hit <ctrl>+x to exit<\/p>\n
Press Y to save the message and then <enter> to leave the filename at its default.<\/p>\n
Now you will be taken back to the mutt screen where you can press \u2018y\u2019 to send the message.\u00a0 If all is well then you can press \u2018q\u2019 to quit mutt, if not then you will need to debug before you continue to the next stage.<\/p>\n
Note that wen you start mutt it will come up with a message at the bottom of the screen telling you there is no such file as \/var\/mail\/bacula; Do not worry about this, it is not a problem because we are only sending mail on this account<\/p>\n
If all is well then you can configure bacula to send email notifications by navigating to the Message stanza with the name \u201cStandard\u201d within \/etc\/bacula\/bacula-dir.conf and change the mail command to<\/p>\n
mailcommand = \"\/usr\/bin\/mutt -F \/home\/bacula\/.muttrc -s \\\"%n \\(%l\\) %e\\\" %r\"<\/pre>\nAnd the operator command to<\/p>\n
operatorcommand = \"\/usr\/bin\/mutt -F \/home\/bacula\/.muttrc -s \\\"\\(Bacula\\) \\<%r\\>\\\" -s \\\"Bacula: Intervention needed for %j\\\" %r\"<\/pre>\nNow set the mail directive to the email address that you want to send the notifications to<\/p>\n
mail = recipient@fqdn.suffix<\/pre>\nNext move down to the Messages stanza with the name Daemon and change the mail command there to<\/p>\n
mailcommand = \"\/usr\/bin\/mutt -F \/home\/bacula\/.muttrc -s \\\"\\(Bacula\\) \\<%r\\>\\\" -s \\\"Bacula daemon message\\\" %r\"<\/pre>\nAnd again you need to set the mail directive to the desired recipient<\/p>\n
mail = recipient@fqdn.suffix<\/pre>\nIf you save and quit then run bconsole and run a job without putting a tape in then you should get an email telling you to do so. Keep an eye on \/var\/log\/mail.log on your email server for errors.<\/p>\n
<\/p>\n
Ejecting tape on completion<\/strong><\/p>\n
There are 2 ways to do this, much of the internet seems to prefer the script version, but I cannot see any advantage over adding a single line to the storage daemon configuration.<\/p>\n
<\/p>\n
Here is the script version, if you feel the necessity:<\/strong><\/p>\n
If you are administering a remote server where the user\u2019s memory needs jogging then you will probably want to eject the tape after you have performed a backup. In order to do this you will need to write a small bash script and change the \u201cAlwaysOpen\u201d option in bacula-sd.conf to \u201cno\u201d<\/p>\n
First<\/p>\n
#touch \/etc\/bacula\/scripts\/autoeject\r\n#chown bacula:bacula \/etc\/bacula\/scripts\/autoeject\r\n#chmod 777 \/etc\/bacula\/scripts\/autoejec<\/pre>\nThen edit \/etc\/bacula\/scripts\/autoeject and add the following lines<\/p>\n
#!\/bin\/sh\r\n# eject the tape\r\nmt -f \/dev\/st0 rewind\r\nmt -f \/dev\/st0 eject<\/pre>\nassuming that st0<\/strong> is your tape drive. If you are unsure then you will need to install lsscsi and run to find your tape drive.<\/p>\n
Once you have finished with the configuration file then you need to go to \/etc\/bacula\/bacula-dir.conf and tell it to run the command after the job completes. To do so you will need to add the following line to each \u201cJob\u201c stanza.<\/p>\n
RunAfterJob = \"\/etc\/bacula\/scripts\/autoeject\"<\/pre>\n<\/p>\n
Alternatively:<\/strong><\/p>\n
Edit \/etc\/bacula\/bacula-sd.conf. Navigate to the Device stanza for your tape drive and add the line<\/p>\n
OfflineOnUnmount = yes<\/pre>\nSave and then restart the daemon.<\/p>\n
<\/p>\n
Troubleshooting<\/strong><\/p>\n
To check to see what user bacula-director is running as run<\/p>\n
#ps axuf | grep bacula-dir<\/pre>\nI had some permission issues and could not back up the catalog. To resolve these I connected to the database using psql and ran<\/p>\n
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO bacula;\r\nGRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO bacula;<\/pre>\nTo test that this has worked if you su to the user \u2018bacula\u2019 and then run<\/p>\n
$pg_dump bacula > ~\/bacula.sql<\/pre>\nYou should not get any errors. If all is well, you should be able to back up the catalog using bconsole<\/p>\n
If everything seems to be working and your mail server log is showing that the messages have been sent, but you cannot find them, check the date and time on the box running bacula. The best thing to do is install ntp<\/strong> on it. to check that it is running run<\/p>\n
#ntpq -p<\/pre>\n<\/p>\n
Monitoring<\/strong><\/p>\n
To monitor a backup job: From the command line type<\/p>\n
#watch -n 1 'echo \"status storage=LTO-3\" | bconsole | grep Bytes | head -n1'<\/pre>\nyou will have to use <ctrl>l+C to escape<\/p>\n
<\/p>\n
Disaster recovery<\/strong><\/p>\n
If you have lost everything including the backup database then you will have to recover the catalog from the tapes before you can do a restore.<\/p>\n
Although all tapes should be physically labeled there is a possibility that they are not labeled correctly, so once you have a version of bacula installed the easiest way to check the digital label is to insert the tape run the btape<\/strong> command<\/p>\n
#btape -v \/dev\/st0<\/pre>\nThis will bring up the btape prompt (which is identical to the bconsole prompt but has a different command set). From the console run readlabel<\/strong> which will then print all the label information to the screen. note that VolName<\/strong> is the label name that you are looking for.<\/p>\n
Note<\/strong>: Usually btape<\/strong> will eject the tape when you quit but if you need to eject the tape manually then exit to the command prompt and run<\/p>\n
#mt -f \/dev\/st0 eject<\/pre>\nFrom the normal command prompt you now need to be running bscan<\/p>\n
#bscan -c bacula-sd.conf -v -V MyVolume_Name \/dev\/st0<\/pre>\nHere MyVolume_Name is the label that was returned using btape<\/strong>\u2019s readlabel<\/strong> command above. This will just go through the motions to check your credentials etc. without actually restoring the catalog.<\/p>\n
If you have set up a password on your postgresql database then you will need to append the command with the password for the bacula user as follows:<\/p>\n
#bscan -c bacula-sd.conf -v -V MyVolume_Name \/dev\/st0 \u2013P mypassword<\/pre>\nOnce you have checked that it goes through the motions then you can move on to re cataloging the tape:<\/p>\n
#bscan \u2013s -m -c bacula-sd.conf -v -V MyVolume_Name \/dev\/st0 \u2013P mypassword<\/pre>\nAs you will note the command is exactly the same, but with the added \u2013s and \u2013m flags.<\/p>\n
Once this is complete you should be able to restore the files from bconsole. If after typing \u201crestore\u201d you press \u201c1\u201d the backup jobs that were written to your tapes.<\/p>\n
If you want to restore the whole file set to a drive press \u201c3\u201d, enter the job id and when you get to the file selection simply type \u201cmark *<\/p>\n
<\/p>\n
Find and Restore a random file without much to go on other than its name<\/strong><\/p>\n
If you want to restore a random file manually then the first thing you need to do is find out where it is. To do this you need to rely on sql statements.<\/p>\n
Connect to the bacula postgresql database using psql and run the following command<\/p>\n
SELECT * FROM filename WHER Name ILIKE \u2018%file%\u2019<\/pre>\nNote that here \u201cfilename\u201d is actually what you want to use here as it is the name of the table.<\/p>\n
Once you have identified a likely filename, do something like this:<\/p>\n
SELECT File.FileId, Path.Path, Filename.Name, Client.Name, Job.JobId, Job.starttime\r\nFROM File\r\nJOIN Filename ON File.FilenameId = Filename.FilenameId\r\nJOIN Path ON File.PathId = Path.PathId\r\nJOIN Job ON Job.JobId = File.JobId\r\nJOIN Client ON Job.ClientId = Client.ClientId\r\nWHERE Filename.Name = 'your filename here';<\/pre>\nThis will give you the name of the client, the job ID it was backed up in, and where on that client it is located and allow you to restore it in the normal way (usually using the job ID and marking it).<\/p>\n
<\/p>\n
Restoring when there is no database<\/strong><\/p>\n
This can be a little bit of a tedious task, but it is sometimes necessary if you have lost the database or are trying to restore data from a failed backup (assuming of course that you know which tape you are trying to restore ti from; if not then it can be a hell of a lot more tedious)<\/p>\n
First you need to get the tape lable using btape in the same manner as described above. Once you have the correct name of the tape you can use bls to read the raw data from the tape NOTE: this can take a long time.<\/p>\n
Before you do this, and if you nave not already done so, you may want to install screen (which keeps ssh sessions alive even if the link is broken or the window closed).<\/p>\n
#apt-get install screen<\/pre>\nType \u201cscreen<\/strong>\u201d from the command line to invoke a session and \u201cscreen -r<\/strong>\u201d to reopen a session.<\/p>\n
If you only have one session then it will display it, if not then it will give you a list, from here you can type screen -r<\/strong> plus the session id that you wish to view<\/p>\n
#screen -r 12350.pts-1.hostname<\/pre>\nGreat, now we have that out of the way, back to reading our tape. To do this we invoke the bls<\/strong><\/p>\n
The format of the command is bls -j -V<\/strong> plus the tape label you got from the btape readlable command, plus the path to your tape drive eg. \/dev\/st0<\/strong> (or whatever you tape drive is identified as) eg.<\/p>\n
#bls -j -V 2019021101-ARCHIVE \/dev\/st0<\/pre>\nIf the recovery gods are smiling down on you then you should eventually be presented with a list of all the write sessions on that particular tape.<\/p>\n
If you are reading this while actuall doing a recover like then then stop here. Go make yourself a brew; hey have a night out, come back in the morning; whatever. As I said, this could take some time.<\/p>\n
Sooner or later, as promised you will be presented with a list that looks something like this<\/p>\n
bls: butil.c:258 Using device: “DDS-4” for reading.
\n11-Jul 11:54 bls: Ready to read from volume “Test1” on device “DDS-4” (\/dev\/nst0).
\nVolume Record: File:blk=0:1 SessId=4 SessTime=1121074625 JobId=0 DataLen=165
\nBegin Job Session Record: File:blk=0:2 SessId=4 SessTime=1121074625 JobId=1 Level=F Type=B
\nBegin Job Session Record: File:blk=0:3 SessId=5 SessTime=1121074625 JobId=5 Level=F Type=B
\nBegin Job Session Record: File:blk=0:6 SessId=3 SessTime=1121074625 JobId=2 Level=F Type=B
\nBegin Job Session Record: File:blk=0:13 SessId=2 SessTime=1121074625 JobId=4 Level=F Type=B
\nEnd Job Session Record: File:blk=0:99 SessId=3 SessTime=1121074625 JobId=2 Level=F Type=B
\nFiles=168 Bytes=1,732,978 Errors=0 Status=T
\nEnd Job Session Record: File:blk=0:101 SessId=2 SessTime=1121074625 JobId=4 Level=F Type=B<\/p>\nIf this list is not followed by a command prompt then it hasn\u2019t finishted; go have another night out and come back later.\u00a0 If it has finished then you now have enough information to restore your session.<\/p>\n
To do so the next thing you need to do is make a file. You can call it anything you like, it is just the contents that need to be correct.<\/p>\n
Peruse the list to find the session that you want to restore. Once you have found the line you will require 2 pieced of information form it (the session id and the session time (SessId<\/strong> and SessTime<\/strong> respectively).<\/p>\n
Open your favourite editor and create your file with the following 3 lines in the format:<\/p>\n
Volume = 2019021101-ARCHIVE\r\nVolSessionId = 7\r\nVolSessionTime = 154979709<\/pre>\nNote that the information appending the \u2018=\u2019 sign will be dependent on the information that you have gathered from the tape.<\/p>\n
Save the file and then begin the recovery process by calling bextract. The format of bextract is \u201cbextract -b plus the path to the file that you created in the previous step (along with the filename) plus the device path (eg \/dev\/st0) and finally the location where you wish the recovered files to be placed. Eg.<\/p>\n
#bextract -b \/path\/to\/myFile \/dev\/st0 \/tmp<\/pre>\nNow go have another couple of well deserved nights out and by the time you sober up it might (at a push) have recoverd your lost files (did I mention that this is slow).<\/p>\n","protected":false},"excerpt":{"rendered":"
No good server would be complete without a system to back up your files and folders. For this purpose we use bacula.\u00a0 In this instance we already have postgresql installed so all we need to do is create a user #createuser bacula -DRSP -U postgres Once we have a user we can begin installing bacula […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[],"class_list":["post-152","post","type-post","status-publish","format-standard","hentry","category-debian-server"],"_links":{"self":[{"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/posts\/152","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=152"}],"version-history":[{"count":2,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/posts\/152\/revisions"}],"predecessor-version":[{"id":327,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/posts\/152\/revisions\/327"}],"wp:attachment":[{"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=152"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=152"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=152"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}