No good server would be complete without a system to back up your files and folders. For this purpose we use bacula. In this instance we already have postgresql installed so all we need to do is create a user
#createuser bacula -DRSP -U postgres
Once we have a user we can begin installing bacula
#apt-get install bacula-server bacula-director-pgsql bacula-common-pgsql bacula-console
You will be prompted to set up a database for bacula. Select “yes” to allow dbconfig-common to automatically configure the database.
Next, we need to create some directories to act as our backup and restore points. Using the “-p” flag with the “mkdir” command creates any necessary parent directories along with the target directory:
#mkdir -p /bacula/backup /bacula/restore /etc/bacula/conf.d/clients /etc/bacula/conf.d/filesets /etc/bacula/conf.d/jobs
We need to change the file permissions so that only the bacula process can access these locations.
#chown -R bacula /etc/bacula #chmod -R 700 /etc/bacula
Configuring bacula
If you are using a tape drive then the first thing that you are probably going to want to know is how to access it. If it is a scsi tape drive then you will need to install lsscsi
#apt-get install lsscsi
Once installed you can run lsscsi to retrieve a list of devices, in all of the installs so far the tape has been on /dev/st0, but that cannot be guaranteed. Running lsscsi should return something like
[5:0:0:0] tape IBM ULTRIUM-TD4 8371 /dev/st0
You can now use this information in your bacula-sd.conf file
bacula-sd.conf
bacula-sd.conf is where you configure the devices (be they directories or hardware) that you will be backing up on to. In the example we are using we have an LTO-4 drive so we will start by uncommenting the LTO-4 stanza. It should now look something like this
Device {
Name = LTO-4
Media Type = LTO-4
Archive Device = /dev/nst0
AutomaticMount = yes; # when device opened, read it
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
Maximum File Size = 5GB
# Changer Command = "/etc/bacula/scripts/mtx-changer %c %o %S %a %d"
# Changer Device = /dev/sg0
# AutoChanger = yes
# Enable the Alert command only if you have the mtx package loaded
# Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
# If you have smartctl, enable this, it has more info than tapeinfo
# Alert Command = "sh -c 'smartctl -H -l error %c'"
}
Now we need to change the Archive Device directive to /dev/st0 to reflect our configuration that was retrieved using lsscsi
Archive Device = /dev/st0
If we are connecting to clients then we need to exit the Storage stanza by commenting out the SDAddress directive
# SDAddress = 127.0.0.1
If we are backing up to a file then we need to edit the FileStorage declaration to match our chosen configuration. This is fairly self explanatory and configurations are so varying that I am not going to go into too much detail, but at the very least you need to set the “Archive Device” directive correctly.
Device {
Name = FileStorage
Media Type = File
Archive Device = /nonexistant/path/to/file/archive/dir
LabelMedia = yes; # lets Bacula label unlabeled media
Random Access = Yes;
AutomaticMount = yes; # when device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
}
To check the syntax run after you have saved your file run
#bacula-sd -tc /etc/bacula/bacula-sd.conf
no output signifies that our configuration file is valid (not necessarily correct, but at least valid).
Mail Notifications
Bacula has its own built in program called bsmtp, but this does not work on our system because port 25 is already being used by postfix. We also have the problem that we need to authenticate. The workaround for this is to use msmtp and mutt.
I initially tried to use sendemail, but could only get it working on a localhost email server that was using the snakeoil certificate (changing the certificate to a self signed one resulted in certificate errors).
MSMTP
My latest attempt at getting email notifcations working is msmpt. To install it run
#apt-get install msmtp #touch /etc/msmtprc
Then add the following lines to the newly created file:
host hostname.fqdn.suffix auth on port 587 user emailuser@hostname.fqdn.suffix password myemailloginpassword tls on
Option 1 (CA signed certificates)
If postfix is using a certificate certified by a certificate authority then you need to add the following line:
tls_trust_file /etc/ssl/certs/ca-certificates.crt
This will allow the SHA fingerprint to be updated automatically and is particularly useful when your server is using letsencrypt for certification and needs to regularly renew your certificates.
Option 2 (Self signed certificates)
If postfix is using a self signed certificate then you will need to do a little extra configuring to get the SHA fingerprint. Save the file and run the following from the command line:
#msmtp --serverinfo --tls-certcheck=off
This should return the SHA1 finterprint. Copy the fingerprint and reopen /etc/msmtp and add the following to the end of the file
tls_fingerprint sd:sf:24:df:........
using your the fingerprint that you got from the previous command. Save the file and you are now ready to test the configuration.
NOTE: From here on in the set up is identical regardless of which option you used above.
Next run:
#msmtp --serverinfo
If all is well this should provide the same information as before, but without needing to turn off tls certificate checking.
The next thing to do is install and configure mutt (although it is usually installed by this point)
#apt-get install mutt
Now you need to make a bacula home directory and grant permissions to the bacula user and group
#mkdir /home/bacula #chown bacula:bacula /home/bacula
Now you can create a file called .muttrc
#touch /home/bacula/.muttrc #chown bacula:bacula /home/bacula/.muttrc
and edit it by adding the following
set sendmail="/usr/bin/msmtp" set use_from=yes set realname="WhateverNameYouLike" set from= emailuser@hostname.fqdn.suffix set envelope_from=yes
Now you can test that it is working by running mutt
#su bacula #mutt
and doing the following
Press ‘m’ and type the email address that you want to send the test mail to;
Press enter and give it a subject;
Pressing enter again will bring up an editor where you can type the body of the message;
Type something in the messge and hit <ctrl>+x to exit
Press Y to save the message and then <enter> to leave the filename at its default.
Now you will be taken back to the mutt screen where you can press ‘y’ to send the message. If all is well then you can press ‘q’ to quit mutt, if not then you will need to debug before you continue to the next stage.
Note that wen you start mutt it will come up with a message at the bottom of the screen telling you there is no such file as /var/mail/bacula; Do not worry about this, it is not a problem because we are only sending mail on this account
If all is well then you can configure bacula to send email notifications by navigating to the Message stanza with the name “Standard” within /etc/bacula/bacula-dir.conf and change the mail command to
mailcommand = "/usr/bin/mutt -F /home/bacula/.muttrc -s \"%n \(%l\) %e\" %r"
And the operator command to
operatorcommand = "/usr/bin/mutt -F /home/bacula/.muttrc -s \"\(Bacula\) \<%r\>\" -s \"Bacula: Intervention needed for %j\" %r"
Now set the mail directive to the email address that you want to send the notifications to
mail = recipient@fqdn.suffix
Next move down to the Messages stanza with the name Daemon and change the mail command there to
mailcommand = "/usr/bin/mutt -F /home/bacula/.muttrc -s \"\(Bacula\) \<%r\>\" -s \"Bacula daemon message\" %r"
And again you need to set the mail directive to the desired recipient
mail = recipient@fqdn.suffix
If you save and quit then run bconsole and run a job without putting a tape in then you should get an email telling you to do so. Keep an eye on /var/log/mail.log on your email server for errors.
Ejecting tape on completion
There are 2 ways to do this, much of the internet seems to prefer the script version, but I cannot see any advantage over adding a single line to the storage daemon configuration.
Here is the script version, if you feel the necessity:
If you are administering a remote server where the user’s memory needs jogging then you will probably want to eject the tape after you have performed a backup. In order to do this you will need to write a small bash script and change the “AlwaysOpen” option in bacula-sd.conf to “no”
First
#touch /etc/bacula/scripts/autoeject #chown bacula:bacula /etc/bacula/scripts/autoeject #chmod 777 /etc/bacula/scripts/autoejec
Then edit /etc/bacula/scripts/autoeject and add the following lines
#!/bin/sh # eject the tape mt -f /dev/st0 rewind mt -f /dev/st0 eject
assuming that st0 is your tape drive. If you are unsure then you will need to install lsscsi and run to find your tape drive.
Once you have finished with the configuration file then you need to go to /etc/bacula/bacula-dir.conf and tell it to run the command after the job completes. To do so you will need to add the following line to each “Job“ stanza.
RunAfterJob = "/etc/bacula/scripts/autoeject"
Alternatively:
Edit /etc/bacula/bacula-sd.conf. Navigate to the Device stanza for your tape drive and add the line
OfflineOnUnmount = yes
Save and then restart the daemon.
Troubleshooting
To check to see what user bacula-director is running as run
#ps axuf | grep bacula-dir
I had some permission issues and could not back up the catalog. To resolve these I connected to the database using psql and ran
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO bacula; GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO bacula;
To test that this has worked if you su to the user ‘bacula’ and then run
$pg_dump bacula > ~/bacula.sql
You should not get any errors. If all is well, you should be able to back up the catalog using bconsole
If everything seems to be working and your mail server log is showing that the messages have been sent, but you cannot find them, check the date and time on the box running bacula. The best thing to do is install ntp on it. to check that it is running run
#ntpq -p
Monitoring
To monitor a backup job: From the command line type
#watch -n 1 'echo "status storage=LTO-3" | bconsole | grep Bytes | head -n1'
you will have to use <ctrl>l+C to escape
Disaster recovery
If you have lost everything including the backup database then you will have to recover the catalog from the tapes before you can do a restore.
Although all tapes should be physically labeled there is a possibility that they are not labeled correctly, so once you have a version of bacula installed the easiest way to check the digital label is to insert the tape run the btape command
#btape -v /dev/st0
This will bring up the btape prompt (which is identical to the bconsole prompt but has a different command set). From the console run readlabel which will then print all the label information to the screen. note that VolName is the label name that you are looking for.
Note: Usually btape will eject the tape when you quit but if you need to eject the tape manually then exit to the command prompt and run
#mt -f /dev/st0 eject
From the normal command prompt you now need to be running bscan
#bscan -c bacula-sd.conf -v -V MyVolume_Name /dev/st0
Here MyVolume_Name is the label that was returned using btape’s readlabel command above. This will just go through the motions to check your credentials etc. without actually restoring the catalog.
If you have set up a password on your postgresql database then you will need to append the command with the password for the bacula user as follows:
#bscan -c bacula-sd.conf -v -V MyVolume_Name /dev/st0 –P mypassword
Once you have checked that it goes through the motions then you can move on to re cataloging the tape:
#bscan –s -m -c bacula-sd.conf -v -V MyVolume_Name /dev/st0 –P mypassword
As you will note the command is exactly the same, but with the added –s and –m flags.
Once this is complete you should be able to restore the files from bconsole. If after typing “restore” you press “1” the backup jobs that were written to your tapes.
If you want to restore the whole file set to a drive press “3”, enter the job id and when you get to the file selection simply type “mark *
Find and Restore a random file without much to go on other than its name
If you want to restore a random file manually then the first thing you need to do is find out where it is. To do this you need to rely on sql statements.
Connect to the bacula postgresql database using psql and run the following command
SELECT * FROM filename WHER Name ILIKE ‘%file%’
Note that here “filename” is actually what you want to use here as it is the name of the table.
Once you have identified a likely filename, do something like this:
SELECT File.FileId, Path.Path, Filename.Name, Client.Name, Job.JobId, Job.starttime FROM File JOIN Filename ON File.FilenameId = Filename.FilenameId JOIN Path ON File.PathId = Path.PathId JOIN Job ON Job.JobId = File.JobId JOIN Client ON Job.ClientId = Client.ClientId WHERE Filename.Name = 'your filename here';
This will give you the name of the client, the job ID it was backed up in, and where on that client it is located and allow you to restore it in the normal way (usually using the job ID and marking it).
Restoring when there is no database
This can be a little bit of a tedious task, but it is sometimes necessary if you have lost the database or are trying to restore data from a failed backup (assuming of course that you know which tape you are trying to restore ti from; if not then it can be a hell of a lot more tedious)
First you need to get the tape lable using btape in the same manner as described above. Once you have the correct name of the tape you can use bls to read the raw data from the tape NOTE: this can take a long time.
Before you do this, and if you nave not already done so, you may want to install screen (which keeps ssh sessions alive even if the link is broken or the window closed).
#apt-get install screen
Type “screen” from the command line to invoke a session and “screen -r” to reopen a session.
If you only have one session then it will display it, if not then it will give you a list, from here you can type screen -r plus the session id that you wish to view
#screen -r 12350.pts-1.hostname
Great, now we have that out of the way, back to reading our tape. To do this we invoke the bls
The format of the command is bls -j -V plus the tape label you got from the btape readlable command, plus the path to your tape drive eg. /dev/st0 (or whatever you tape drive is identified as) eg.
#bls -j -V 2019021101-ARCHIVE /dev/st0
If the recovery gods are smiling down on you then you should eventually be presented with a list of all the write sessions on that particular tape.
If you are reading this while actuall doing a recover like then then stop here. Go make yourself a brew; hey have a night out, come back in the morning; whatever. As I said, this could take some time.
Sooner or later, as promised you will be presented with a list that looks something like this
bls: butil.c:258 Using device: “DDS-4” for reading.
11-Jul 11:54 bls: Ready to read from volume “Test1” on device “DDS-4” (/dev/nst0).
Volume Record: File:blk=0:1 SessId=4 SessTime=1121074625 JobId=0 DataLen=165
Begin Job Session Record: File:blk=0:2 SessId=4 SessTime=1121074625 JobId=1 Level=F Type=B
Begin Job Session Record: File:blk=0:3 SessId=5 SessTime=1121074625 JobId=5 Level=F Type=B
Begin Job Session Record: File:blk=0:6 SessId=3 SessTime=1121074625 JobId=2 Level=F Type=B
Begin Job Session Record: File:blk=0:13 SessId=2 SessTime=1121074625 JobId=4 Level=F Type=B
End Job Session Record: File:blk=0:99 SessId=3 SessTime=1121074625 JobId=2 Level=F Type=B
Files=168 Bytes=1,732,978 Errors=0 Status=T
End Job Session Record: File:blk=0:101 SessId=2 SessTime=1121074625 JobId=4 Level=F Type=B
If this list is not followed by a command prompt then it hasn’t finishted; go have another night out and come back later. If it has finished then you now have enough information to restore your session.
To do so the next thing you need to do is make a file. You can call it anything you like, it is just the contents that need to be correct.
Peruse the list to find the session that you want to restore. Once you have found the line you will require 2 pieced of information form it (the session id and the session time (SessId and SessTime respectively).
Open your favourite editor and create your file with the following 3 lines in the format:
Volume = 2019021101-ARCHIVE VolSessionId = 7 VolSessionTime = 154979709
Note that the information appending the ‘=’ sign will be dependent on the information that you have gathered from the tape.
Save the file and then begin the recovery process by calling bextract. The format of bextract is “bextract -b plus the path to the file that you created in the previous step (along with the filename) plus the device path (eg /dev/st0) and finally the location where you wish the recovered files to be placed. Eg.
#bextract -b /path/to/myFile /dev/st0 /tmp
Now go have another couple of well deserved nights out and by the time you sober up it might (at a push) have recoverd your lost files (did I mention that this is slow).
Recent Comments