OnStream DI-30+RedHat Backup mini-HOWTO
by JP Vossen, CISSP; jp@jpsdomain.org, http://www.jpsdomain.org/
v0.9.2, 16-Jun-2001
This document describes how to use an OnStream DI30 tape drive with RedHat 6.2 and several free backup utilities. It is intended for a anyone intending to use an OnStream DI-30 tape drive, or anyone trying to backup Linux, especially RedHat. Most especially, it’s intended for anyone trying to do both!
It assumes some basic hardware and Linux/UNIX knowledge but little to no knowledge of tape drives and tape backup software on Linux. It provides the tools, all of which are free) and information need to implement a relatively simple rotating weekly backup scheme which is suitable for home or small business use.
The most recent version of this document and the following scripts may be found at:
http://www.jpsdomain.org/linux/OnStream_DI-30-RedHat_Backup_mini-HOWTO.html
http://www.jpsdomain.org/public/rpms/jpbackup-0.9.2-1.i386.rpm
Installing the Drive &
Configuring the System
Finding Out What Is On An “Unknown”
Tape
My File System (More Or Less
Typical)
Part 3: Putting It All Together
Some Notes About Common Backup
Programs
Hints from OnStream Tech Support
If you have not already purchased an OnStream drive to use with Linux, read Part 1 to make sure this is an acceptable solution for you. You might also want to skim the rest to make sure you are comfortable with everything. Then check the OnStream site at http://www.onstreamdata.com/ especially the DI30 product page at http://www.onstreamdata.com/desktop/di30_d.html. If you have already bought the drive – read on!
I wrote this because I could not find anything already out there that answered my need. Since I had to do the research anyway, why not document it properly? I am going to be pretty specific with this mini-HOWTO, because I do not have a lot of resources (time or equipment) to spend on this. If you have different experiences, or can add information, please let me know. Contact information is included with the history section at the end.
I am running a clean install of RedHat 7.1 on a dual-P166MMX Tyan Tomcat IV Dual motherboard. I have an OnStream DI-30 (30 Gig internal IDE interface) that is the master drive on my secondary IDE interface. The slave drive on that interface is a 48x CD-ROM. I have a 45 Gig Western Digital hard drive as the master on my primary IDE interface, and an IDE ZIP drive as the slave. See the backup section for information about drive partitions.
Also, I am not affiliated in any way with RedHat, OnStream or anyone else mentioned herein.
Copyright © 2001 JP Vossen
This howto and the associated documentation and scripts are distributed in the hope that they will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
In no event shall the author be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or any other pecuniary loss) arising out of the use of or inability to use this documentation or scripts.
If you have questions or comments, please contact me at vossenjp@bigfoot.com.
http://www.onstreamdata.com/desktop/di30_d.html
For our purposes, the interesting specs are:
I had a lot of trouble accomplishing this, which is largely this reason I wrote this document – I couldn’t find anything to help me. Two issues particularly stick out in my mind. 1) I was not very comfortable re-compiling or installing kernels. 2) I had no idea how it should work, and what it would look like then it was, in fact, working!
What I’ve learned is that the Linux kernel is pretty darn resilient – it’s hard to screw it up (but I managed!). When you do screw it up, it’s very good about telling you, “No, dumb-ass, you have to turn on packet filtering to allow DNS to run” or whatever. I’ve also learned how to make the drive work, and what it should look like when it does. I hope you find that the information below answers your questions!
After finally getting everything to work, and writing the backup scripts included below, I am very happy with this solution. It is very inexpensive by any metric you want to use; cost of data loss, cost of alternative high-capacity solutions, cost in media and tape-swapping time for other low-capacity solutions, etc. It works quite well for me, but of course your mileage may vary.
First, install the hardware as covered in the documentation that came with your drive. In my case, I had to move the master/slave jumper to master, but that was the only change I made. Otherwise, I took it out of the box and plugged it in. Note that the red stripe on the IDE cable (for pin 1) goes AWAY from the power connector in the drive, which is the opposite of hard drives and CD-ROMs!
Per OnStream Tech Support, do not make your drive a slave to an IDE hard drive. Either make it a master on the second IDE interface, or make it a slave to IDE CD-ROM. Do not make an IDE hard drive a slave to an OnStream tape either.
NOTE: if you are using RedHat 7.1 and/or Kernel 2.4.x skip to the next section entitled “Reboot.” You do not have to actually reboot, just start reading there. Kernel 2.4.x (which is used by RedHat 7.1 and better) already includes the necessary IDE-Tape patch! I am still testing RedHat 7.1, and so far the results are mixed, as they had always been for me. I’m beginning to wonder more about my tape media than anything else...
While not trivial, this is not as bad as you think it is. Sooner or later, if you continue to use Linux, you’re going to have to learn how to compile stuff, especially the kernel. Why not start now?
Also, per http://www.redhat.com/support/hardware/intel/62/rh6.2-hcl-i.ld-7.html:
“The new OnStream Drive (30gig drive in IDE, SCSI, and parallel flavors) does NOT work under 6.2. OnStream Inc. is currently working to develop a driver however. To reiterate, not even the SCSI version works yet.”
Obviously, they are wrong, but they are right with one thing – OnStream tape drives will not work with RedHat kernels – you do have to roll your own (except for RedHat 7.1 and above!).
1. Get the latest pristine kernel source. Do not use 2.2.16 as it has security issues. As I write this, I am using 2.2.18 (ftp://ftp.us.kernel.org/pub/linux/kernel/v2.2/linux-2.2.18.tar.gz)
2. Get the latest version of the IDE-Tape patch for your kernel version. I used ide.2.2.18.1221.patch.gz (ftp://ftp.us.kernel.org/pub/linux/kernel/people/hedrick/ide-2.2.18/ide.2.2.18.1221.patch.gz).
3. Make sure you have the latest drive firmware (http://www.onstreamdata.com/support/linux_dos_firmware.html).
4. Follow the instructions on the OnStream site. The instructions for building the 2.2.16 kernel are close enough for the 2.2.18 (and 2.4.x) as well. (http://www.onstreamdata.com/support/linux/di30_patch.html, http://www.onstreamdata.com/support/linux/linux_kernel216_rebuild.html and check out http://www.linuxtapecert.org/di30_install.html too.)
The trick here is to enable all the stuff you need, without enabling stuff you don’t need. I recommend using “make menuconfig” or if running X “make xconfig” as they are far easier to use and more tolerant of changing your mind than just “make config.” I had some trouble getting “make menuconfig” to run. It kept whining about “curses” so I eventually had to install all the ncurses RPMs on the RedHat 6.2 CD. Something did the trick (I suspect the ncurses-devel package), because it worked after that.
This mini-HOWTO covers both of the situations where you must reboot Linux. That is, you should only ever have to reboot Linux when adding or replacing non-hot-swappable hardware, and when you need to switch to a different kernel version. Other than that, you should never need to reboot! Take THAT Windows.
You should see something like the following on bootup. If you miss it, try “dmesg | grep -i hd” once you have finished booting):
hda: WDC WD450AA-00BAA0, ATA DISK drive
hdb: IOMEGA ZIP 100, ATA DISK drive
hdc: OnStream DI-30, ATAPI TAPE
drive
hdd: ATAPI CDROM 48X, ATAPI CDROM drive
After some services start, and if you are using Kudzu (RedHat’s hardware recognition program), you will be asked if you want to configure your new OnStream DI-30, ATAPI TAPE drive. Say yes.
That’s it! Once the system is completely up and you log in, you should be able to access the drive at /dev/ht0 or /dev/nht0 (assuming this is your first tape drive). You can test this with this command:
mt -f /dev/nht0 status
There are two types of tape device files, the rewinding device and the non-rewinding device. As you can probably guess, the rewinding device rewinds the tape after each operation, the non-rewinding device doesn’t. Be careful with this! If you use the rewinding device, then make two consecutive backups, the second will overwrite the first, which is probably not what you wanted to do! They are specified by the device name for the rewinding device, and the device name prefixed with “n” (for non) for the non-rewinding device. For example, a correctly installed DI-30’s devices are: /dev/ht0 and /dev/nht0.
Some tape software looks for an environment variable, imaginatively called TAPE, to see what tape device to use if nothing is specified. TAPE is often set to /dev/tape, which may or may not actually exist on your system. Also, note that /dev/tape is the rewinding device! Type “echo $TAPE” to see if it’s set on your system. You can edit your /etc/profile and add “TAPE=/dev/tape” if necessary.
You also want to create or verify the following symbolic links (this is for a DI-30):
ln –s /dev/ht0 /dev/tape Rewinding device
ln –s /dev/nht0 /dev/ntape Non-rewinding device
See the man page for the “mt” command – you’re going to need it. Some highlights to get you started are:
|
Function |
Command |
Comments |
|
Rewind |
mt –f /dev/ht0 rewind |
Rewind the tape to the beginning (remember about the rewinding and non-rewinding devices!) |
|
Erase |
mt –f /dev/ht0 erase |
Erase from the current position to the end of the tape (some versions only). Thus, to clear the whole tape, rewind first. This also initialized a new tape. |
|
Status |
mt –f /dev/ht0 status |
Get tape status (more below). |
|
Eject |
mt –f /dev/ht0 offline |
Does not actually eject the tape on DI-30 tape drives, but may help if the tape will not come out when you manually press the eject button. |
|
retension |
mt –f /dev/ht0 retension |
Rewind, fast forward to the end of the tape, then rewind. This increases the life of the tape. |
|
fast forward |
mt –f /dev/ht0 fsf # |
Fast forward to the beginning of a next archive, where # is the number of archives to skip over. |
|
end (of data) |
mt –f /dev/ht0 eod |
Fast forward to the end of the last archive. |
Try the tape status command from above. It looks like this if it works and there is an initialized tape in the drive:
/root# mt status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 32768 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (41000000):
BOT ONLINE
And the tape is not initialized, it’ll take about 10 minutes for the command to come back and fail like this:
/root# mt status
SCSI 2 tape drive:
File number=0, block number=-1, partition=0.
Tape block size 32768 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (1000000):
ONLINE
If there is no tape in the drive you’ll get this:
/root# mt status
/dev/ht0: Device or resource busy
If you got the first message, congratulations. You’re all set! If you got the second one, you forgot to erase the tape, which you need to do before using it for the first time. This initializes the ADRL header, which you probably just got a bunch of console messages about.
OK, we now have something to backup to. Now, what data do we want to backup, and how do we want to do it? Before we get to that, however, we have one more quick tape operation issue.
With a DI-30, there is no easy way to find out what is on a tape. You have to just know. Thus, having a simple system, labeling and cataloging are important. If you don’t know, the best you can do is try various table of contents (ToC) commands from various tape backup programs to see what you get. You can also try using mt to fsf and try ToC commands again. Using my tape script, you could look for tar and afio (cpio) archives like this:
tape rewind
tape ttoc
tape toc
tape findnext 1
tape ttoc
tape toc
etc.
When you get “afio: "/dev/tape": No input” you are past the data (i.e. archives) on the tape. If you got nothing, then you are using the wrong program, which means the archives are not tar or afio (cpio) or there is nothing on the tape. If you get “tar: This does not look like a tar archive” – well, you can figure that one out. Likewise, afio will say “afio: "/dev/tape": Unrecognizable archive”
First, as I was recently told by a friend and UNIX guru specializing in very high-end high-availability and clustering, you have to be able to RECOVER – you do not necessarily have to backup. Think about that for a moment before you continue. What do you need to have to be able to RECOVER?
Rather than rewrite what has already been well written, I now refer you to the following two chapters:
“The Linux System Administrators' Guide: Chapter 10. Backups” (http://www.linuxdoc.org/LDP/sag/c2202.html) and “Linux Administration Made Easy: Chapter 8. Backup and Restore Procedures” (http://www.linuxdoc.org/LDP/lame/LAME/linux-admin-made-easy/c1315.html).
The two articles above are very good, and I strongly recommend reading them. However, those authors were trying to be Linux generic, and I am being OnStream and RedHat specific, so on with the show.
You need to have a well thought out strategy if you are to recover data successfully. There is no “one size fits all” strategy, because every environment is too different. There are, however, some guidelines:
There are many different types of backups and thinking about them all gives me a headache. However, you have to understand at least a little about some of the types in order to decide which ones you need to implement.
A full backup is just that – you backup the full system – everything. But even that’s not true, as there are always things you never want to backup. As I mentioned above, the /proc filesystem and temp directories at the best example. /proc doesn’t really exist. It’s a made-up filesystem containing all the details about the system. It’s only a filesystem because everything in UNIX is treated as a file. Backup it up is not only useless, some of the system is recursive (it points to itself, more or less) so it can really confuse and even crash your backup. Likewise, backing up the temporary directories is pretty silly. There’s a reason they are called temporary!
Differential and Incremental backups are just different ways of backing up data that has been changes since the last full backup. Likewise, the “levels” use by some program (such as dump/restore) are just ways of representing data that’s changed since the last higher level backup. And there are all kinds of minor variations on all of the above, especially the fact that you can use one type of backup on one day, and another type the next day.
Finally, each type presents different problems to backups and more importantly, restores. As you will see below, it could easily take restores of data from 5 or mare tapes to get back you where you left off. And try to find just one or two specific versions of specific files? OK, I’ll pause while you go take some aspirin. Come to think of it, I’ll take two while you’re at it.
This figure illustrates the difference between a differential and an incremental backup. Note that in a standard Differential Backup each backup uses the same tape, while in a Modified Differential Backup each backup uses a different tape.
[ F U L L ]
[ B A C K U P ]
{ DIFFERENTIAL }
{ DIFFERENTIAL }
{ DIFFERENTIAL }
{ DIFFERENTIAL }
[------------------------------------------------------------------]
[ CHANGES TO YOUR DATA OVER TIME ]
[------------------------------------------------------------------]
[ F U L L ]
[ B A C K U P ] {INCREMENTAL}{INCREMENTAL}{INCREMENTAL}{INCREMENTAL}
[------------------------------------------------------------------]
[ CHANGES TO YOUR DATA OVER TIME ]
[------------------------------------------------------------------]
Then of course, there’s the data to consider. I think that’s best explained by example.
The following table summarizes most of the important information about my environment:
|
Directory |
File system |
Recovery
Criteria |
|
/a |
Symlink to /mnt/floppy_dos |
SKIP |
|
/bin |
/ |
Static |
|
/boot |
/boot |
Static |
|
/cd |
Symlink to /mnt/cdrom |
SKPI |
|
/dev |
/ |
Static, easily recreated on system reinstall |
|
/etc |
/ |
Dynamic, important |
|
/fpy |
Symlink to /mnt/floppy_ext2 |
SKIP |
|
/home |
/home |
Dynamic, important |
|
/lib |
/ |
Static |
|
/lost+found |
/ |
SKIP |
|
/misc |
/ |
??? |
|
/mnt |
/ |
SKIP, easily recreated |
|
/opt |
/ |
Static, not easily recreated on system reinstall |
|
/proc |
/proc (pseudo) |
SKIP! |
|
/root |
/ |
Dynamic, important |
|
/sbin |
/ |
Static |
|
/tmp |
/ |
SKIP |
|
/usr |
/ |
Static |
|
/var |
/var |
Varies widely. Much data in /var is useless and should not be backed up, while other data such as log files, mail spools and printer configuration is important and should be saved. |
|
/var/tmp |
/var |
SKIP |
|
/var/lock |
/var |
SKIP |
|
/var/log |
/var/log |
Dynamic, important |
|
/var/spool/mail |
/var |
Dynamic, important |
|
/var/spool/lpd |
/var |
Dynamic, important (printer configuration data as well as actually spool file – bad design!) |
|
/zip |
Symlink to /mnt/zip |
SKIP |
See http://www.pathname.com/fhs/ for the Filesystem Hierarchy Standard, (v2.1 as of this writing). This details what things should be located where, and is an excellent reference.
The easiest thing to do is a full backup once a day, week or month, depending on your environment and then just call it a day. Depending on how much data you have, how big your tapes are and how fast your tape drive is, this may work for you. Most of the time, not all of your data will fit on one tape (less of a problem with 30 Gig DI-30 tapes), or it’ll take too long to do a full backup, or something. Also, it can take a lot of tapes, which do not grow on trees.
Seven (7) tapes labeled: Week1, Week2, Week3, Month1, Month2, Month3, Month4. The Week tapes are used every week, either Monday to Friday, or just Friday (note these tapes will need to be replaced most often, as they will get the most use). Month1 is used at the end of the first month, and so on. Either the previous week or the previous month tape is moved off-site. Depending on space requirements, the Monday to Friday backups could be incremental, differential or full, and could be appended to each backup set. The Month tapes are complete system backups. This strategy also gives you a 4 month window to recover data, but you may lose the weekly/daily backups of different versions of highly dynamic data, depending on exactly how you set it up.
Another possible option is eleven (11) tapes labeled: Monday, Tuesday, Wednesday, Thursday, Friday1, Friday2, Friday3, Month1, Month2, Month3, Month4. The Monday to Thursday tape are used every Monday to Thursday (note these tapes will need to be replaced most often, as they will get the most use). Friday1 is used on the Friday of the first week, Friday2 at the end of the second week, and so on. Month1 is used at the end of the first month, and so on. Each week, the preceding Friday tape (which will sometimes be a Month tape) is taken to a secure off-site location, while the old off-site tape is brought back. Either the Friday or the Month tapes are complete system backups of EVERYTHING, while the Monday to Thursday tapes are full backups of “dynamic and important” data. This strategy gives you a 4 month window to recover data, plus 5 days of different versions of highly dynamic data. The most you would have to restore is two tapes, the most recent full system, and the most recent full data tapes. However, this requires a good number of tapes, and may take a while to do a full backup. The Monday to Thursday backups could also be differential or incremental, that that will substantially increase restore complexity, while substantially lowering backup time. Additional monthly tapes may be added to give any number of “archival” copies.
Finally, a cheaper way to do it is four (4) tapes labeled: Tape1, Tape2, Tape3, Tape4. These are used either everyday or at the end of the week as above, with the previous tape being taken off-site.
Needless to say, the above barely even scratches the surface of the possible options. If the number of tapes is not an issue, all sorts of other plans will work well. I have found the above plans to work well for me, in my environment over the last several years – your mileage may vary.
My solution in this case is to use eight (8) tapes labeled: Week1, Week2, Week3, Month1, Month2, Month3, Month4, Month5. The weekly tapes are used once a week, early Monday morning (note these tapes will need to be replaced most often, as they will get the most use). Month1 is used at the end of the first month, and so on. Either the previous week or the previous month tape is moved off-site. The Monday backups are “full” backups of the dynamic data, the monthly tapes are complete system backups (with excepts for junk). This strategy also gives you at least a 4 month window to recover different versions.
Now a problem crops up because it works out that some months have five Mondays in them. There are a couple of ways to solve that problem, but I took the easiest – I ignored it. So periodically your “Monthly” tapes will get out of sync with the last Monday of the month. Too bad.
My weekly backup set is the set of: /etc /home /root /var/log /var/named /var/spool
My monthly backup set is the set of: /, minus the set of: /a /cd /fpy /mnt /proc /tmp /var/tmp /zip
Interestingly, it turns turn that the space and time different between my two sets is not very much. I could just use the slightly larger full or monthly set for all tapes, but keep the rotation and other part of the strategy the same. That came as a surprise to me. I expected there to be much more difference. So you’ll just have to try it, and see how you make out. I’m leaving it alone as I see no compelling reason to change it.
I also use the RedHat KickStart and mkbootdisk tools with my own RDISK script. I have written a shell script that automates everything except changing the tape, and I have a 7 day windows to remember to do that.
OK, given everything above, let’s actually get into the details.
KickStart is a RedHat automated installer program. If you install and then use the “mkkickstart” program, you can create an “answer file” that allows you to automatically install everything exactly the same as you just did. That, combined with your CD-ROM, allows for a pretty cool recovery tool in case of disaster. Just replace the failed hardware (or in most cases get close enough) and you’re set. See the RDISK script below and http://www.redhat.com/support/manuals/RHL-6.2-Manual/ref-guide/ch-kickstart2.html for more details.
You should also install and use the “mkbootdisk” command to make a recovery disk that may be able to boot your system if something goes wrong. It’s not a bad idea to keep two of these, and alternate using them when you make changes.
RDISK (the name comes from the similar facility on NT) uses mkkickstart and mkbootdisk, fdisk, rpm and du to capture a lot of critical information about your system, mostly your file system configuration. You can write the data to a floppy, or not (in which case mkbootdisk does not run).
This simple script just copies important files someplace else. You can copy them to a floppy, if they fit, or to another drive or server or whatever. You can keep multiple versions if you want. How you implement it is up to you. It is never used in any of the other scripts here, and is included only as a convenience.
Another useful strategy is to create a /root/updates directory (or whatever) and keep all the installed patches and updates in it. You do updates your system as necessary, don’t you? If you need to use the KickStart file, then restore, it’s amazingly easier to bring the system back up to speed when you can go into /root/updates and basically do an rpm –Fvh *.rpm. OK, it’s a little more complicated than that for some updates such as the kernel, but that works 90% of the time. Also, this directory doubles as a record of how your system differs from a “stock” installation.
jpbackup is the heart of the system. It pulls the other scripts together and actually runs the show. It implements and automates the 3 weekly and 5 monthly tape scheme above, and under normal circumstances (i.e. you do not have to do a restore) all you have to do is change the tape between every Monday. You should really look at the logs as well. In particular, I’ve added code that shows how long the backup took, and how big the backup set is on disk, then how big it is on tape. Given that information, you can tailor your compression settings to speed up your backups if your tapes are big enough. It uses the rewinding device, and pretty much forces you to have only one archive per tape. Conceivably, this wastes tape, but is far more simple in many ways. It keeps a catalog of what is on each tape, named “Monday_1.cat,” “Monday_2.cat,” etc. The afio log file is also kept, with the same name except .log. Finally, a backup.log is kept with start times, and the data sizes.
Here is an outline of operation:
Sample backup.log
Thu Feb 22 02:58:41 EST 2001; START: weekly backup to Monday_1...
Thu Feb 22 06:38:06 EST 2001; FINISH: weekly backup to Monday_1...
Thu Feb 22 06:38:06 EST 2001; START: Verify weekly Monday_1...
Data size on disk: 12069397386, size on tape (GZ 4) 4269799725.
Thu Feb 22 09:25:02 EST 2001; FINISH: Verify weekly Monday_1...
From this you can see that the backup took under 4 hours, that 12 Gig was backed up, but using only Gzip compression level 4 (9 is best compression/slowest, 6 is default) it took up only 4.2 Gig on tape. We also see that the verify took just under three hours.
If you look at Monday_1.log, you’ll also see some verify errors such as the following. That's because some files CHANGED between when they got backed up and when they got verified. This is normal! For example, the backup.log file was updated by the backup itself, after it got backed up. Thus it fails the verify since the disk version is different than the tape version.
afio: "var/log/backup/backup.log": Archive data and file cannot be aligned (disk 1) at Wed Feb 21 16:07:56 2001
afio: "var/log/backup/backup.log": Corrupt archive data (disk 1) at Wed Feb 21 16:07:56 2001
See the end of the backup script for the options I used. See the afio man page for all the options, of which there are a plethora.
Tape is just a simple front end to keep all the tape related commands in one place. Especially since afio has so many options, it’s a real pain to remember and type them all. So you can edit tape to work on your system, then pretty much just run it ad hoc if needed.
The ability to restore or recover is the entire point of this exercise, yet there is not all that much I can say about it. There are many variables, but by now you should be getting a feel for them. The following questions might help. Note that I am dealing only with restoring from tape. Rebuilding the system, using KickStart, recovering from hardware failures, etc. are all beyond the scope of this document.
To restore everything from a compressed tape archive, and to overwrite, you need to be in the root ( / ) directory. To restore everything from a compressed tape archive to a different directory, you need to be in that directory. Then:
afio
-ivxZ -b 32k -M 10m -L /var/log/jpbackup/jpbackup.log -@ root /dev/tape
To restore just “/root” from a compressed tape archive, and to overwrite, you need to be in the root ( / ) directory. To restore everything from a compressed tape archive to a different directory, you need to be in that directory. It can even handle the leading / in the path (even with the use of relative paths)! Then:
afio
-ivxZ -b 32k -M 10m -L /var/log/jpbackup/jpbackup.log -@ root -y
"/root/" /dev/tape
Read the code. It is well documented and there are more notes and tricks in it.
FHS-2.1 compliant (more or less, but RedHat 6.x files locations, not RedHat 7.x – e.g. /usr/doc not /usr/share/doc).
jpbackup is not really meant for interactive use, so it’s not too important where it is. Tape is interactive, so you may want to make a symlink to it from /usr/bin or /usr/sbin.
|
/usr/doc/jpbackup |
Documentation directory |
|
/opt/jpbackup |
Package directory and scripts |
|
/var/opt/jpbackup |
Variable data (exception and flag files) |
|
/var/log/jpbackup |
Log and catalog files |
|
/usr/bin/afio |
afio binary (from afio RPM) |
|
/usr/doc/afio-2.4.6 |
afio documentation (from afio RPM) |
|
/usr/man/man1/ |
afio man page (from afio RPM) |
Backup is the script that actually backs up my system. It’s called from cron every Monday, and it figures out what kind of backup (weekly or monthly) to do by itself.
#/bin/sh
# jpbackup -- to an OnStream DI-30 tape using afio
# http://www.jpsdomain.org/linux/OnStream_DI-30-RedHat_Backup_mini-HOWTO.html
# v0.9 27-Feb-2001 JP Vossen (jp@jpsdomain.org)
# v0.9.1 02-Jun-2001 JPV Added code to delete TapeName.log to avoid appending
# Removed the "-@ ${EMAIL}" by default (see Note 3 below).
# Added data on disk size to Finished backup log message, just in case.
# Updated for RPMs and minor typos
# v0.9.2 16-Jun-2001 JPV bugfixes in tape, corrections to OnStream docs.
MYVER="v0.9.2 16-Jun-2001"
# Copyright 2001 JP Vossen
# This script is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
# In no event shall the author be liable for any damages whatsoever
# (including, without limitation, damages for loss of business profits,
# business interruption, loss of business information, or any other
# pecuniary loss) arising out of the use of or inability to use this script.
# http://www.jpsdomain.org/linux/OnStream_DI-30-RedHat_Backup_mini-HOWTO.html
# REQUIRES:
# afio, developed and tested with afio-2.4.6, which has a minor bug that prints
# "-- compressed" twice during verify operations. That bug is fixed in my RPMs
# (http://www.jpsdomain.org/public/public.html#rpms).
# See http://www.linux.org/apps/AppId_266.html
# and http://metalab.unc.edu/pub/linux/system/backup/afio-2.4.6.tgz (unpactched)
# calcsum.sh requires ksh to be installed on the system. This is provided
# by the RedHat "pdksh" package.
# RDISK optionally requires mkbootdisk, fdisk and mkkickstart. Lack of these
# programs will not affect backups, but then RDISK will not be able to gather
# some of the information it tries to gather before a backup. fdisk is provided
# by the util-linux package, the other two are provided by packages of the same
# name as the tool.
# NOTES [see Other Notes below]:
# 1. This script assumes (actually forces) only one archive per tape.
# 2. See the nobackup file to see stuff you should probably never backup.
# NEVER backup /proc -- it's utterly useless, and could crash your tape
# software!
# 3. Remove the "-@ ${EMAIL}" lines when you get tired of getting boring
# e-mail about it.
# 4. Uses my RDISK script without using a floppy (see RDISK for details)
# 5. Calculates size of data on disk v. size of data on tape. This way you
# can see how your compression options are working, and how you are on
# tape space. If you have space, you can lower compression to speed up
# the backup. The totals are not 100% accurate, as the "ls -lR" method
# includes the sizes of the files in which sub-directory information is
# stored, and the "afio -r" method does not. Still, they are close
# enough. (SizeOnDisk should ALWAYS be bigger) I do this instead of
# using du because that includes slack space (allocated disk use)
# rather than file size, which is a LOT less meaningful in this context.
# 6. Your verify will have errors such as the following. That's because some
# files CHANGED between when they got backed up and when they got verified.
# This is normal!
# afio: "var/log/backup/backup.log": Archive data and file cannot be aligned (disk 1) at Wed Feb 21 16:07:56 2001
# afio: "var/log/backup/backup.log": Corrupt archive data (disk 1) at Wed Feb 21 16:07:56 2001
# 7. This script complies with FSH-2.1 (http://www.pathname.com/fhs/)
####################################################################
# Variables you might need to change
# Unfortunately, if you install a new version of jpbackup, any changes here
# will be lost.
# These are the actual directories that get backed up for each job or type.
# You can make them the same if you want
# Remember that no matter what is listed here, directories in nobackup file
# will not get backed up!
BUWEEKLYJOB="/etc /home /root /var/log /var/named /var/spool"
BUMONTHLYJOB="/"
BUJOB=""
TAPEDEVICE=/dev/tape # Tape device to use
EMAIL=root # Address to send completion e-mail to
GZIPCOMPRESSION=6 # GZip compression level; 1=fastest, 9=smallest; GZ default=6
# Number of tapes (weekly and monthly) [I use 8, see Other Notes below]
NUMWK=3 # Weekly
NUMMO=5 # Monthly
# My other scripts that we need too. If you don't want to use them
# comment them out here and where they are checked for and called.
CALCSUM=/opt/jpbackup/calcsum.ksh # Provides total space used (disk and tape)
RDISK=/opt/jpbackup/RDISK # Gathers recovery info about file system(s)
####################################################################
# Variables you probably should not change
# File path and names (per FSH-2.1)
LOGFILEPATH=/var/log/jpbackup/ # Default log file patch
LOGFILE=${LOGFILEPATH}jpbackup.log # The log file for this script
EXFILEPATH=/var/opt/jpbackup/ # Default EXception and flag file path
NOCOMPRESS=${EXFILEPATH}nocompress # Files not to compress
NOBACKUP=${EXFILEPATH}nobackup # Directories not to backup
LASTMONTHLY=${EXFILEPATH}lastmonthly # The last Monthly tape used
LASTWEEKLY=${EXFILEPATH}lastweekly # The last Weekly tape used
LASTTAPE=${EXFILEPATH}lasttape # The last tape used, period
####################################################################
# Make sure the files we need exist, create them with useful defaults if not.
if [ ! -f "${LASTMONTHLY}" ]; then # The combination of this setting and the next 2...
echo "${NUMMO}" > ${LASTMONTHLY}
fi
if [ ! -f "${LASTWEEKLY}" ]; then # ... will start off so that the first backup...
echo "${NUMWK}" > ${LASTWEEKLY}
fi
if [ ! -f "${LASTTAPE}" ]; then # ... is weekly 1 (Monday_1)
echo "monthly" > ${LASTTAPE}
fi
if [ ! -f "${NOCOMPRESS}" ]; then # These are the defaults, except the last two, which I added for Ghost images
echo ".Z .z .gz .bz2 .tgz .arc .zip .rar .lzh .lha .uc2 .tpz .taz .tgz .rpm .zoo .deb .gif .jpeg .jpg .tif .tiff .png .gho .GHO" > ${NOCOMPRESS}
fi
if [ ! -f "${NOBACKUP}" ]; then # I really doubt you want to back these up (esp. /proc)
echo "/proc" > ${NOBACKUP} # Add others to real file once created...
echo "/tmp" >> ${NOBACKUP}
echo "/var/tmp" >> ${NOBACKUP}
fi
if [ ! -d "${LOGFILEPATH}" ]; then # Make the log file directory
mkdir ${LOGFILEPATH}
fi
if [ ! -d "${EXFILEPATH}" ]; then # Make the EXception and flag file directory
# Note this will make parent directories as needed
mkdir -p ${EXFILEPATH}
fi
if [ ! -f "${CALCSUM}" ]; then # Is calcsum.ksh there?
echo "${CALCSUM} not found. Aborting. Reinstall jpbackup."
exit 3
fi
if [ ! -f "${RDISK}" ]; then # Is rdisk there?
# (Note RDISK will not complain on errors, it just won't work right. Better check it.)
echo "${RDISK} not found. Aborting. Reinstall jpbackup."
exit 3
fi
####################################################################
# Figure out what we are doing
# What did we do last time?
MYLASTTAPE=`cat ${LASTTAPE}`
MYLASTWEEKLY=`cat ${LASTWEEKLY}`
if [ ${MYLASTTAPE} = weekly ]; then # If we did a weekly last time
if [ ${MYLASTWEEKLY} -ge ${NUMWK} ]; then # And we did the LAST weekly
BUTYPE=monthly # Do a monthly now
BUJOB=${BUMONTHLYJOB} # Use the Monthly data set
MYLASTMONTHLY=`cat ${LASTMONTHLY}` # Which monthly did we do last?
if [ ${MYLASTMONTHLY} -ge ${NUMMO} ]; then # If we did the LAST monthly
THISTAPE=1 # Start at the beginning again
TAPENAME=Month_${THISTAPE}
else # Otherwise, just do the monthly
let THISTAPE=${MYLASTMONTHLY}+1
fi
TAPENAME=Month_${THISTAPE}
else # OK; we're in the middle of the weekly cycle: just do it
BUTYPE=weekly
BUJOB=${BUWEEKLYJOB} # Use the Weekly data set
let THISTAPE=${MYLASTWEEKLY}+1
TAPENAME=Monday_${THISTAPE}
fi
elif [ ${MYLASTTAPE} = monthly ]; then # If we did a monthly last time, now we restart the weekly cycle
BUTYPE=weekly
BUJOB=${BUWEEKLYJOB}
THISTAPE=1
TAPENAME=Monday_${THISTAPE}
else # What the heck? Should never get here
echo "" | tee -a ${LOGFILE}
echo "`date`; ERROR: MyLastTape not set right!" | tee -a ${LOGFILE}
echo "" | tee -a ${LOGFILE}
exit 2
fi
# Now that we know what to do -- set the names (You probably will not need
# to touch these). Note DumpFiles are in TMP so they will not get backed up
# (see nobackup), should also be removed, so if they are still there,
# you probably had a problem.
DUMPFILE=/tmp/${TAPENAME} # File find dump (easy to capture to verify it's as expected)
CATFILE=${LOGFILEPATH}${TAPENAME}.cat # "Catalog" for the tape
AFIOLOG=${LOGFILEPATH}${TAPENAME}.log # afio's log file
if [ -f "${AFIOLOG}" ]; then # Remove old TapeName.log file
rm -f ${AFIOLOG}
fi
####################################################################
# Provide a ton of operation feedback
echo ""
echo "Running $0 '${MYVER}' on `date`"
echo "BUType: ${BUTYPE} TapeName: ${TAPENAME}"
echo "NumMO: ${NUMMO} NumWk: ${NUMWK}"
echo "ThisTape: ${THISTAPE} GZip Level: ${GZIPCOMPRESSION}"
echo "BUWeekly Job: ${BUWEEKLYJOB}"
echo "BUMonthly Job: ${BUMONTHLYJOB}"
echo "This BU Job: ${BUJOB}"
echo ""
echo "LastTape: `cat ${LASTTAPE}` Path: ${LASTTAPE}"
echo "LstMntly: `cat ${LASTMONTHLY}` Path: ${LASTMONTHLY}"
echo "LstWkly: `cat ${LASTWEEKLY}` Path: ${LASTWEEKLY}"
echo ""
echo "DumpFile path: ${DUMPFILE}"
echo "CatFile path: ${CATFILE}"
echo "afioLog path: ${AFIOLOG}"
echo "LogFile path: ${LOGFILE}"
echo "NoBackup path: ${NOBACKUP}"
echo "NoCmprss path: ${NOCOMPRESS}"
echo "CalcSum: ${CALCSUM}"
echo "rdisk: ${RDISK}"
echo ""
####################################################################
# Now go do it
# Make sure the tape is rewound, then erase it (also initializes new tapes)
STARTDATE=`date`
echo "${STARTDATE}; START: ${BUTYPE} backup to ${TAPENAME}..." | tee -a ${LOGFILE}
mt rewind
mt erase
# Run my RDISK script, but do not write anything to floppy
${RDISK} nopfy
echo "Doing ${BUTYPE} backup to ${TAPENAME} -- Collecting files..."
# Find the files to backup, the use grep to remove anything listed in the
# NoBackup file. Save data twice, once as just names, and once with file
# size too. Note the NoBackup file (grep) only gets applied to the file
# size data stream here. afio applies it to the backup later using the
# name only stream.
find ${BUJOB} -fprintf ${DUMPFILE}.name "%p\n" -printf "%s\t%p\n" | grep -v -f ${NOBACKUP} > ${DUMPFILE}.size
SIZEONDISK=`cut -f1 ${DUMPFILE}.size | ${CALCSUM}` # Get the files sizes (on disk), and sum them
# Actual afio command (all the rest of this crap is just gravy!)
# We take the data set we collected in the find above, send it through
# grep to apply the NoBackup file, then give it to afio to backup.
# To remove all compression, nuke (here and below) the -Z, -E ${NOCOMPRESS}
# and -Q "-c${GZIPCOMPRESSION}"
# To remove the boring e-mail, nuke (here and below) the -@ ${EMAIL}
# -b is critical block size for DI-30! If you have memory problems, play
# with -c and -M
# -c 1024 = 32 Meg of I/O buffering, -M 10m = 10 Meg of GZip buffering!
# cat ${DUMPFILE}.name | grep -v -f ${NOBACKUP} | afio -ov -b 32k -c 1024 -M 10m -Z -Q "-c${GZIPCOMPRESSION}" -E ${NOCOMPRESS} -@ ${EMAIL} -L ${AFIOLOG} ${TAPEDEVICE}
cat ${DUMPFILE}.name | grep -v -f ${NOBACKUP} | afio -ov -b 32k -c 1024 -M 10m -Z -Q "-c${GZIPCOMPRESSION}" -E ${NOCOMPRESS} -L ${AFIOLOG} ${TAPEDEVICE}
# Misc. admin
rm -f ${DUMPFILE}.name ${DUMPFILE}.size
echo "`date`; FINISH: ${BUTYPE} backup to ${TAPENAME} (data: ${SIZEONDISK})..." | tee -a ${LOGFILE}
echo "`date`; START: Verify ${BUTYPE} ${TAPENAME}..." | tee -a ${LOGFILE}
cd / # Have to be in / or the verify will fail with "No such file or directory" (relative paths)
# We verify and create a "catalog" by redirecting STDOUT and STDERR to the
# catalog file
# The egrep -v in the next line works around a minor afio bug that prints
# "-- compressed" twice. That bug is fixed in my RPMs
# (http://www.jpsdomain.org/public/public.html#rpms).
# To remove all compression, nuke (here and above) the -Z and -E ${NOCOMPRESS}
# To remove the boring e-mail, nuke (here and above) the -@ ${EMAIL}
# -b is critical block size for DI-30! (Not sure if -c and -M help on verify,
# but...)
# -c 1024 = 32 Meg of I/O buffering, -M 10m = 10 Meg of GZip buffering!
# afio -rv -b 32k -c 1024 -M 10m -L ${AFIOLOG} -Z -E ${NOCOMPRESS} -@ ${EMAIL} ${TAPEDEVICE} 2>&1 | egrep -v "^ -- compressed$" | tee ${CATFILE}
afio -rv -b 32k -c 1024 -M 10m -L ${AFIOLOG} -Z -E ${NOCOMPRESS} ${TAPEDEVICE} 2>&1 | egrep -v "^ -- compressed$" | tee ${CATFILE}
# The egrep gets only digits, optionally surrounded by white space, since the cut gets other crap now and then
SIZEONTAPE=`cut -b 31-40 ${CATFILE} | egrep "^\W*[[:digit:]]+\W*$" | ${CALCSUM}` # Get the files sizes (on tape)
###### If we (finally ever) get to here, update the flag files that keep track
# of what we did. Since we don't update them before this, if we barfed above,
# the next job will pick up where we left off. That means you'd better notice
# and fix the problem and run it again before next week!
echo ${BUTYPE} > ${LASTTAPE} # Keep track of the last type of backup we did
if [ ${BUTYPE} = weekly ]; then # We did a weekly
echo ${THISTAPE} > ${LASTWEEKLY}
else # We did a monthly
echo ${THISTAPE} > ${LASTMONTHLY}
fi
# Retension the tape (FF to end, Rewind to beginning), then offline it.
# On a DI-30, offline does not eject the tape, but it seems to help make
# it pop out faster when you do manually hit the eject button.
mt retension
mt offline
# Set/re-set some permissions
chmod 600 ${LOGFILEPATH}*
chmod 600 ${EXFILEPATH}*
# Print exit message (note first line to console only, others to console and log)
echo "${STARTDATE}; START: ${BUTYPE} backup to ${TAPENAME}..."
echo "`date`; FINISH: Verify ${BUTYPE} ${TAPENAME}..." | tee -a ${LOGFILE}
echo "Data size on disk: ${SIZEONDISK}, size on tape (GZ ${GZIPCOMPRESSION}) ${SIZEONTAPE}." | tee -a ${LOGFILE}
echo "" | tee -a ${LOGFILE}
####################################################################
####################################################################
# Other notes
### To Restore (which is NOT automated here, but see my "tape" script)
# To restore everything from a compressed tape archive
# To overwrite, you need to be in the / dir. (see the -n option!)
# If you are someplace else, the dir. structure will be re-created.
# afio -ivxZ -b 32k -M 10m -L /var/log/jpbackup/jpbackup.log -@ root /dev/tape
# To restore just "/root" from a compressed tape archive
# To overwrite, you need to be in the / dir. (see the -n option!)
# If you are someplace else, the dir. structure will be re-created.
# Can handle leading / in the path.
# afio -ivxZ -b 32k -M 10m -L /var/log/jpbackup/jpbackup.log -@ root -y "/root/" /dev/tape
# jpbackup uses 8 tapes
# Monday_1, Monday_2, Monday_3 = 1st, 2nd, 3rd Monday of the month,
# backup important, dynamic stuff
# Month_1, Month_2 ... Month_5 = 4th Monday of 1st, 2nd .. 5th month,
# backup just about everything
# RedHat Vixi Cron entry to run every Monday at 4:20 am
# 20 4 * * Mon /opt/jpbackup/jpbackup
# afio options used (see 'man afio' for more information)
# afio -ov -b 32k -c 1024 -M 10m -Z -Q "-c${GZIPCOMPRESSION}" \
# -E ${NOCOMPRESS} -L ${AFIOLOG} -@ ${EMAIL} ${TAPEDEVICE}
# -o reads pathnames from the standard input and writes an archive
# -v verbose; report pathnames as they are processed, -t gives an ls -l style
# -b {size} read or write {size} archive blocks
# -c {count} buffer count archive blocks between I/O operations
# -M {size} specifies the maximum amount of memory to use for compression
# [default -M 2m]
# -Z gzip the files on the way out and in
# -Q "{opt}" pass the option {opt} to the compression program
# -E read non-compressible file extensions separated by white space,
# from {filename}
# -L {Log_file_path} the name of the file to log errors and the final totals to
# -@ {address}send email to address when the operation is complete
# /dev/tape & ntape are symlinks to the real /dev/ht0 & nht0 tape devices
# Other options (do not use -Q, even if used in backup creation, do need -b,
# also need -Z if -Z used in creation.
# -t reads an archive and writes a table-of-contents to the standard output
# -i installs the contents of an archive relative to the working directory
# Use -iy to do a partial or selective restore (E.G. -iy "root/stuff*")
# -n protect newer existing files (comparing file modification times)
# -x retain file ownership and setuid/setgid permissions (default
# for root)
# -r reads archive and verifies it against the filesystem.
# -W {filename} do not process files whose names match shell patterns in
# {filename}
# Note, we could have used -W instead of the "grep -v -f" above, but using
# the find/grep is easier to debug, because you can get to see the file list
# *before* afio gets it.
# Other Monthly method we could have used
# We used the one above because it catches more, even if non-standard
# directories are added to /
# find /bin /boot /dev /lib /misc /opt /sbin /usr /var/lib -print > ${DUMPFILE}
# cat ${DUMPFILE} | afio -ovxZ -b 32k -c 64 -M 10m -Q "-c9" -L ${LOGFILE} -E ${NOCOMPRESS} -@ ${EMAIL} /dev/tape
#!/bin/ksh
# calcsum -- takes integer input and calculates the sum
# http://www.jpsdomain.org/~vossenjp/linux/OnStream_DI-30+RedHat_Backup_mini-HOWTO.html
# v1.0 21-Feb-2001 JP Vossen
# Copyright 2001 JP Vossen
# This script is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
# In no event shall the author be liable for any damages whatsoever
# (including, without limitation, damages for loss of business profits,
# business interruption, loss of business information, or any other
# pecuniary loss) arising out of the use of or inability to use this script.
# NOTE: MUST use ksh (which on my Linux is a symlink to zsh) because
# sh, bash, bash2 can't add! At 2147483648 they start counting BACKWARDS.
TOT=0
while read N; do
let TOT=${TOT}+${N}
done
echo "${TOT}"
A generic front-end, so I don’t have to remember all the block size and other options.
#!/bin/sh
# tape -- simple shell for OnStream DI-30 tape drive under RedHat Linux 6.2
# http://www.jpsdomain.org/linux/OnStream_DI-30-RedHat_Backup_mini-HOWTO.html
# v0.9 27-Feb-2001 JP Vossen (jp@jpsdomain.org)
# v0.9.1 06-Jun-2001 JPV Added eject options
# v0.9.2 15-Jun-2001 JPV Was "tar -tvbf 64 /dev..." but should have been
# "tar tvb 64 -f /dev..." Thanks to David Burleigh for pointing this out.
MYVER="v0.9.2 15-Jun-2001"
# Copyright 2001 JP Vossen
# This script is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
# In no event shall the author be liable for any damages whatsoever
# (including, without limitation, damages for loss of business profits,
# business interruption, loss of business information, or any other
# pecuniary loss) arising out of the use of or inability to use this script.
# NOTES:
# 0. Requires afio if you use the afio options (duh!):
# http://www.jpsdomain.org/public/public.html#rpms.
# 1. Requires a symlink of /dev/tape and /dev/ntape to your tape device.
# e.g. for an OnStream DI-30:
# "ln -s /dev/ht0 /dev/tape && ln -s /dev/nht0 /dev/ntape"
# 2. Assumes (actually forces) only one archive per tape.
# 3. The DI-30 requires a block size of 32k in ALL operations.
# e.g. tar uses blocks of 512b, so 32k=512b*64
# 4. The parameters specified here will work for the DI-30. Your mileage
# may vary.
# 5. See my "jpbackup" script for much better afio code.
# 6. mt does not really need the "-f /dev/tape" if the TAPE environment
# variable is correctly set.
# See how we were called.
case "$1" in
tbackup)
if [ -z "$2" ]; then
echo ""
echo "Must specify a directory to backup!"
exit 2
fi
tar cvb 64 -f /dev/tape $2
;;
ttoc)
tar tvb 64 -f /dev/tape
;;
tverify)
cd /
tar dvb 64 -f /dev/tape
;;
trestore)
mt rewind
# e.g. tape trestore
# e.g. tape trestore root/mydoc*
tar xvb 64 -f /dev/tape $2
;;
backup)
if [ -z "$2" ]; then
echo ""
echo "Must specify a directory to backup!"
exit 2
fi
find $2 | afio -ovZ -b 32k -c 64 -M 10m -Q "-c6" /dev/tape
;;
toc)
afio -tvxZ -b 32k -M 10m /dev/tape
;;
verify)
cd /
afio -rvxZ -b 32k -M 10m /dev/tape
;;
restore)
mt rewind
if [ -z "$2" ]; then
# e.g. tape restore
afio -ivxZ -b 32k -M 10m /dev/tape
else
# e.g. tape restore root/mydoc*
afio -ivxZ -b 32k -M 10m -y "$2" /dev/tape
fi
;;
offline|eject)
echo ""
echo "Note: this will not eject the tape, but may help allow you to"
echo "manually eject it."
mt -f /dev/tape offline
;;
erase)
mt -f /dev/tape rewind
mt -f /dev/tape erase
;;
status)
mt -f /dev/tape status
;;
retension)
# This spelling is correct
mt -f /dev/tape retension
;;
retention)
echo ""
echo "Next time you may want to spell it retension..."
mt -f /dev/tape retension
;;
rewind)
mt -f /dev/tape rewind
;;
findnext)
if [ -z "$2" ]; then
echo ""
echo "Must specify the number of archives to skip!"
exit 2
fi
# Goto the start of the next archive.
# See mt man page: fsfm, bsf, bsfm, asf, eod, etc.
# Note NON-rewinding device...
mt -f /dev/ntape fsf $2
;;
end)
# Goto the end of the existing data.
# Note NON-rewinding device...
mt -f /dev/ntape eod
;;
*)
echo "$0 ${MYVER} Copyright 2001 JP Vossen"
echo " http://www.jpsdomain.org/"
echo " Licensed under the GNU GENERAL PUBLIC LICENSE:"
echo " See http://www.gnu.org/copyleft/gpl.html for full text and details."
echo ""
echo "Usage: $0 [tbackup [dir]|ttoc|tverify|trestore [dir]] - using tar"
echo " $0 [backup [dir]|toc|verify|restore [dir]] - using afio"
echo " $0 [offline|eject|erase|status|retension|rewind|findnext|end] - tape device"
echo ""
echo "Notes:"
echo "1. You need to be in / for a verify to work ($0 does this for you)..."
echo "2. If you are in / a restore will over-write the existing"
echo " directory. If not in /, a new structure will be created."
echo "3. findnext will find the next archive on the tape, but this"
echo " script will never create more than one archive per tape."
echo "4. You should really read this code and understand what it"
echo " does before trying to use it."
echo ""
exit 1
;;
esac
#!/bin/sh
# RDisk - Rescue Disk: use "mkbootdisk" to make a rescue disk for this system
# an gather other useful information.
# http://www.jpsdomain.org/~vossenjp/linux/OnStream_DI-30+RedHat_Backup_mini-HOWTO.html
# v1.0 08-Nov-1998 JPV
# v1.3 28-Nov-1999 JPV Updated for new Disk (hdc1)
# v1.4 21-Sep-2000 JPV, updated for new RH 6.2 Kernel
# v1.5 21-Sep-2000 JPV, added "fdisk.info" stuff
# v1.6 16-Feb-2001 JPV, added "rpm -qa" stuff, updated kernel
# v1.7 20-Feb-2001 JPV, added nofpy parameter and variables for cmd paths
# Copyright 2001 JP Vossen
# This script is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
# In no event shall the author be liable for any damages whatsoever
# (including, without limitation, damages for loss of business profits,
# business interruption, loss of business information, or any other
# pecuniary loss) arising out of the use of or inability to use this script.
# NOTES:
# 1. This script requires that you have a working floppy drive, with an
# appropriate /etc/fstab entry. My /etc/fstab has this:
# /dev/fd0 /mnt/floppy_ext2 ext2 noauto,owner 0 0
# /dev/fd0 /mnt/floppy_dos vfat noauto,owner 0 0
# I have symlinks /a -> /mnt/floppy_dos and /fpy -> /mnt/floppy_ext2
# This lets people who expect to find stuff in /mnt be OK, but is easier
# and gives me more control over floppy mounts. Since your system is
# probably different, you'll have to edit the mount/umount lines.
# 2. Every time you update the kernel, you have to update the mkbootdisk
# line too. See /lib/modules/
# 3. I only grab fdisk -l info from /dev/hda, if you have more hard
# drives, add them too.
# 4. You can do everything but the floppy stuff by "$0 nofpy"
# 5. Since this may not always run "su -" I have included paths. You'll
# have to adjust them if your system is different.
MKBOOTDISK=/sbin/mkbootdisk
FDISK=/sbin/fdisk
MKKICKSTART=/usr/sbin/mkkickstart
if [ -z "$1" ]; then
echo ""
echo Making boot disk...
# mkbootdisk --device /dev/fd0 2.2.14-5.0smp
# mkbootdisk --device /dev/fd0 2.2.16-3smp
${MKBOOTDISK} --device /dev/fd0 2.2.18
fi
# Gather some disk info
date > /root/rdisk/fdisk.info
echo Getting fdisk info...
echo "" >> /root/rdisk/fdisk.info
echo "fdisk -l /dev/hda" >> /root/rdisk/fdisk.info
echo "-----------------" >> /root/rdisk/fdisk.info
echo "" >> /root/rdisk/fdisk.info
${FDISK} -l /dev/hda >> /root/fdisk.info
echo Adding df -h...
echo "" >> /root/rdisk/fdisk.info
echo "df -h" >> /root/rdisk/fdisk.info
echo "-----" >> /root/rdisk/fdisk.info
df -h >> /root/rdisk/fdisk.info
echo Adding fstab...
echo "" >> /root/rdisk/fdisk.info
echo "/etc/fstab" >> /root/rdisk/fdisk.info
echo "----------" >> /root/rdisk/fdisk.info
cat /etc/fstab >> /root/rdisk/fdisk.info
echo "" >> /root/rdisk/fdisk.info
# Create KickStart info too, can use to rebuild from CD-ROM
echo Making kickstart file...
${MKKICKSTART} > /root/rdisk/kickstart.info
# Create RPM Info
echo Getting RPM info...
date > /root/rdisk/rpm-qa.info
echo "" >> /root/rdisk/rpm-qa.info
rpm -qa >> /root/rdisk/rpm-qa.info
if [ -z "$1" ]; then
echo Copying files to floppy...
# Copy the fdisk.info & kickstart files to the floppy
mount /fpy
cp /root/rdisk/*.info /fpy
umount /fpy
fi
echo ""
echo "$0 Finished..."
echo ""
This is never used in any of the other scripts here, and is included only as a convenience.
#!/bin/sh
# configbackup -- copies various important files to some specified backup location
# http://www.jpsdomain.org/~vossenjp/linux/OnStream_DI-30+RedHat_Backup_mini-HOWTO.html
# v1.0 25-Feb-2001 JP Vossen
# From CDBackup.bat and Srv-Bk.bat
# Copyright 2001 JP Vossen
# This script is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
# In no event shall the author be liable for any damages whatsoever
# (including, without limitation, damages for loss of business profits,
# business interruption, loss of business information, or any other
# pecuniary loss) arising out of the use of or inability to use this script.
if [ -z $1 ]; then
echo ""
echo "Usage: $0 {destination directory}"
echo ""
echo " Copies various important files to {dest} as backups."
echo ""
exit 1
fi
# Could do other interesting things here, like making the destination dir:
# Mon, Tue, Wed, Jan, Feb, Mar, week number (+%W) etc.
# Use the date command, like DESTDIR=/root/`date +%a` or `date +%b`
# Then disable the usage above
DESTDIR=${1%%/} # Remove any trailing /
# Output is suitable for redirection into a log file, if you care...
echo "Starting $0 on `date`"
echo "Copy Web Server stuff"
# cp -fPR /home/home/httpd/ ${DESTDIR}/Web/ # This gets the Apache manual too.
cp -fPR /home/httpd/cgi-bin/ ${DESTDIR}
echo "Copy Specific, individual config files"
cp -vfP /etc/HOSTNAME ${DESTDIR}
cp -vfP /etc/MACHINE.SID ${DESTDIR}
cp -vfP /etc/TextConfig ${DESTDIR}
cp -vfP /etc/X11/XF86Config ${DESTDIR}
cp -vfP /etc/aliases ${DESTDIR}
cp -vfP /etc/aliases.db ${DESTDIR}
cp -vfP /etc/at.deny ${DESTDIR}
cp -vfP /etc/auto.master ${DESTDIR}
cp -vfP /etc/auto.misc ${DESTDIR}
cp -vfP /etc/bashrc ${DESTDIR}
cp -vfP /etc/comanche.conf ${DESTDIR}
cp -vfP /etc/conf.linuxconf ${DESTDIR}
cp -vfP /etc/conf.modules ${DESTDIR}
cp -vfP /etc/crontab ${DESTDIR}
cp -vfP /etc/csh.cshrc ${DESTDIR}
cp -vfP /etc/dhcpd.conf ${DESTDIR}
cp -vfP /etc/dosemu.conf ${DESTDIR}
cp -vfP /etc/dosemu.users ${DESTDIR}
cp -vfP /etc/dumpdates ${DESTDIR}
cp -vfP /etc/exports ${DESTDIR}
cp -vfP /etc/fstab ${DESTDIR}
cp -vfP /etc/ftpaccess ${DESTDIR}
cp -vfP /etc/ftpconversions ${DESTDIR}
cp -vfP /etc/ftpgroups ${DESTDIR}
cp -vfP /etc/ftphosts ${DESTDIR}
cp -vfP /etc/ftpusers ${DESTDIR}
cp -vfP /etc/gettydefs ${DESTDIR}
cp -vfP /etc/group ${DESTDIR}
cp -vfP /etc/host.conf ${DESTDIR}
cp -vfP /etc/hosts ${DESTDIR}
cp -vfP /etc/hosts.allow ${DESTDIR}
cp -vfP /etc/hosts.deny ${DESTDIR}
cp -vfP /etc/inetd.conf ${DESTDIR}
cp -vfP /etc/inittab ${DESTDIR}
cp -vfP /etc/inputrc ${DESTDIR}
cp -vfP /etc/isapnp.gone ${DESTDIR}
cp -vfP /etc/issue ${DESTDIR}
cp -vfP /etc/issue.net ${DESTDIR}
cp -vfP /etc/lilo.conf ${DESTDIR}
cp -vfP /etc/lmhosts ${DESTDIR}
cp -vfP /etc/login.defs ${DESTDIR}
cp -vfP /etc/logrotate.conf ${DESTDIR}
cp -vfP /etc/lynx.cfg ${DESTDIR}
cp -vfP /etc/mail.rc ${DESTDIR}
cp -vfP /etc/man.config ${DESTDIR}
cp -vfP /etc/mtab ${DESTDIR}
cp -vfP /etc/named.boot ${DESTDIR}
cp -vfP /etc/named.conf ${DESTDIR}
cp -vfP /etc/networks ${DESTDIR}
cp -vfP /etc/ntp.conf ${DESTDIR}
cp -vfP /etc/nwserv.conf ${DESTDIR}
cp -vfP /etc/nwserv.stations ${DESTDIR}
cp -vfP /etc/pam.conf ${DESTDIR}
cp -vfP /etc/passwd ${DESTDIR}
cp -vfP /etc/profile ${DESTDIR}
cp -vfP /etc/protocols ${DESTDIR}
cp -vfP /etc/pwdb.conf ${DESTDIR}
cp -vfP /etc/redhat-release ${DESTDIR}
cp -vfP /etc/resolv.conf ${DESTDIR}
cp -vfP /etc/rmt ${DESTDIR}
cp -vfP /etc/rpc ${DESTDIR}
cp -vfP /etc/screenrc ${DESTDIR}
cp -vfP /etc/securetty ${DESTDIR}
cp -vfP /etc/sendmail.cf ${DESTDIR}
cp -vfP /etc/sendmail.cw ${DESTDIR}
cp -vfP /etc/sendmail.mc ${DESTDIR}
cp -vfP /etc/services ${DESTDIR}
cp -vfP /etc/shadow ${DESTDIR}
cp -vfP /etc/shells ${DESTDIR}
cp -vfP /etc/smb.conf ${DESTDIR}
cp -vfP /etc/smbpasswd ${DESTDIR}
cp -vfP /etc/smbusers ${DESTDIR}
cp -vfP /etc/snmpd.agentinfo ${DESTDIR}
cp -vfP /etc/snmpd.conf ${DESTDIR}
cp -vfP /etc/squid.conf ${DESTDIR}
cp -vfP /etc/squid.conf.default ${DESTDIR}
cp -vfP /etc/syslog.conf ${DESTDIR}
cp -vfP /etc/tw.config ${DESTDIR}
cp -vfP /etc/wgetrc ${DESTDIR}
echo "Copy all rc and .conf files"
cp -vfP /etc/*rc ${DESTDIR}
cp -vfP /etc/*.rc* ${DESTDIR}
cp -vfP /etc/*.conf ${DESTDIR}
echo "Copy rc.d files"
cp -vfPR /etc/rc.d/* ${DESTDIR}
echo "Copy config files for log rotations"
cp -vfP /etc/logrotate.d/* ${DESTDIR}
echo "Copy security files"
cp -vfPR /etc/security/* ${DESTDIR}
echo "Copy DNS files (chrooted)"
cp -vfPR /home/dns/var/named/* ${DESTDIR}
echo "Finished $0 on `date`"
See the following URLs for lists of Linux backup tools. Some of these tools are free, some are commercial, and some are in-between. I’m only going to talk about free ones.
The list below is of tools I’ve found to out there and interesting looking. I use afio, but I’m not making any recommendations that you should too. Just look at the list. I’ve quoted http://www.linux.org/apps/all/Administration/Backup.html and the home pages of some of the tools quite a bit (read, I stole their descriptions). That text remains the property of its respective owner.
One interesting issue is that of relative paths. Older UNIX tar commands stored absolute paths (e.g. / home/user/mystuff by default. This is bad, because you can only restore to the exact same directory you backed up from. You may not always want to do that. Most GNU tools use relative paths (home/user/mystuff) so you can restore wherever you want. The downside is that unless you are in the root of the filesystem when you do a verify, it will fail, because it will use a relative path to find the files to compare with the tape and it won’t find them. For example, if you are in /home/user, trying to verify a backup of your home directory, the tape software will be looking for /home/user/home/user, which is probably not there. The moral of the story is, “cd /” before doing a verify.
The same goes for restores, except you might actually want to do this. There are often time when I need to restore /home/user, but I do not want to actually mess with /home/user, I just want a part of it. One solution is to do a partial restore. The other is to restore to the relative path, get what you need, then nuke the rest. Remember, this is only with the newer, usually GNU versions. The “traditional” tar does not work this way.
http://man-pages.net/linux/man1/tar.1.html
http://www2.linuxjournal.com/lj-issues/issue22/1216.html
Tar is the most widely known UNIX backup tool. It stands for Tape ARchive and does not have to actually use tape. You have almost certainly seen a .tar, .tar.Z or .tgz file. These all use tar. It has some problem though. Most notably, IMHO, it compresses the entire archive, so if tape is damaged, entire archive lost. That’s a bit of a problem. So I don’t like tar, but you pretty much have to know about it anyway.
|
Operation |
Command |
Comments |
|
Full Backup |
tar cvb 64 –f /dev/tape / |
|
|
Full Restore |
tar xvb 64 -f /dev/tape |
|
|
Partial Backup |
tar cvb 64 –f /dev/tape {directories} |
See the man page about how tar deals with directory selection (note I did not say file selection). |
|
Partial Restore |
tar xvb 64 –f /dev/tape {directories} |
Ditto. |
|
Verify |
tar dvb 64 –f /dev/tape |
This will fail unless you are in the root directory – relative paths. |
|
Table of Contents |
tar tvb 64 –f /dev/tape |
|
http://man-pages.net/linux/man8/dump.8.html
Next to tar, dump is another of the most widely known tools. As far as I know, it does not do compression at all. It uses “levels” from 0 to 9 to determine what to backup. You can create very complex and convoluted schemes to backup different things at different times. As I said above, thinking about this stuff gives me a headache.
“The dump package contains both dump and restore. Dump examines files in a filesystem, determines which ones need to be backed up, and copies those files to a specified disk, tape or other storage medium. The restore command performs the inverse function of dump; it can restore a full backup of a filesystem. Subsequent incremental backups can then be layered on top of the full backup. Single files and directory subtrees may also be restored from full or partial backups.”
http://man-pages.net/linux/man1/cpio.1.html
cpio is that last of the big three most widely known UNIX backup tools. It’s interface is a bit different than tar or dump, in that it must be used as a filter (e.g. “find / -print | cpio -ov --block-size=64 -C 32768 >/dev/ht0”). It also suffers from the same compression issues as tar.
http://www.frognet.net/help/manpages/docs/afio.html
http://www.linux.org/apps/AppId_266.html
http://metalab.unc.edu/pub/linux/system/backup/afio-2.4.6.tgz
“Afio makes cpio-format archives. It deals somewhat gracefully with input data corruption, supports multi-volume archives during interactive operation, and can make compressed archives that are much safer than compressed tar or cpio archives. Afio is best used as an `archive engine' in a backup script.”
I like afio a lot. It works well with the DI-30, and I can script it to just exactly what I want. It is used as a filter, the same as cpio, and in fact uses the cpio format (as do RPMs). See my scripts in the appendix.
The following examples are all very simple, and use gzip compression. Unlike tar or cpio, afio compresses each file, rather than the entire archive. That means if you have a media error, only the data where the error is are lost, instead of the entire archive.
|
Operation |
Command |
Comments |
|
Full Backup |
find / -print | afio -ovZ -b 32k /dev/tape |
|
|
Full Restore |
afio -ivZ -b 32k /dev/tape |
Relative paths! |
|
Partial Backup |
find /home/user -print | afio -ovZ -b 32k /dev/tape |
|
|
Partial Restore |
afio -ivZ -b 32k –y “home/user/*” /dev/tape |
Relative paths! |
|
Verify |
afio -rvZ -b 32k /dev/tape |
Relative paths! |
|
Table of Contents |
afio -tvZ -b 32k /dev/tape |
|
http://www.linux.org/apps/AppId_302.html
http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/man/star.html
“Star is able to make backups with more than 12MB/s if the disk and tape drive support such a speed. This is more than double the speed that ufsdump will get. Star performs 13.5 MB/s with a recent DLT tape drive while ufsdump gets a maximum speed of about 6MB/s with the same hardware. Star development started 1982, development is still in progress although it is stable to use.”
See the tar command reference.
http://www.linux.org/apps/AppId_303.html
http://www.e-survey.net.au/taper/
http://www2.linuxjournal.com/lj-issues/issue22/1216.html
“Taper is a tape backup and restore program that provides a friendly user interface to allow backing/restoring files to a tape drive. Alternatively, files can be backed up to hard disk files. Selecting files for backup and restore is very similar to the Midnight Commander interface and allows easy traversal of directories. Recursively selected directories are supported. Incremental backup and automatic most recent restore are defaults settings. SCSI, ftape, zftape, and removable drives are supported.”
Note the last line. Taper was developed for ftape (floppy tapes, like the QIC series drives). It is not recommended for use with OnStream drives. If you know different – let me know!
http://kbackup.sourceforge.net/
“KBackup is a backup program for UNIX machines. It supports any OS supported tape drive. It can use tar or afio to create the archives. It can even compress using gzip. It supports include lists, exclude lists, and even backing up to a file.”
“KBackup is an easy-to-use backup package for Unix. It was originally written by Karsten Balluder. Currently, it's development has stagnated, and several fixes are needed. The main mailing-list for KBackup is in egroups (www.egroups.com).”
http://www.bru.com/bruinfo.php
OK, I lied. I said I would only talk about free programs, and BRU is not free. But it’s probably the most popular backup system for small Linux systems, so…
“BRU Backup & Restore Utility features data-verified backups, scalability, configurability, and ease of use, for functionality with Linux and UNIX.”
Please note that you MUST use a 32k blocksize when writing to the DI30 drive. Also note that the tar statement uses "-b 64" due to its 512 blocksize e.g. “bru -cvvf /dev/ht0 -b 32k /home”. Get a complete BRU configuration file for this drive from http://www.estinc.com/downloads/brutabs/adr.bt
The following are references to useful material on the Web. All the various links I’ve used above are here again, along with a bunch of other neat material. The manual (man) pages are provided in case you do not have access to a Linux machine to get the details, and because they are easier to read and print out.
|
Description |
Site |
|
My other Linux information |
|
|
My afio RPM |
|
|
My afio RPM |
http://www.jpsdomain.org/public/rpms/jpbackup-0.9.2-1.i386.rpm http://www.jpsdomain.org/public/rpms/jpbackup-0.9.2-1.src.rpm |
|
|
|
|
OnStream Linux Support |
|
|
OLD, but FYI: OnStream ADR-30 Tape Problems Have Been Identified |
|
|
Important Information For OnStream DI30 Drive Users (Installation details and disclaimer) |
|
|
Tar and Taper for Linux |
|
|
Backing Up In Linux |
|
|
The Tape Guy’s Questions and Answer Message Board (also the rest of the site) |
|
|
OSST tester's page (A new driver for OnStream tape drives) |
|
|
|
|
|
Linux 2.2.18 kernel and IDE patch for same |
ftp://ftp.us.kernel.org/pub/linux/kernel/v2.2/linux-2.2.18.tar.gz ftp://ftp.us.kernel.org/pub/linux/kernel/people/hedrick/ide-2.2.18/ide.2.2.18.1221.patch.gz |
|
OnStream Drive Firmware |
|
|
Installation Instructions |
http://www.onstreamdata.com/support/linux/di30_patch.html http://www.onstreamdata.com/support/linux/linux_kernel216_rebuild.html |
|
|
|
|
The Linux System Administrator’s Guide: Backups and ToC. |
|
|
Linux Administration Made Easy: Backups and ToC. |
http://www.linuxdoc.org/LDP/lame/LAME/linux-admin-made-easy/c1315.html http://www.linuxdoc.org/LDP/lame/LAME/linux-admin-made-easy/book1.html |
|
The Linux Network Administrator’s Guide: ToC. |
|
|
UNIX Backup and Recovery By Preston, Curtis W. (Especially check the ToC) |
http://www.oreilly.com/catalog/unixbr/ http://www1.fatbrain.com/asp/bookinfo/bookinfo.asp?theisbn=1565926420 |
|
RedHat 6.2 HCL re: OnStream (kind of wrong) |
http://www.redhat.com/support/hardware/intel/62/rh6.2-hcl-i.ld-7.html |
|
OLD: Torture-testing Backup and Archive Programs: Things You Ought to Know But Probably Would Rather Not |
|
|
The Filesystem Hierarchy Standard |
http://www.jpsdomain.org/linux/OnStream_DI-30-RedHat_Backup_mini-HOWTO.html
http://www.jpsdomain.org/public/tools/jpbackup-0.9.tgz
|
Ver |
Date |
Comment |
|
V0.9 |
27-Feb-2001 |
DRAFT: First public release, so various technical reviewers can access it. |
|
V0.9.1 |
02-Jun-2001 |
Minor corrections for typos, etc. The script itself needs work, and I need to do more testing with RedHat 7.1 before the “public” release. |
|
V0.9.2 |
16-Jun-2001 |
Corrected a bug with all tar examples. Was “tar -tvbf 64 /dev...” but should have been “tar tvb 64 –f /dev…” Also, changed “www.onstream.com” to www.onstreamdata.com. Thanks to David Burleigh for pointing those out. Also other minor corrects to docs. |
|
V1.0 |
|
Will be first general public release. |