Index: openacs-4/packages/acs-core-docs/www/backup-recovery.html =================================================================== RCS file: /usr/local/cvsroot/openacs-4/packages/acs-core-docs/www/backup-recovery.html,v diff -u -r1.16 -r1.17 --- openacs-4/packages/acs-core-docs/www/backup-recovery.html 19 Nov 2003 15:44:49 -0000 1.16 +++ openacs-4/packages/acs-core-docs/www/backup-recovery.html 11 Dec 2003 23:08:45 -0000 1.17 @@ -7,267 +7,201 @@ recovery are always risky; here are some steps that minimize the chance recovery is necessary:
- Store everything on a fault-tolerant disk array (RAID 1 or 5 - or better). -
- Use battery backup. -
- Use more reliable hardware, such as SCSI instead of IDE. -
These steps improve the chances of successful recovery:
- Store backups on a third disk on another controller -
- Store backups on a different computer on a different network - in a different physical location. (Compared to off-line - backup such as tapes and CDRs, on-line backup is faster and - more likely to succeed, but requires maintenance of another machine.) -
- Plan and configure for recovery from the beginning. -
- Test your recovery strategy from time to time. -
- Make it easy to maintain and test your recovery strategy, so - that you are more likely to do it. -
- OpenACS installations comprise files and database contents. - If you follow the reference install and put all files, - including configuration files, in - /var/lib/aolserver/service0/, - and back up the database nightly to a file in - /var/lib/aolserver/service0/database-backup, - then you can apply standard file-based backup strategies to - /var/lib/aolserver/service0 -
This section describes how to make a one-time backup and - restore of the files and database. This is useful for rolling back to - known-good versions of a service, such as at initial - installation and just before an upgrade. First, you back up the - database to a file within the file tree. Then, you back up the - file tree. All of the information needed to rebuild the site, - including the AOLserver config files, is then in tree for - regular file system backup.
Back up the database to a file.�
Oracle.�
- Download the backup script. Save the file export-oracle.txt as - /tmp/export-oracle.txt -
- Login as root. The following commands will install the export script: -
-joeuser:~$ su - -Password: *********** -root:~# cp /tmp/export-oracle.txt /usr/sbin/export-oracle -root:~# chmod 700 /usr/sbin/export-oracle
- Setup the export directory; this is the directory where backups will - be stored. We recommend the directory - /ora8/m02/oracle-exports.
-root:~# mkdir /ora8/m02/oracle-exports -root:~# chown oracle.dba /ora8/m02/oracle-exports -root:~# chmod 770 /ora8/m02/oracle-exports
- Now edit - /usr/sbin/export-oracle and - change the SERVICE_NAME and - DATABASE_PASSWORD fields to - their correct values. If you want to use a directory other than - /ora8/m02/oracle-exports, you - also need to change the - exportdir setting. -
- Test the export procedure by running the command: -
-root:~# /usr/sbin/export-oracle -mv: /ora8/m02/oracle-exports/oraexport-service_name.dmp.gz: No such file or directory + Store everything on a fault-tolerant disk array (RAID 1 or 5 + or better). +
+ Use battery backup. +
+ Use more reliable hardware, such as SCSI instead of IDE. +
These steps improve the chances of successful recovery:
+ Store backups on a third disk on another controller +
+ Store backups on a different computer on a different network + in a different physical location. (Compared to off-line + backup such as tapes and CDRs, on-line backup is faster and + more likely to succeed, but requires maintenance of another machine.) +
+ Plan and configure for recovery from the beginning. +
+ Test your recovery strategy from time to time. +
+ Make it easy to maintain and test your recovery strategy, so + that you are more likely to do it. +
+ OpenACS installations comprise files and database contents. + If you follow the reference install and put all files, + including configuration files, in + /var/lib/aolserver/service0/, + and back up the database nightly to a file in + /var/lib/aolserver/service0/database-backup, + then you can apply standard file-based backup strategies to + /var/lib/aolserver/service0 +
This section describes how to make a one-time backup and + restore of the files and database. This is useful for rolling back to + known-good versions of a service, such as at initial + installation and just before an upgrade. First, you back up the + database to a file within the file tree. Then, you back up the + file tree. All of the information needed to rebuild the site, + including the AOLserver config files, is then in tree for + regular file system backup.
Back up the database to a file.�
Oracle.�
+ Download the backup script. Save the file export-oracle.txt as + /tmp/export-oracle.txt +
+ Login as root. The following commands will install the export script: +
+ joeuser:~$ su - + Password: *********** + root:~# cp /tmp/export-oracle.txt /usr/sbin/export-oracle + root:~# chmod 700 /usr/sbin/export-oracle
+ Setup the export directory; this is the directory where backups will + be stored. We recommend the directory + /ora8/m02/oracle-exports.
+ root:~# mkdir /ora8/m02/oracle-exports + root:~# chown oracle.dba /ora8/m02/oracle-exports + root:~# chmod 770 /ora8/m02/oracle-exports
+ Now edit + /usr/sbin/export-oracle and + change the SERVICE_NAME and + DATABASE_PASSWORD fields to + their correct values. If you want to use a directory other than + /ora8/m02/oracle-exports, you + also need to change the + exportdir setting. +
+ Test the export procedure by running the command: +
+ root:~# /usr/sbin/export-oracle + mv: /ora8/m02/oracle-exports/oraexport-service_name.dmp.gz: No such file or directory -Export: Release 8.1.6.1.0 - Production on Sun Jun 11 18:07:45 2000 + Export: Release 8.1.6.1.0 - Production on Sun Jun 11 18:07:45 2000 -(c) Copyright 1999 Oracle Corporation. All rights reserved. + (c) Copyright 1999 Oracle Corporation. All rights reserved. -Connected to: Oracle8i Enterprise Edition Release 8.1.6.1.0 - Production -With the Partitioning option -JServer Release 8.1.6.0.0 - Production -Export done in US7ASCII character set and US7ASCII NCHAR character set -. exporting pre-schema procedural objects and actions -. exporting foreign function library names for user SERVICE_NAME -. exporting object type definitions for user SERVICE_NAME -About to export SERVICE_NAME's objects ... -. exporting database links -. exporting sequence numbers -. exporting cluster definitions -. about to export SERVICE_NAME's tables via Conventional Path ... -. exporting synonyms -. exporting views -. exporting stored procedures -. exporting operators -. exporting referential integrity constraints -. exporting triggers -. exporting indextypes -. exporting bitmap, functional and extensible indexes -. exporting posttables actions -. exporting snapshots -. exporting snapshot logs -. exporting job queues -. exporting refresh groups and children -. exporting dimensions -. exporting post-schema procedural objects and actions -. exporting statistics -Export terminated successfully without warnings.
PostGreSQL.�Create a backup file and verify that it was created and has a reasonable size (several megabytes).
[root@localhost root]# su - service0
-[service0@localhost service0]$ pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp service0
-[service0@localhost service0]$ ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp
--rw-rw-r-x 1 service0 service0 4005995 Feb 21 18:28 /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp
-[service0@localhost service0]$ exit
-[root@localhost root]#
-su - service0
-pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp openacs-dev
-ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp
-exit
Back up the file system.�Back up all of the files in the service, including the - database backup file but excluding the auto-generated - supervise directory, which is - unneccesary and has complicated permissions.
In the tar command,
c create a - new tar archive
p preserves permissions.
s preserves file sort order
j compresses the output with bz2.
The --exclude clauses skips some daemontools files that - are owned by root and thus cannot be backed up by the - service owner. These files are autogenerated and we don't - break anything by omitting them.
The --file clause - specifies the name of the output file to be generated; we - manually add the correct extensions.
The last clause, - /var/lib/aolserver/service0/, - specifies the starting point for backup. Tar defaults to - recursive backup.
[root@yourserver root]# su - service0 -[service0@yourserver service0]$ tar -cpsj --exclude /var/lib/aolserver/service0/etc/daemontools/supervise --file /tmp/service0-backup.tar.bz2 /var/lib/aolserver/service0/ -tar: Removing leading `/' from member names -[service0@yourserver service0]$
Suffer a catastrophic failure on your production system.�(We'll simulate this step)
[root@yourserver root]# svc -d /service/service0
-[root@yourserver root]# mv /var/lib/aolserver/service0/ /var/lib/aolserver/service0.lost
-[root@yourserver root]# rm /service/service0
-rm: remove symbolic link `/service/service0'? y
-[root@yourserver root]# ps -auxw | grep service0
-root 1496 0.0 0.0 1312 252 ? S 16:58 0:00 supervise service0
-[root@yourserver root]# kill 1496
-[root@yourserver root]# ps -auxw | grep service0
-[root@yourserver root]# su - postgres
-[postgres@yourserver pgsql]$ dropdb service0
-DROP DATABASE
-[postgres@yourserver pgsql]$ dropuser service0
-DROP USER
-[postgres@yourserver pgsql]$ exit
-logout
-[root@yourserver root]#
Recovery.�
Restore the operating system and required software. - You can do this with standard backup processes or by - keeping copies of the install material (OS CDs, OpenACS - tarball and supporting software) and repeating the install - guide. Recreate the service user (service0).
Restore the OpenACS files and database backup file.
[root@yourserver root]# su - service0 -[service0@yourserver service0]$ cd /var/lib/aolserver -[service0@yourserver aolserver]$ tar xjf /tmp/service0-backup.tar.bz2 -[service0@yourserver aolserver]$ chmod -R 775 service0 -[service0@yourserver aolserver]$ chown -R service0.web service0 + Connected to: Oracle8i Enterprise Edition Release 8.1.6.1.0 - Production + With the Partitioning option + JServer Release 8.1.6.0.0 - Production + Export done in US7ASCII character set and US7ASCII NCHAR character set + . exporting pre-schema procedural objects and actions + . exporting foreign function library names for user SERVICE_NAME + . exporting object type definitions for user SERVICE_NAME + About to export SERVICE_NAME's objects ... + . exporting database links + . exporting sequence numbers + . exporting cluster definitions + . about to export SERVICE_NAME's tables via Conventional Path ... + . exporting synonyms + . exporting views + . exporting stored procedures + . exporting operators + . exporting referential integrity constraints + . exporting triggers + . exporting indextypes + . exporting bitmap, functional and extensible indexes + . exporting posttables actions + . exporting snapshots + . exporting snapshot logs + . exporting job queues + . exporting refresh groups and children + . exporting dimensions + . exporting post-schema procedural objects and actions + . exporting statistics + Export terminated successfully without warnings.
PostgreSQL.�Create a backup file and verify that it was created and has a reasonable size (several megabytes).
[root@localhost root]# su - service0 + [service0@localhost service0]$ pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp service0 + [service0@localhost service0]$ ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp + -rw-rw-r-x 1 service0 service0 4005995 Feb 21 18:28 /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp + [service0@localhost service0]$ exit + [root@localhost root]# + su - service0 + pg_dump -f /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp openacs-dev + ls -al /var/lib/aolserver/service0/database-backup/before_upgrade_to_4.6.dmp + exit
Back up the file system.�Back up all of the files in the service, including the + database backup file but excluding the auto-generated + supervise directory, which is + unneccesary and has complicated permissions.
In the tar command,
c create a + new tar archive
p preserves permissions.
s preserves file sort order
j compresses the output with bz2.
The --exclude clauses skips some daemontools files that + are owned by root and thus cannot be backed up by the + service owner. These files are autogenerated and we don't + break anything by omitting them.
The --file clause + specifies the name of the output file to be generated; we + manually add the correct extensions.
The last clause, + /var/lib/aolserver/service0/, + specifies the starting point for backup. Tar defaults to + recursive backup.
[root@yourserver root]# su - service0 + [service0@yourserver service0]$ tar -cpsj --exclude /var/lib/aolserver/service0/etc/daemontools/supervise --file /tmp/service0-backup.tar.bz2 /var/lib/aolserver/service0/ + tar: Removing leading `/' from member names + [service0@yourserver service0]$
Suffer a catastrophic failure on your production system.�(We'll simulate this step)
[root@yourserver root]# svc -d /service/service0
+ [root@yourserver root]# mv /var/lib/aolserver/service0/ /var/lib/aolserver/service0.lost
+ [root@yourserver root]# rm /service/service0
+ rm: remove symbolic link `/service/service0'? y
+ [root@yourserver root]# ps -auxw | grep service0
+ root 1496 0.0 0.0 1312 252 ? S 16:58 0:00 supervise service0
+ [root@yourserver root]# kill 1496
+ [root@yourserver root]# ps -auxw | grep service0
+ [root@yourserver root]# su - postgres
+ [postgres@yourserver pgsql]$ dropdb service0
+ DROP DATABASE
+ [postgres@yourserver pgsql]$ dropuser service0
+ DROP USER
+ [postgres@yourserver pgsql]$ exit
+ logout
+ [root@yourserver root]#
Restore the operating system and required software. + You can do this with standard backup processes or by + keeping copies of the install material (OS CDs, OpenACS + tarball and supporting software) and repeating the install + guide. Recreate the service user (service0).
Restore the OpenACS files and database backup file.
[root@yourserver root]# su - service0 + [service0@yourserver service0]$ cd /var/lib/aolserver + [service0@yourserver aolserver]$ tar xjf /tmp/service0-backup.tar.bz2 + [service0@yourserver aolserver]$ chmod -R 775 service0 + [service0@yourserver aolserver]$ chown -R service0.web service0 -
Restore the database
Oracle.�
Set up a clean Oracle database user and - tablespace (more information).
Invoke the import command
imp service0/service0 FILE=/var/lib/aolserver/service0/database-backup/nighty_backup.dmp
Postgres.�
Because of a bug in Postgres backup-recovery, database objects are not guaranteed to be created in the right order. To compensate, we pre-creating some critical items first, which leads to some harmless errors.
[root@yourserver root]# su - postgres -[postgres@yourserver pgsql]$ createuser service0 -Shall the new user be allowed to create databases? (y/n) y -Shall the new user be allowed to create more new users? (y/n) y -CREATE USER -[service0@yourserver web]$ createdb service0 -CREATE DATABASE -[service0@yourserver web]$ psql -f /var/lib/aolserver/service0/packages/acs-kernel/sql/postgresql/postgresql.sql service0 -(many lines omitted) -[service0@yourserver web]$ psql service0 < /var/lib/aolserver/service0/database-backup/database-backup.dmp -(many lines omitted) -[service0@yourserver web]$ exit -[postgres@yourserver pgsql]$ exit -logout -
Activate the service
[root@yourserver root]# ln -s /var/lib/aolserver/service0/etc/daemontools /service/service0 -[root@yourserver root]# sleep 10 -[root@yourserver root]# svgroup web /service/service0 -[root@yourserver root]#
Backup can encompass all files in - /var/lib/aolserver/service0. Use one cron job to back up the database to a file in - /var/lib/aolserver/service0/database-backup, and a second cron job to back up the entire file tree.
Backing up the database consists of creating a file - which is a picture of the database at a particular moment. - Postgres can be backed up while running.
Depending on your overall backup strategy, you can - create a series of database backup files (recommended for a development server where you are using the section called “Using CVS for backup-recovery”), or you can create a - single nightly backup file which is then collected into a - bigger backup file that includes the other parts of the - service. The latter technique is more generally recommended. To make a new file every - night, edit the crontab file for service0:
[service0@yourserver service0]$ export EDITOR=emacs;crontab -e
Add this line to the file. The numbers and stars at the beginning are cron columns that specify when the program should be run - in this case, whenever the minute is 0 and the hour is 1, i.e., 1:00 am every day.
0 1 * * * /usr/local/pgsql/bin/pg_dump -f /var/lib/aolserver/service0/database-backup/service0_`date +\%Y-\%m-\%d`.dmp service0
If you plan to back up the whole /var/lib/aolserver/service0 directory, then it would be redundant to keep a history of database backups. In that case, set up the cron job to overwrite the previous backup each time:
0 1 * * * /usr/local/pgsql/bin/pg_dump -f /var/lib/aolserver/service0/database-backup/service0_nightly.dmp service0
On a test service, make sure that your backup-recovery process work. After backing up the database and file system, delete the service as detailed below and then recover it.
Earlier strategies, included here because this section - hasn't been fully updated yet.
Edit the backup script to save the backup - file in /var/lib/aolserver/service0/database-backup. - While you're working with Oracle, you should configure it to do - automatic exports. An export is a separate backup copy of the - database. This copy includes all of the database's state at the - time that the export was initiated. If your database is corrupted, - you can restore from one of these backups. You should do this step as - root. -
- Automating backups is accomplished using the UNIX - crontab facility.
- While still root, run the - following command. You can replace the - EDITOR="emacs -nw" - portion with whatever editor your prefer, such as - EDITOR=vi. -
-root:~# export EDITOR="emacs -nw" -root:~# crontab -e
Now add the following line on a line by itself
-0 23 * * * /usr/sbin/export-oracle
- Save the file, exit the editor. Verify that the addition - succeeded by checking the output of the following command.
-root:~# crontab -l | grep export-oracle -0 23 * * * /usr/sbin/export-oracle -root:~# exit -; Logout
If you see the line, go ahead and log out.
This is an alternate method to the crontabls - backup. - Dowload this script - to /tmp. At the top of the script - are several variables that you'll need to customize: -
- bak - location where you want - local backups to be saved -
- servername - name of your server - (and database instance) -
- ftp_user - username on your ftp - account -
- ftp_password - password on your - ftp account -
- ftp_dir - path on the remote - server where your backups will be uploaded -
- ftp_server - your ftp server -
- - Next, we'll save this file to our server's - tcl directory so that it will be - loaded on startup. It will automatically be run every night at - midnight. Note that this script only backs up the database - not the - OpenACS scripts and file content. -
-joeuser:~$ cp /tmp/acs-pgbackup-init.txt ~/var/lib/aolserver/birdnotes/tcl/acs-pgbackup-init.tcl -joeuser:~$ restart-aolserver birdnotes
- That's it! The script will email you with each successful backup (or - if it fails, it will send you an email with the reason) -
If you are already using CVS, you probably don't - need to do anything to back up your files. Just make - sure that your current work is checked into the system. - You can then roll back based on date - note the - current system time, down to the minute. For maximum - safety, you can apply a tag to your current - files. You will still need to back up your database.
Note that, if you did the CVS options in this document, the /var/lib/aolserver/service0/etc directory is not included in cvs and you may want to add it.
[root@localhost root]# su - service0
-[service0@localhost service0]$ cd /var/lib/aolserver/service0
-[service0@localhost service0]$ cvs commit -m "last-minute commits before upgrade to 4.6"
-cvs commit: Examining .
-cvs commit: Examining bin
-(many lines omitted)
-[service0@localhost service0]$ cvs tag before_upgrade_to_4_6
-cvs server: Tagging bin
-T bin/acs-4-0-publish.sh
-T bin/ad-context-server.pl
-(many lines omitted)
-[service0@localhost service0]$ exit
-[root@localhost root]#
-su - service0
-cd /var/lib/aolserver/service0
-cvs commit -m "last-minute commits before upgrade to 4.6"
-cvs tag before_upgrade_to_4_6
-exit
To restore files from a cvs tag such as the one used above:
[root@localhost root]# su - service0
-[service0@localhost service0]$ cd /var/lib/aolserver/service0
-[service0@localhost service0]$ cvs up -r current
-[service0@localhost service0]$ exit
-su - service0
-cd /var/lib/aolserver/service0
-cvs up -r current