CT2 Contingency Plan
From UG
Contents |
Summary
The objective of CT2 Contigency Plan is to be able to recover and support continuity of business of all critical systems hosted by Rackspace resulting from a full datacenter outage. Currently CyberTrax, SugarCRM, Mantis, Wiki, Homepage websites are hosted by Rackspace in Texas. This wiki should document all steps necessary to failover to a Disaster Recovery site (currently designated as Valley Stream, NY office). This includes steps to restore databases, bring up web servers, application servers, FTP servers, and SMTP functionality. Normalization steps (reverting system from the Disaster Recovery back to Production site) will need to be documented as well.
External interfaces including, but not limited to EDI transfers to Trendset, EDI transfers to Descartes (TMS), and all other systems that interface with CyberTrax are in scope for this contingency plan.
At this point, there is no specific Recovery Point Objective (RPO) or Recovery Time Objective (RTO) for this plan. These parameters will be determined at a later time.
Deployment Architecture/Technology Stack
Recovery
Prerequisites
- DNS TTL of ct.jaguarfreight.com is by default 24 hours. Should be set to lower value, say 300 seconds. To be done in https://my.rackspace.com, once a week.
- Currently, inbound TMS FTP accounts are the same for dev and production. If dev is to be recovery target, they must not conflict. Accounts should either be made different, or an additional virtual ftp server should be setup on dev. The latter may conflict with ongoing network reshuffle.
[Perry] I'm not clear on the above statement regarding TMS FTP accounts. Does TMS use FQDN to connect to us or do they connect via IP address?
[Vlad] After ct2 is redirected to dev2, both ct2 and dev2 point to the same IP address. Since the same username is used, in both production and dev, files dev and ct files will get mixed. Although dev2 could be redirected to some other address for duration of outage, it's not guaranteed to take effect immediately. So to avoid this kind collision it's better to change account names used in dev (the names are descartes and airdescartes).
- Edit SPF DNS record to authenticate mail sent by dev.
[Perry] Can this be done in advance, without impacting our current production system's ability to send email?
[Vlad] Already done
- Synchronize database/files backup of production to minimize inconsistency
[Perry] What do you mean syncrhonize database/files backups? I thought that we copied the data from production to the NY server every night?
[Vlad] Backups should be moved closer to each other and away from Rackspace backup. Files should be backed after database. Since backup is resource consuming operation, backup should be started when during period of historically low server utilization. I need to gather some statistics for that.
Steps
- Stop scheduled backups of production. On dev2 execute: "touch /.no-ct2-backup"
- Check that the latest db backup has been restored on dev2. Go to https://dev2.jaguarfreight.com/LastBackup.html. TBD
- Stop dev, staging environments. Go to https://dev.jaguarfreight.com/manager/html, https://staging.jaguarfreight.com/manager/html, stop /Clinet and /internal apps there
- Stop xlive environment. Go to https://xlive.jaguarfreight.com/manager/html, stop /servlets/cybertrax
- Pull app. files from the latest backup to working directory. On dev2 execute: "mv /var/cybertrax2 /var/cybertrax2.save; XXX /var/cybertrax2". TBD
- Deploy latest WAR files from backup to production. On dev2 execute: "cp /backup/ct2_webapps/2012-02-XXX/*.war /srv/tomcat6/ct"
- Redirect ct.jaguarfreight.com to 69.74.55.203. Go to https://my.rackspace.com. Account number 22130. Network->Domains->jaguarfreight.com
- TBD
[Perry] I'm assuming the above steps are all to be performed on DEV2?
[Vlad] Yes, with exception of part to be done on rackspace
[Vlad to complete]
Estimated recovery time 10 minutes
Database
Web Server
Application Server
FTP server
SMTP server
Recovery steps......
Normalization
Steps
- Stop application
- Start dev, staging, xlive environments.
- Backup database
- Compress database
- Copy compressed database to production
- Backup production database for examination later
- Drop production database
- Restore production database from backup
- Save existing TMS files in production for examination later
- Copy outbound TMS files to production
- DNS switch ct.jaguarfreight.com from dev to production
- Copy inbound TMS files to production
- Reenable backup of production. On dev2 execute: "rm /.no-ct2-backup"
- TBD
[Perry] I'm assuming the above Normalization steps are to be executed on DEV2. But what about the steps necessary on CT2 to bring Production back online?
[Vlad] CT2 will be brought back by Rackspace. They will restore whatever (latest) backup they have. Data from second site will overwrite data from the backup.
[Vlad to complete]
Estimated normalization time 2 hours
Database
Web Server
Application Server
FTP server
SMTP server
Normalization steps...
Proposed CT2 Disaster Recovery Environment
Objectives
- Recovery Time Objective (RTO) - 2 hours to recover in Disaster Recovery environment
- Recovery Point Objective (RPO) - < 1 hour of potential data loss
- Eliminate all Single Points of Failure - incorporate redundancies in all nodes
