CT2 Contingency Plan
From UG
Contents |
Summary
The objective of CT2 Contigency Plan is to be able to recover and support continuity of business of all critical systems hosted by Rackspace resulting from a full datacenter outage. Currently CyberTrax, SugarCRM, Mantis, Wiki, Homepage websites are hosted by Rackspace in Texas. This wiki should document all steps necessary to failover to a Disaster Recovery site (currently designated as Valley Stream, NY office). This includes steps to restore databases, bring up web servers, application servers, FTP servers, and SMTP functionality. Normalization steps (reverting system from the Disaster Recovery back to Production site) will need to be documented as well.
External interfaces including, but not limited to EDI transfers to Trendset, EDI transfers to Descartes (TMS), and all other systems that interface with CyberTrax are in scope for this contingency plan.
At this point, there is no specific Recovery Point Objective (RPO) or Recovery Time Objective (RTO) for this plan. These parameters will be determined at a later time.
Deployment Architecture/Technology Stack
Recovery
Prerequisites
- DNS TTL of ct.jaguarfreight.com is by default 24 hours. Should be set to lower value, say 300 seconds. To be done in https://my.rackspace.com, once a week.
- Currently, inbound TMS FTP accounts are the same for dev and production. If dev is to be recovery target, they must not conflict. Accounts should either be made different, or an additional virtual ftp server should be setup on dev. The latter may conflict with ongoing network reshuffle.
[Perry] I'm not clear on the above statement regarding TMS FTP accounts. Does TMS use FQDN to connect to us or do they connect via IP address?
- Edit SPF DNS record to authenticate mail sent by dev.
[Perry] Can this be done in advance, without impacting our current production system's ability to send email?
- Synchronize database/files backup of production to minimize inconsistency
[Perry] What do you mean syncrhonize database/files backups? I thought that we copied the data from production to the NY server every night?
Steps
- Stop scheduled backups of production. On dev2 execute: "touch /.no-ct2-backup"
- Check that the latest db backup has been restored on dev2. Go to https://dev2.jaguarfreight.com/LastBackup.html. TBD
- Stop dev, staging environments. Go to https://dev.jaguarfreight.com/manager/html, https://staging.jaguarfreight.com/manager/html, stop /Clinet and /internal apps there
- Stop xlive environment. Go to https://xlive.jaguarfreight.com/manager/html, stop /servlets/cybertrax
- Pull app. files from the latest backup to working directory. On dev2 execute: "mv /var/cybertrax2 /var/cybertrax2.save; XXX /var/cybertrax2". TBD
- Deploy latest WAR files from backup to production. On dev2 execute: "cp /backup/ct2_webapps/2012-02-XXX/*.war /srv/tomcat6/ct"
- Redirect ct.jaguarfreight.com to 69.74.55.203. Go to https://my.rackspace.com. Account number 22130. Network->Domains->jaguarfreight.com
- TBD
[Perry] I'm assuming the above steps are all to be performed on DEV2?
[Vlad to complete]
Estimated recovery time 10 minutes
Database
Web Server
Application Server
FTP server
SMTP server
Recovery steps......
Normalization
Steps
- Stop application
- Start dev, staging, xlive environments.
- Backup database
- Compress database
- Copy compressed database to production
- Backup production database for examination later
- Drop production database
- Restore production database from backup
- Save existing TMS files in production for examination later
- Copy outbound TMS files to production
- DNS switch ct.jaguarfreight.com from dev to production
- Copy inbound TMS files to production
- Reenable backup of production. On dev2 execute: "rm /.no-ct2-backup"
- TBD
[Perry] I'm assuming the above Normalization steps are to be executed on DEV2. But what about the steps necessary on CT2 to bring Production back online?
[Vlad to complete]
Estimated normalization time 2 hours
Database
Web Server
Application Server
FTP server
SMTP server
Normalization steps...
