CT2 Contingency Plan
From UG
(→Prerequisites) |
(→Normalization Steps) |
||
| Line 47: | Line 47: | ||
Recovery steps...... | Recovery steps...... | ||
| - | + | ==Normalization Steps== | |
| - | + | ===Steps=== | |
| + | * Stop application | ||
| + | * Start dev, staging, xlive environments. | ||
| + | * Backup database | ||
| + | * Compress database | ||
| + | * Copy compressed database to production | ||
| + | * Backup production database for examination later | ||
| + | * Drop production database | ||
| + | * Restore production database from backup | ||
| + | * Save existing TMS files in production for examination later | ||
| + | * Copy outbound TMS files to production | ||
| + | * DNS switch ct.jaguarfreight.com from dev to production | ||
| + | * Copy inbound TMS files to production | ||
| + | * TBD | ||
[Vlad to complete] | [Vlad to complete] | ||
| - | Estimated normalization time | + | Estimated normalization time 2 hours |
Revision as of 13:28, 20 February 2012
Contents |
Summary
The objective of CT2 Contigency Plan is to be able to recover and support continuity of business of all critical systems hosted by Rackspace resulting from a full datacenter outage. Currently CyberTrax, SugarCRM, Mantis, Wiki, Homepage websites are hosted by Rackspace in Texas. This wiki should document all steps necessary to failover to a Disaster Recovery site (currently designated as Valley Stream, NY office). This includes steps to restore databases, bring up web servers, application servers, FTP servers, and SMTP functionality. Normalization steps (reverting system from the Disaster Recovery back to Production site) will need to be documented as well.
External interfaces including, but not limited to EDI transfers to Trendset, EDI transfers to Descartes (TMS), and all other systems that interface with CyberTrax are in scope for this contingency plan.
At this point, there is no specific Recovery Point Objective (RPO) or Recovery Time Objective (RTO) for this plan. These parameters will be determined at a later time.
Deployment Architecture/Technology Stack
Recovery
Prerequisites
- DNS TTL of ct.jaguarfreight.com is by default 24 hours. Should be set to lower value, say 300 seconds. To be done in https://my.rackspace.com, once a week.
- Currently, inbound TMS FTP accounts are the same for dev and production. If dev is to be recovery target, they must not conflict. Accounts should either be made different, or an additional virtual ftp server should be setup on dev. The latter may conflict with ongoing network reshuffle.
- Edit SPF DNS record to authenticate mail sent by dev.
Steps
- Stop scheduled backups of production. On dev2 execute: "touch /.no-ct2-backup"
- Check that the latest db backup has been restored on dev2. Go to https://dev2.jaguarfreight.com/LastBackup.html. TBD
- Stop dev, staging environments. Go to https://dev.jaguarfreight.com/manager/html, https://staging.jaguarfreight.com/manager/html, stop /Clinet and /internal apps there
- Stop xlive environment. Go to https://xlive.jaguarfreight.com/manager/html, stop /servlets/cybertrax
- Pull app. files from the latest backup to working directory. On dev2 execute: "mv /var/cybertrax2 /var/cybertrax2.save; XXX /var/cybertrax2". TBD
- Deploy latest WAR files from backup to production. On dev2 execute: "cp /backup/ct2_webapps/2012-02-XXX/*.war /srv/tomcat6/ct"
- Redirect ct.jaguarfreight.com to 69.74.55.203. Go to https://my.rackspace.com. Account number 22130. Network->Domains->jaguarfreight.com
[Vlad to complete]
Estimated recovery time 10 minutes
Database
Web Server
Application Server
FTP server
SMTP server
Recovery steps......
Normalization Steps
Steps
- Stop application
- Start dev, staging, xlive environments.
- Backup database
- Compress database
- Copy compressed database to production
- Backup production database for examination later
- Drop production database
- Restore production database from backup
- Save existing TMS files in production for examination later
- Copy outbound TMS files to production
- DNS switch ct.jaguarfreight.com from dev to production
- Copy inbound TMS files to production
- TBD
[Vlad to complete]
Estimated normalization time 2 hours
Database
Web Server
Application Server
FTP server
SMTP server
Normalization steps...
