High Availability (HA)
Overview
A semi-automatic instance replacement solution in the event of the VNS3 Primary controller becoming unhealthy. If/when the VNS3 Primary fails, VNS3:ms can trigger an automated failover to the VNS3:ha Backup.
Requirements
- Currently restricted to AWS VPC resident controllers.
- Existing VNS3 controller with one or many IPsec tunnels negotiated, connected, and healthy.
- Use of an Elastic IP for the public IP of the VNS3 controller.
- VNS3:ha backup instance running VNS3 v3.5.1.14–20160315 or later in the same VPC as the VNS3 primary controller.
- VNS3:ha backup instance needs to have the same API password as the primary VNS3 instance it will be associated with.
- Use of Overlay or Underlay networks for cloud-based instances.
- Ability to create and add cloud API credentials to allow your VNS3:ms instance to access your cloud account and run the required actions (ec2:Describe* + ec2:AssociateAddress, ec2:DisassociateAddress, ec2:ModifyInstanceAttribute, ec2:ReplaceRoute, and ec2:StopInstances). OR
- Ability to create the required IAM Policy and IAM Role/ that will be attached to the instance.
HA Modes: Cold, Warm, Hot
VNS3:ha is managed via VNS3:ms and has three flavors - cold, warm and hot.
VNS3:ha is an automated instance-based failover to provide HA capabilities for customers that can’t deploy BGP/Dynamic Routing redundant IPsec Tunnels.
For each ‘flavor’ of HA, there is a reciprocating IAM policy that will need to be created. This policy will then be attached to a Role that is attach to the VNS3:ms instance giving it the necessary permissions to perform automated HA/failover actions. If you are managing VNS3 in different AWS accounts, the required policy will be attached to a IAM User that resides in the same AWS account of the VNS3 controller.
Cold HA provides an automated process for instance-based failover for VNS3 controller instances at a reduced cost (no running or stopped backup instance runtime costs) and reduced RTO (instance launch times are longer than reboot or start times). In a Cold HA failover activation scenario, VNS3:ms automatically launches a backup instance from an AMI ID specified by the user. Once the instance is available, VNS3:ms remaps the primary instance IP to the backup instance, then initializes and configures the backup instance using a locally stored snapshot file from the primary controller (updated automatically every 30 mins - RPO). After the backup instance reboots, it takes the place of failed VNS3 controller.
Warm HA provides an automated process for instance-based failover of VNS3 controller instances at a reduced cost to Hot HA but also a slightly reduced RTO (additional time to start the instance). In a Warm HA failover scenario, VNS3:ms automatically starts the user specified backup instance ID. Once the backup instance is available, VNS3:ms remaps the primary instance IP to the backup instance, then initializes and configures the backup instance using a locally stored snapshot file from the primary controller (updated automatically every 30 mins - RPO). After the backup instance reboots, it takes the place of the failed VNS3 controller.
For Hot HA, VNS3:ha backup instances are associated with a running and healthy VNS3 controller. In the event the VNS3 primary controller fails (instance, hardware, or connection failure), VNS3:ms can run an automated failover process where the VNS3:ha backup takes the place of failed VNS3 primary controller.
HA Configuration
When managing VNS3 controllers that are in the same AWS account as the VNS3:ms instance, you will use dynamic cloud credentials via an IAMs Role attached to the VNS3:ms instance. For managing VNS3 controllers that are in different AWS accounts, you will use static cloud credentials via an IAMs User. You, or your customer, will need share the Access Key ID and Secret Key associated with the IAMs User in order to create a VNS3:ms Cloud Credential.
This cloud credential provides the necessary permissions for the VNS3:ms instance to perform HA/failover. Depending on the flavor of HA you are configuring, there is a specific policy (cold, warm hot), found below, that will be attached to the IAMs Role or User.
Creating Cloud Credentials
Recommended best practices are to use an AWS IAM Role attached to the VNS3:ms instance for temporary/dynamic AWS API keys (NOTE: IAM roles do support cross account access).
In situations where the IAM Role aren’t an option, long term/static API credentials can be used but it is recommended that a specific IAM programatic access account is created for the VNS3:ms system. This can be done by creating a IAM User with the necessary Policy attached.
You will first need to create the appropriate IAM policy and attach it to a Role. The Policies can be found under the HA Configuration section oF this document. From the EC2 Console, select the VNS3:ms instance > Actions > Instance Settings > Attach/Replace IAM Role.
Select the IAM Role that your just created.
From the VNS3:ms Web UI, in the top right corner select Admin > Cloud Credentials. Select “Add Cloud Creds”, then provide a Name, Description, and Credential Type: EC2. Select the Checkbox that says “Use VNS3:ms IAM Role”.
When managing a VNS3 controller in a different AWS account, create an IAMs User in the account of the VNS3 instance and attach the appropriate Cold/Warm/Hot IAMs Policy to the User. Retrieve the Key ID and Secret Access Key associated with this User. From the VNS3:ms Cloud Credentials page, select “Add Cloud Creds”. Provide a Name, Description, Credential Type: EC2, Access Key, and Shared Key for this Static Cloud Credential.
Cold HA
Now that we have our Cloud Credential made, in the VNS3:ms Web UI navigate the the VNS3 controller and select HA Configuration. Select the “HA Enabled” checkbox and “Cold” for the HA Mode. Paste in the latest VNS3 AMI-ID into the HA Back Server AMI section and select the Cloud Credential you just made from the Cloud Credential dropdown list. Fill out the desired instance size, Availability Zone, and subnet that the backup instance will be deployed in. Update HA Backup details to initialize the configuration.
To activate Cold HA, go to the HA Configuration dashboard and scroll to the bottom of the page. Click on the big RED button that says “Activate”. This will bring up a new VNS3 instance, swap the EIP to the new instance, and configure the VNS3 controller with a snapshot.
Cold IAM Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ActionsRequiredforVNS3haCold",
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:*:*:instance/*",
"arn:aws:ec2:*:*:key-pair/*",
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:volume/*",
"arn:aws:ec2:*:*:network-interface/*",
"arn:aws:ec2:*:*:subnet/*"
]
},
{
"Sid": "LaunchingVNS3fromAMIownedByCohesive",
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*::image/ami-*",
"Condition": {
"StringEquals": {
"ec2:Owner": "678554804139"
}
}
},
{
"Sid": "ActionsRequiredforStopStartInstnaceWithVNS3tag",
"Effect": "Allow",
"Action": [
"ec2:StopInstances"
],
"Resource": "arn:aws:ec2:*:*:instance/*"
},
{
"Sid": "ActionsRequiredforVNS3msHA",
"Effect": "Allow",
"Action": [
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeClassicLinkInstances",
"ec2:DescribeCustomerGateways",
"ec2:DescribeDhcpOptions",
"ec2:DescribeEgressOnlyInternetGateways",
"ec2:DescribeFlowLogs",
"ec2:DescribeImageAttribute",
"ec2:DescribeImages",
"ec2:DescribeInstance*",
"ec2:DescribeInternetGateways",
"ec2:DescribeNatGateways",
"ec2:DescribeNetwork*",
"ec2:DescribePrefixLists",
"ec2:DescribeRegions",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroupReferences",
"ec2:DescribeSecurityGroups",
"ec2:DescribeStaleSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeTags",
"ec2:DescribeTransit*",
"ec2:DescribeVpc*",
"ec2:DescribeVpn*",
"ec2:AssociateAddress",
"ec2:DisassociateAddress",
"ec2:UnassignPrivateIpAddresses",
"ec2:AssignPrivateIpAddresses",
"ec2:ReplaceRoute",
"ec2:CreateTags",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyNetworkInterfaceAttribute"
],
"Resource": "*"
},
{
"Sid": "GetInstanceTypes",
"Effect": "Allow",
"Action": "pricing:GetAttributeValues",
"Resource": "*"
}
]
}
Warm HA
Before configuring Warm HA on VNS3:ms, make sure the Warm HA Policy (found below) is attached to the IAMs Role/User that is associated with the VNS3:ms Cloud Credential. For VNS3:ms Warm HA you will need to create a Backup VNS3 Controller.
Go to your AWS EC2 console and bring up a new VNS3 instance in the same VPC as the existing/primary VNS3 instance. Configure this VNS3 instance just as the primary instance. On the Instance Configuration page, select disable on the “Auto-Assign Public IP” drop down menu on the Instance Configuration step when bringing up the new VNS3. You will attach an Elastic IP (EIP) to this instance after it is launched.
This new VNS3 instance will act as the backup server when you activate an HA event. Once the instance is created, attach an EIP to the instance. Log into this backup instance and change the Admin passwords (Web UI and API) to match the primary VNS3 instance. In the EC2 console, copy the instance ID to your clipboard and Stop the instance.
From the VNS3:ms dashboard, navigate to the VNS3 controller and select the ‘HA Configuration’ tab. Check the ‘HA Enabled’ checkbox, select Warm HA as the HA Mode, paste in the instance ID from your clipboard into the HA Backup Server Instance Id box, and the select the appropriate Warm Cloud Credential.
Click on the ‘Update HA Backup Details’ button and wait for the HA Status to change to Active. (<1 minute)
Once the HA Status is Active you can click on the big red button to activate a Warm HA event.
Warm IAM Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ActionsRequiredforVNS3haWarmStopStartInstnaceWithVNS3tag",
"Effect": "Allow",
"Action": [
"ec2:StartInstances",
"ec2:StopInstances"
],
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"ForAnyValue:StringLike": {
"ec2:ResourceTag/Name": [
"*vns3*",
"*VNS3*",
"*VNS*",
"*Vns3*:"
]
}
}
},
{
"Sid": "ActionsRequiredforVNS3msHA",
"Effect": "Allow",
"Action": [
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeClassicLinkInstances",
"ec2:DescribeCustomerGateways",
"ec2:DescribeDhcpOptions",
"ec2:DescribeEgressOnlyInternetGateways",
"ec2:DescribeFlowLogs",
"ec2:DescribeImageAttribute",
"ec2:DescribeImages",
"ec2:DescribeInstance*",
"ec2:DescribeInternetGateways",
"ec2:DescribeNatGateways",
"ec2:DescribeNetwork*",
"ec2:DescribePrefixLists",
"ec2:DescribeRegions",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroupReferences",
"ec2:DescribeSecurityGroups",
"ec2:DescribeStaleSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeTags",
"ec2:DescribeTransit*",
"ec2:DescribeVpc*",
"ec2:DescribeVpn*",
"ec2:AssociateAddress",
"ec2:DisassociateAddress",
"ec2:UnassignPrivateIpAddresses",
"ec2:AssignPrivateIpAddresses",
"ec2:ReplaceRoute",
"ec2:CreateTags",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyNetworkInterfaceAttribute"
],
"Resource": "*"
}
]
}
Hot HA
Bring up a new VNS3 instance just as you would with Warm HA. Be sure to disable “Auto-assign Public IP” as this instance and attach an EIP to the instance after it is launched. This will be your Hot HA backup server.
NOTE: If you are using the Underlay Network, remember to Disable Src/Dst checks on the backup instance.
( Actions > Networking > Change Source/Dest. Check > Disable )
Once the instance is running, log into the VNS3 Web UI using the default username and password (instance-ID). Once logged in, change the Admin passwords (Web UI and API) to match the primary instance. Near the bottom of the homepage you will see the highlighted Configuration Backup Server UUID. Copy this UUID to your clipboard and navigate to the VNS3:ms Web UI.
Select to the VNS3 controller that you will be setting up HOT HA for. Go to the HA Configuration Tab and check the “HA enabled” checkbox. Select “Hot” for the HA Mode and paste in the UUID from your clipboard. Paste in the public IP of this backup server and specify the necessary Cloud Credential (HOT). Be sure that the HOT policy is attached to your dynamic or static cloud credential. Save the HA configuration.
Once the HA Status changes to Activate, click the big red Activate button to activate an HA event.
Hot IAM Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ActionsRequiredforVNS3haHotStopInstnaceWithVNS3tag",
"Effect": "Allow",
"Action": "ec2:StopInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"ForAnyValue:StringLike": {
"ec2:ResourceTag/Name": [
"*vns3*",
"*VNS3*",
"*VNS*",
"*Vns3*:"
]
}
}
},
{
"Sid": "ActionsRequiredforVNS3msHA",
"Effect": "Allow",
"Action": [
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeClassicLinkInstances",
"ec2:DescribeCustomerGateways",
"ec2:DescribeDhcpOptions",
"ec2:DescribeEgressOnlyInternetGateways",
"ec2:DescribeFlowLogs",
"ec2:DescribeImageAttribute",
"ec2:DescribeImages",
"ec2:DescribeInstance*",
"ec2:DescribeInternetGateways",
"ec2:DescribeNatGateways",
"ec2:DescribeNetwork*",
"ec2:DescribePrefixLists",
"ec2:DescribeRegions",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroupReferences",
"ec2:DescribeSecurityGroups",
"ec2:DescribeStaleSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeTags",
"ec2:DescribeTransit*",
"ec2:DescribeVpc*",
"ec2:DescribeVpn*",
"ec2:AssociateAddress",
"ec2:DisassociateAddress",
"ec2:UnassignPrivateIpAddresses",
"ec2:AssignPrivateIpAddresses",
"ec2:ReplaceRoute",
"ec2:CreateTags",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyNetworkInterfaceAttribute"
],
"Resource": "*"
}
]
}
Administration
VNS3:ha stop primary
Depending on the version of the Primary VNS3 controller instance, there may or may not be a Stop old primary checkbox available.
Starting with VNS3 version 3.5.1.14 and later, this option will be available. Stopping the old primary instance after a successful failover ensures all old connections to that device are moved over to the failed over VNS3:ha instance.
VNS3:ha Configured State
Once a new VNS3:ha backup instance has been initialized and synced with the latest configuration of the Primary, you will see completed status messages.
VNS3:ms will then continue to update the VNS3:ha controller every 30 minutes with the primary VNS3 configuration. This ensure the recovery point objective for a VNS3:ha failover is always a configuration state of at most 30 minutes old.
You can view the Sync Log under the “HA Status” section.
