CloudWatch Logs

CloudWatch Logging Container Detail

The CloudWatch Logging container was created for VNS3 version 4.8.3+. If you are using an VNS3 version prior to 4.8.3, please scroll down to the Manual Configuration section of this document.

Getting Started with VNS3 Plugin System

The AWS CloudWatch Logs agent is deployed to VNS3 (version 4.8.3+) as a plugin using the container system.

Please be familiar with the VNS3 Plug-In Configuration Guide.

CloudWatch Logging Container - What does it do?

The CloudWatch Logging Container pulls live logs from a host VNS3 controller via the AWS CloudWatch Logs agent and displays these logs in AWS CloudWatch. A CloudFormation template is used to automatically create all the necessary AWS components.

The CloudFormation template creates metrics filters based on potential log messages that notify/alarm an SNS topic when triggered. (text message, email, etc.)

The goal is to minimize outages by monitoring the VNS3 controller’s IPsec, overlay, and/or system logs by sending automatic alerts when unfavorable conditions arise.

CloudWatch Logging Container - What does it need?

User input to deploy the CloudWatch Logs Agent:

  • Create CloudFormation Stack via the template URL
  • Define a log Groupname and SNS topic (email or phone number)
  • Attach Role to VNS3 Instance
  • Open AWS Security Groups to TCP port 80
  • Add the appropriate VNS3 firewall rules
  • Upload the CloudWatch Logging Container image
  • Provide configuration options (Groupname and Logs) to the container via environment variables

Deploying the CloudWatch Logging Container

Deploying the CloudFormation Template

Create a new CloudFormation Stack with the following URL:
https://vns3-containers-read-all.s3.amazonaws.com/CloudWatch_Logger/CWloggerCFtemplate_20191023.yaml

This is a read-only Amazon S3 storage location. Only Cohesive Networks can update or modify files stored in this location.

The CloudFormation template will automatically create all the AWS components (IAM Role, SNS topic, CloudWatch metric filters, alarms, and a log group). You will need to define a unique log Groupname and provide an email address that will be used for the SNS topic.

NOTE: Once the CloudFormation template is deployed, you will receive a SNS subscription confirmation email. Confirm the subscription BEFORE deploying the container.

The CloudFormation template creates an IAM Role, attach this Role to your existing VNS3 controller (the Role name will correlate to the CloudFormation Stack name). From the EC2 Console, select the VNS3 controller, Select Actions > Instance Settings > Attach/Replace IAM Role Attaching the Role to the VNS3 instance

If you already have an IAMs Role attached to the instance, you will need to apply the two new policies to the existing Role. (see Developers section below)

Edit your AWS Secuirty Groups

The AWS CloudWatch agent requires TCP port 80 inbound traffic. You will need to add the following rule into the AWS Security Group that is associated with the VNS3 controller:

Add TCP port 80 to AWS Security Groups

Uploading the Container Image to the VNS3 Plugin System

From the Containers page in the VNS3 Web UI, select Container —> Images, select Upload Image.

WARNING: Your Image name must include the word "logger".

The name can be upper, lower, or mixed case. The word “logger” is required to provide containers allocated from this image with the appropriate log files from VNS3.

Copy and paste the CloudWatch Logs Container Image file URL:
https://vns3-containers-read-all.s3.amazonaws.com/CloudWatch_Logger/CloudWatch_Logger_20191022.tar.gz

Click Upload.

Uploading the Container Image to the VNS3 Plugin System

CloudWatch Container Firewall Rules

The CloudWatch Container requires the following firewall rules to be added to the VNS3 controller BEFORE allocating the container:

Anywhere it appears, replace <container_ip> with the IP you will use for your CloudWatch container.

#SNAT for container network
MACRO_CUST -o eth0 -s <container_ip> -j MASQUERADE

Allocating the Container from the Image

When the Image has imported successfully, it will say Ready in the Status Column.

To launch a container from the image, choose Allocate from the Action menu.

Launching the CloudWatch Logging Container

After selecting “Allocate”, provide a name for your container, /usr/bin/supervisord as the Command and a description if you’d like. You can allow VNS3 to auto-assign a container network IP, but we recommend that you choose one manually.

Once the conatiner is allocated, go the the Plugin Manager in the dropdown menu.

Navigate to the plugin manager

Read the commented section in the parameters.conf file and edit the file accoridingly. Once it is save you will need to RUN the exectuable in order for the CloudWatch agent to configure to AWS.

The other way to configure this container on initial boot is with the use of variables (groupname and logs, as seen in the picture below). The environment variables section is where you define what system logs will be sent to CloudWatch.

Environment variables format:

groupname="<log group name>", logs="<log1> <log2> <log3>"

If you are using a version of VNS3 older than 4.8.3, you must define the groupname and log streams by SSHing into the container and editing the awslogs.conf configuration file. (See the Developers section below.)

Launching the CloudWatch Logs Plugin

Here are the system logs which can be offloaded to CloudWatch for monitoring and alerting:

  • auth.log
  • boot.log
  • dmesg
  • kern.log - VNS3 kernel log
  • syslog - VNS3 system log
  • vnscubed_api_background.log
  • vnscubed_api_server.log
  • vnscubed_bgpmon.log
  • vnscubed_monitor.log
  • vnscubed_webui.log
  • webserver_access.log
  • webserver_error.log
  • vnscubed_api_background/completed/ - Directory showing completed API calls
  • vnscubed_connection_logs/ipsec.log - IPsec system log file
  • vnscubed_connection_logs/tun0.log - Overlay Network server connection log files
  • vnscubed_connection_logs/unauthorized.log

If you would like to send multiple logs to CloudWatch, use a space-separated list in quotes as seen in the picture below:

Manual Configuration (for developers)

When using a VNS3 version prior to 4.8.3

After allocating the container and applying the necessary firewall rules to VNS3, you can SSH into the container on port 44.

The username is container_admin, and the default password is container_admin_123!

We recommend that you change this password immediately: ~$ sudo passwd container_admin

If you are using a version of VNS3 prior to 4.8.3, SSH into the container, and cd into the /opt/awslogs/ directory. You will need to know what AWS region the VNS3 is deployed in for the next step.

Run: curl http://169.254.169.254/latest/meta-data/placement/availability-zone to get the region. Do not include the letter at the end of the region. (i.e. us-east-1 instead of us-east-1b )

Once you have the region, from the /opt/awslogs/ directory run the following command:
sudo python ./awslogs-agent-setup.py --region us-east-1 (change region accordingly)

This will launch the aws-logs agent in interactive mode. You do not have to specify the Access Key ID or Secret Access Key when prompted. Skip through till it asks for “Path of log to upload [var/log/syslog]:". Here, you will copy and paste the desired log path that will be sent to CloudWatch from the list below:

  • /mnt/logs/auth.log
  • /mnt/logs/boot.log
  • /mnt/logs/dmesg
  • /mnt/logs/kern.log - VNS3 kernel log
  • /mnt/logs/syslog - VNS3 system log
  • /mnt/logs/vnscubed_api_background.log
  • /mnt/logs/vnscubed_api_server.log
  • /mnt/logs/vnscubed_bgpmon.log
  • /mnt/logs/vnscubed_monitor.lo
  • /mnt/logs/vnscubed_webui.log
  • /mnt/logs/webserver_access.log
  • /mnt/logs/webserver_error.log
  • /mnt/logs/vnscubed_api_background/completed/ - Directory showing completed API calls
  • /mnt/logs/vnscubed_connection_logs/ipsec.log - IPsec system log file
  • /mnt/logs/vnscubed_connection_logs/tun0.log - Overlay Network server connection log files
  • /mnt/logs/vnscubed_connection_logs/unauthorized.log

Only select one log, you can add more later.

The next input is “Destination of Log Group name”. This must be identical to what you defined as the Log Groupname when you deployed the CloudFormation template. If you forgot the Log Groupname, simply go to the CloudWatch console and select Logs in the left column. You will see the Log Groupname under Log Groups.

For the Log Stream name, you must use the log path minus the /mnt/logs/ path.

Example:

If you are defined /mnt/logs/vnscubed_connection_logs/ipsec.log as the “Path to log to upload”, you will need to define the logstream name as just vnscubed_connection_logs/ipsec.log

We keep the default “timestamp format” and “initial position” but feel free to change these settings at our own discretion. You will then have the option to add other log streams now, otherwise type n to complete the configuration.

The configuration files will be made although the logging agent will not run successfully until you add the following lines to the top of the /var/awslogs/etc/awslogs.conf file:

[general]
state_file = /var/awslogs/state/agent-state

Once the awslogs.conf file is edited, run: sudo service awslogs restart to pull in the completed configuration.

Editing the Configuration (post setup)

SSH into the container with the steps above. If you would like to add additional log streams or edit any log stream data, you can edit the configuration file found at /var/awslogs/etc/awslogs.conf.

Variables that can be edited: log group name, log stream name, initial position, log file, date format, and buffer duration (default is the minimum, =5000ms).

To add additional log streams, copy the format that is in awslogs.conf and change the highlighted sections:

Adding log streams to the awslogs.conf file

Once you have finished editing this file, you need need to start and stop the container in order for the new configuration to take effect.

Attaching IAMs Policies to Pre-existing Role

Two policies are made by the CloudFormation Template: CloudWatch List and Write, IAM Security Credential Pass Policy

If you have a pre-existing IAM Role attached to your VNS3 instance, you will need to manually add the policies created by the CloudFormation template to the pre-existing role.

The Role name will correlate to The CloudFormation Stack name. Copy and paste the two policies to the Role that is already attached to the instance.

Creating Additional CloudWatch Metric Filters and Alarms

Navigate to the CloudWatch Console and select Logs. Select the VNS3 Logs and Create Metric Filter.

Define a Filter Pattern and Assign the Metric.

Create a Metric filter

Once the Metric Filter is created, you can create an Alarm based on the Filter. Select Create Alarm and define the appropriate parameters.

Create an alarm based on the metric filter

For Developers

The container uses the AWS-logs agent to send system logs to AWS CloudWatch.

When the container is initially allocated, a startup script runs which creates a file with the groupname and log(s) streams you chose to send to CloudWatch (via the environment variables). This startup script also adds a file into the supervisord configuration /etc/supervisor/conf.d to make the AWS-logs agent run on boot.

Export a Container Image

In the event that your VNS3 controller needs to be replaced or upgraded, you will need a copy of your configured CloudWatch Agent Container. We recommend downloading an image of your configured container as part of the deployment process:

From the Containers page in the VNS3 Web UI, select Action > Save as Image for your CloudWatch Container. Once that process is complete, you’ll be brought to the Images page. Select Action > Export on the new image, and provide a name.

Once Exporting is complete, you will have the option to download the image locally.