Monitoring Applications at Scale in AWS

The Problem

There are various scenarios when we chose to deploy our applications in different AWS accounts:

1. There are multiple MicroServices deployed in different AWS account in different regions based on use-case.
2. For an organization, there could be multiple AWS accounts configured which will deploy related/unrelated services
3. One AWS account one AWS region etc.

AWS provides local metrics and monitoring via AWS CloudWatch. But things will get complicated when we need to monitor multiple applications from all these accounts to extrapolate and make decisions based on the metrics.

The Technology

AWS released cross-account observability which will allow monitoring applications spanning across multiple accounts in a region. This will allow accounts to share:

• CloudWatch Metrics
• CloudWatch Log groups and
• AWS X-Ray Traces

to a central monitoring accounts.

Glossary

• Monitoring Account: A central AWS account which will view and interact with the data generated by source accounts.
• Source Accounts: These are individual AWS accounts which will generate the above data. They are ’n’ in numbers.

Limitations

• 1 source account can share observability with at most 5 monitoring accounts.
• Cross-region observability is not allowed.
• Only Source account can disable sharing with Monitoring account.

The Solution

cross-account observability and analyzing data

Above is a high level diagram of sharing the metrics, logs and X-Ray data from multiple source accounts to central metrics account using cross-account observability and then extending the solution to further analyze metrics data using ETL, Data Warehouse and QuickSight. Let me discuss each section:

1. Monitoring Account: The central account to stream and visualize the CloudWatch and x-ray data from source accounts. In order to this certain AWS permission and configuration is required.

a. Login using below permissions to configure Oam Sink or log in as admin

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSinkModification",
            "Effect": "Allow",
            "Action": [
                "oam:CreateSink",
                "oam:DeleteSink",
                "oam:PutSinkPolicy",
                "oam:TagResource"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowReadOnly",
            "Effect": "Allow",
            "Action": ["oam:Get*", "oam:List*"],
            "Resource": "*"
        }
    ]
}

b. List of source accounts. In order to allow broader source accounts, organization id can be used. Replace “aws:ResourceAccount” with “aws:PrincipalOrgID”

{
            "Action": [
                "oam:CreateLink",
                "oam:UpdateLink"
            ],
            "Effect": "Allow",
            "Resource": "arn:*:oam:*:*:sink/*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceAccount": [
                        "999999999999"
                    ]
                }
            }
        }

c. Final Policy for an AWS Organization unit. You can remove any resource type which is not needed for your use case.

"Name": "SampleSinkPolicy",
"Policy": {
   "Version": "2012-10-17",
   "Statement": [{
       "Effect": "Allow",
       "Principal": "*",
       "Resource": "*",
       "Action": [ "oam:CreateLink", "oam:UpdateLink" ],
       "Condition": {
        "StringEquals": {"aws:PrincipalOrgID":"o-xxxxxxxxxxx"},
        "ForAllValues:StringEquals": {
          "oam:ResourceTypes": [
            "AWS::CloudWatch::Metric",
            "AWS::Logs::LogGroup",
            "AWS::XRay::Trace"
          ]
        }
       }
    }]
}

d. Configure using AWS Console:

Login -> CloudWatch -> Monitoring account configuration -> Configure (select appropirate options). Once the configuration is complete, you can navigate to CloudWatch -> Monitoring account configuration->Resources to link accounts and select the type of configuration to download Cloudformation Template or copy the URL. This will be used later to configure the source accounts

e. Using CDK

const sinkPolicy = new iam.PolicyDocument({
      statements: [
        new iam.PolicyStatement({
          actions: ['oam:CreateLink', 'oam:UpdateLink'],
          resources: ['*'],
          principals: ['*'],
          conditions: {
            'ForAllValues:StringEquals': {
              'oam:PrincipalOrgID': 'o-xxxxxxxxxxx'
            }
          }
        })
      ]
    });

    this.sink = new oam.CfnSink(this, 'SampleMonitoringSink', {
      name: 'SampleMonitoringSink',
      policy: sinkPolicy
    });

2. Source Accounts: Source accounts configuration can be done in 3 ways

a. Using CloudFormation: Once Monitoring account is configured, and cloudformation is downloaded, it can be used to deploy the stack using AWS CloudFormation. The CloudFormation will look like

{
    "LabelTemplate": "SampleLabel",
    "ResourceTypes": [
        "AWS::CloudWatch::Metric",
        "AWS::Logs::LogGroup",
        "AWS::XRay::Trace"
    ],
    "SinkIdentifier": "arn:aws:oam:eu-north-1:1111111111111111:sink/EXAMPLE-206d-4daf-9b42-1e17d5f145ef"
}

This will create a link between source and monitor account

b. Using URL: Login to source account and paste the url. This will pre-populate the page with all the details, select Link and Confirm.

c. Using CDK

new oam.CfnLink(this, 'SampleLink', {
          labelTemplate: 'SampleLabel',
          resourceTypes: [
                 'AWS::CloudWatch::Metric',
                 'AWS::Logs::LogGroup',
                 'AWS::XRay::Trace'
          ],
          sinkIdentifier: 'arn:aws:oam:eu-north-1:1111111111111111:sink/EXAMPLE-206d-4daf-9b42-1e17d5f145ef'
        });

Once done, all the selected resources i.e. Metric, LogGroup or X-Ray trace will be available in the monitoring accounts from source accounts.

Things to Consider

• Never use ‘*’ as resource without account id or organization id
• If ‘*’ is used without any check in place, this will open a risk on Monitoring account to allow any source account to establish a Link
• Always use list of account id’s or organization id in monitoring account configuration

Bonus

The above will provide cross-account observability to view and monitor logs, metrics and X-Ray from source accounts to a single Monitoring account. This section will discuss transporting the log and metrics data to data warehouse to analyze the extrapolate using QuickSight.

This will need below configuration in Monitoring account.

1. Create S3 Bucket: This bucket will store metrics and logs from source accounts
2. Configure Metrics Stream: In order to stream and store the data from source accounts into monitoring account S3 bucket, you’ll need to configure Metrics Stream. In order to do so:
   Navigate to CloudWatch -> Metrics -> Streams, select Create a metric stream
   Click on include source account metrics and select namespaces (or all namespaces) as required.
   Follow different configuration as required
3. Once configured, a new metric stream will be created. This will stream the data from all linked source accounts to the defined S3 bucket.
4. You can configure an ETL job on top of S3 data (Or use a firehose to do this as well)
5. Use QuickSight or any other visualization application to visualize collected data
6. Make decision!!

You can connect with me on Linkedin and Medium