• Home
  • About
  • Résumé
  • RunLog
  • Posts
    • All Posts
    • All Tags

Terraform and AWS ECS Execute Command on Fargate

20 Dec 2021

Reading time ~6 minutes

While we were setting up bastion on ECS fargate at work, I decided to look into AWS ECS Execute Command. This feature seems relatively new to the scene, so I decided to write down what steps I took to get this working.

We are setting up the ECS services to support ECS Execute command and Cloudwatch logging.

  • Requirements
  • VPC Endpoint
  • ECS Service
    • Service Resources
    • Task IAM Policy Resources
    • Task Definition
    • User IAM Policy Resources
  • Logging and Auditing
    • Cluster Resources
  • Tools
    • AWS CLI
  • Terraform Modules
  • References

Requirements

  • AWS CLI
  • Session Manager plugin

VPC Endpoint

In order for us to access the fargate containers, we will need to setup a VPC Endpoint for ssmmessages.

Notice! The VPC endpoint inbound rule needs https port 443 open in order for the fargate tasks to communicate.

resource "aws_vpc_endpoint" "this" {
    private_dns_enabled   = true
    security_group_ids    = [
        "sg-*****",
    ]
    service_name          = "com.amazonaws.us-west-2.ssmmessages"
    subnet_ids            = [
        "subnet-*****",
        "subnet-*****",
        "subnet-*****",
    ]
    vpc_endpoint_type     = "Interface"
    ...
}

ECS Service

Notice! In order to enable logging to cloudwatch, your container images will need to have script(1) and cat(1). On Alpine/CentOS, the package is util-linux, bsdutils for Debian based distros.

Service Resources

Enable enable_execute_command in aws_ecs_service.

resource "aws_ecs_service" "service" {
    launch_type                        = "FARGATE"
    enable_execute_command             = true
    ...

Task IAM Policy Resources

The policy will need the following Permissions.

  • ssmmessages:
    • These are required for ecs-exec-command to work.
  • kms:
    • They are used for both cloudwatch and ecs-exec-command. See Logging and Auditing
  • logs:
    • They are required to write logs to cloudwatch. See Logging and Auditing
resource "aws_iam_role_policy" "task_execution" {
    policy = jsonencode(
        {
            Statement = [
                {
                    Action   = [
                        "kms:Decrypt",
                        "kms:DescribeKey",
                        "kms:Encrypt",
                        "kms:GenerateDataKey*",
                        "kms:ReEncrypt*",
                        "logs:CreateLogGroup",
                        "logs:CreateLogStream",
                        "logs:Describe*",
                        "logs:Get*",
                        "logs:PutLogEvents",
                        "ssmmessages:CreateControlChannel",
                        "ssmmessages:CreateDataChannel",
                        "ssmmessages:OpenControlChannel",
                        "ssmmessages:OpenDataChannel",
                        ...
                    ]
                    Effect   = "Allow"
                    Resource = "*"
                },
            ]
        }
    )
}

Task Definition

In the task definition of the services, you will need to enable initProcessEnabled.

$ aws ecs describe-task-definition --task-definition qa-bastion-01 | jq -r '.taskDefinition.containerDefinitions[] | {linuxParameters: .linuxParameters}'
{
  "linuxParameters": {
    "initProcessEnabled": true
    }
  ]
}

User IAM Policy Resources

In order for users to run ecs-execute-command, you will need to create a policy and attach it to the user/group IAM.

From what I can tell, we do need to grant the following Actions.

  • kms:GenerateDataKey
  • ecs:ExecuteCommand

What I am also doing is ONLY allow if the container-name does NOT match bastion.

Notice! I THINK theres a bug in the AWS documentation regarding Resource. I have to have all 4 in order for it to work: :task/* and :cluster/*.

resource "aws_iam_policy" "iam_policy" {
    path        = "/"
    policy      = jsonencode(
        {
            Statement = [
                {
                    Action   = "kms:GenerateDataKey"
                    Resource = "arn:aws:kms:us-west-2:*****:key/*****"
                },
                {
                    Action    = "ecs:ExecuteCommand"
                    Condition = {
                        StringNotLike = {
                            ecs:container-name = [
                                "qa-bastion-*",
                            ]
                        }
                    }
                    Effect    = "Allow"
                    Resource  = [
                        "arn:aws:ecs:us-west-2:*****:task/qa-default-01/*",
                        "arn:aws:ecs:us-west-2:*****:task/qa-default-01",
                        "arn:aws:ecs:us-west-2:*****:cluster/qa-default-01/*",
                        "arn:aws:ecs:us-west-2:*****:cluster/qa-default-01",
                    ]
                },
            ]
        }
    )
}

Logging and Auditing

We want to enable auditing in cloudtrail and logging in cloudwatch log. For aws_ecs_cluster we set this up under configuration section.

Cluster Resources

There are some settings needed in the ECS cluster, Here are the essentials. We are enabling cloudwatch log with KMS.

resource "aws_ecs_cluster" "ecs_cluster" {
    ...
    configuration {
        execute_command_configuration {
            kms_key_id = "arn:aws:kms:us-west-2:*****:key/*****"
            logging    = "OVERRIDE"

            log_configuration {
                cloud_watch_encryption_enabled = true
                cloud_watch_log_group_name     = "/ecs/cluster/qa-default-01"
                s3_bucket_encryption_enabled   = false
            }
        }
    }
}

Cloudwatch log with KMS key.

resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
    id                = "/ecs/cluster/qa-default-01"
    kms_key_id        = "arn:aws:kms:us-west-2:*****:key/*****"
    name              = "/ecs/cluster/qa-default-01"
    retention_in_days = 30
    ...
}

KMS key requires the following policy: kms: and access to logs resource.

resource "aws_kms_key" "kms_key" {
    deletion_window_in_days            = 7
    description                        = "ECS Cluster Key - qa-default-01"
    is_enabled                         = true
    key_usage                          = "ENCRYPT_DECRYPT"
    policy                             = jsonencode(
        {
            Statement = [
                {
                    Action    = "kms:*"
                    Effect    = "Allow"
                    Principal = {
                        AWS = "arn:aws:iam::*****:root"
                    }
                    Resource  = "*"
                },
                {
                    Action    = [
                        "kms:ReEncrypt*",
                        "kms:GenerateDataKey*",
                        "kms:Encrypt*",
                        "kms:Describe*",
                        "kms:Decrypt*",
                    ]
                    Effect    = "Allow"
                    Principal = {
                        Service = "logs.us-west-2.amazonaws.com"
                    }
                    Resource  = "*"
                },
            ]
        }
    )
}

resource "aws_kms_alias" "kms_alias" {
    id             = "alias/qa-default-01-key"
    name           = "alias/qa-default-01-key"
    target_key_id  = "*****"
}

Tools

I wrote a tool that interacts with ecs-exec-command called ecsrun. This along with other tools in the aws-utils repo.

For validating requirements are met, Amazon ECS Exec Checker is very useful.

$ bash <(curl -Ls https://raw.githubusercontent.com/aws-containers/amazon-ecs-exec-checker/main/check-ecs-exec.sh) qa-default-01 *****
-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/local/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.4.6 Python/3.9.9 Darwin/21.1.0 source/x86_64 prompt/off)
  Session Manager Plugin | OK (1.2.279.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-west-2
Cluster: qa-default-01
Task   : *****
-------------------------------------------------------------
  Cluster Configuration  |
     KMS Key       : arn:aws:kms:us-west-2:*****:key/*****
     Audit Logging : OVERRIDE
     S3 Bucket Name: Not Configured
     CW Log Group  : /ecs/cluster/qa-default-01, Encryption Enabled: true
  Can I ExecuteCommand?  | arn:aws:iam::*****:role/Admin
     ecs:ExecuteCommand: allowed
     kms:GenerateDataKey: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "qa-bastion-01"
         ...
    ----------
      Init Process Enabled (qa-bastion-01:20)
    ----------
         1. Enabled - "qa-bastion-01"
         ...
    ----------
      Read-Only Root Filesystem (qa-bastion-01:20)
    ----------
         1. Disabled - "qa-bastion-01"
         ...
  Task Role Permissions  | arn:aws:iam::*****:role/qa-bastion-01-task-execution
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
     -----
     kms:Decrypt: allowed
     -----
     logs:DescribeLogGroups: allowed
     logs:CreateLogStream: allowed
     logs:DescribeLogStreams: allowed
     logs:PutLogEvents: allowed
  VPC Endpoints          |
    Found existing endpoints for vpc-*****:
      - com.amazonaws.us-west-2.ssmmessages
      ...

AWS CLI

Check for ExecuteCommand status.

$ ecslookup qa-default-01 | \
    while read line; do \
        aws ecs describe-services --cluster qa-default-01 --service ${line} | \
        jq -r '.services[] | [.serviceName, .status, .launchType, .platformVersion, .createdAt, .enableExecuteCommand] | join(",")'; done | \
        column -ts, | \
        sort
qa-bastion-01               ACTIVE  FARGATE  1.4.0  2021-12-27T13:03:28.958000-08:00  true
...
qa-notification-01          ACTIVE  FARGATE  1.4.0  2021-03-12T14:47:07.209000-08:00  false
qa-notification-02          ACTIVE  FARGATE  1.4.0  2021-11-09T12:18:50.024000-08:00  false
...
qa-rest-green               ACTIVE  FARGATE  1.4.0  2021-03-12T13:56:35.764000-08:00  true

Auditing with Cloudwatch and Cloudtrail.

$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ExecuteCommand | \
jq -r '.Events[] | .CloudTrailEvent' | \
jq -r '[.userIdentity.arn, .eventTime, .sourceIPAddress, (.requestParameters | to_entries[] | .value)] + [if .responseElements != null then .responseElements.session.streamUrl | gsub(".*data-channel/(?<match>[a-z0-9-]+)?.*"; .match) else "not-available" end] | join(",")' | \
column -ts, | \
head -n 2
arn:aws:sts::*****:assumed-role/Admin/cwong@your-company.com  2021-12-20T16:32:07Z  199.***.237.28  qa-default-01  qa-bastion-01             /bin/sh  true  *****-service-task-*****  not-available
arn:aws:sts::*****:assumed-role/Admin/cwong@your-company.com  2021-12-20T06:17:04Z  199.***.237.28  qa-default-01  qa-recommendation-api-01  /bin/sh  true  *****-service-task-*****  ecs-execute-command-0ee3598c40e47bf6e

$ aws logs describe-log-streams --log-group-name /ecs/cluster/qa-default-01 --log-stream-name-prefix ecs-execute-command
{
    "logStreams": [
        {
            "logStreamName": "ecs-execute-command-0ee3598c40e47bf6e",
            "creationTime": 1639966898013,
            ...
        },
...

$ aws logs get-log-events --log-group-name /ecs/cluster/qa-default-01 --log-stream-name ecs-execute-command-0ee3598c40e47bf6e
{
    "events": [
        {
            "timestamp": 1639966897981,
            "message": "Script started on 2021-12-20 02:21:36+00:00\nsh-4.4# \r\nsh-4.4# exit\r\nexit\r\n\nScript done on 2021-12-20 02:21:36+00:00",
            "ingestionTime": 1639966898034
        }
    ],
    ...
}

Invoke ecs-exec-command.

$ aws ecs execute-command --cluster qa-default-01 --container qa-bastion-01 --task ***** --interactive --command /bin/sh

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.

Starting session with SessionId: ecs-execute-command-0d8772eca1533f471
This session is encrypted using AWS KMS.
/ #

Terraform Modules

  • vpc-endpoints
  • ecs-cluster-fargate
  • ecs-service

References

  • Using Amazon ECS Exec for debugging
  • 5 Steps: Using Amazon ECS Exec to pass through Fargate/ECS into containers

technologydocdevopsawsterraformecsfargate Share Tweet +1