Making Amazon ECS Container Service as easy to use as Docker run

One of the reasons Docker caught fire was that it was soo easy to use. You could build and start a docker container in a matter of seconds. With Amazon ECS this is not so. You have to learn a whole new lingo (Clusters, Task definitions, Services and Tasks), spin up an ECS cluster, write a nasty looking JSON file or wrestle with a not-so-user-friendly UI before you have your container running in ECS.
In the blog we will show you that Amazon ECS can be as fast, by presenting you a small utility named ecs-docker-run which will allow you to start a Docker container almost as fast as with Docker stand-alone by interpreting the Docker run command line options. Together with a ready-to-run CloudFormation template, you can be up and running with Amazon ECS within minutes!

ECS Lingo

Amazon ECS uses different lingo than Docker people, which causes confusion. Here is a short translation:

Cluster – one or more Docker Hosts.
Task Definition – A JSON representation of a docker run command line.
Task – A running docker instance. When the instance stops, the task is finished.

Service – A running docker instance, when it stops, it is restarted.
In the basis that is all there is too it. (Cutting a few corners and skimping on a number of details).
Once you know this, we are ready to use ecs-docker-run.

ECS Docker Run

ecs-docker-run is a simple command line utility to run docker images on Amazon ECS. To use this utility you can simply type something familiar like:

[code]
ecs-docker-run \
--name paas-monitor \
--env SERVICE_NAME=paas-monitor \
--env SERVICE_TAGS=http \
--env "MESSAGE=Hello from ECS task" \
--env RELEASE=v10 \
-P \
mvanholsteijn/paas-monitor:latest
[/code]

substituting the ‘docker run’ with ‘ecs-docker-run’.
Under the hood, it will generate a task definition and start a container as a task on the ECS cluster. All of the following Docker run command line options are functionally supported.

-P	publishes all ports by pulling and inspecting the image.
–name	the family name of the task. If unspecified the name will be derived from the image name.
-p	add a port publication to the task definition.
–env	set the environment variable.
–memory	sets the amount of memory to allocate, defaults to 256
–cpu-shares	set the share cpu to allocate, defaults to 100
–entrypoint	changes the entrypoint for the container
–link	set the container link.
-v	set the mount points for the container.
–volumes-from	set the volumes to mount.

All other Docker options are ignored as they refer to possibilities NOT available to ECS containers. The following options are added, specific for ECS:

–generate-only	will only generate the task definition on standard output, without starting anything.
–run-as-service	runs the task as service, ECS will ensure that ‘desired-count’ tasks will keep running.
–desired-count	specifies the number tasks to run (default = 1).
–cluster	the ECS cluster to run the task or service (default = cluster).

Hands-on!

In order to proceed with the hands-on part, you need to have:

jq installed
aws CLI installed (version 1.7.44 or higher)
aws connectivity configured

docker connectivity configured (to a random Docker daemon).

checkout ecs-docker-run

Get the ecs-docker-run sources by typing the following command:

[code]
git clone git@github.com:mvanholsteijn/ecs-docker-run.git
cd ecs-docker-run/ecs-cloudformation
[/code]

import your ssh key pair

To look around on the ECS Cluster instances, import your public key into Amazon EC2, using the following command:

[code]
aws ec2 import-key-pair \
--key-name ecs-$USER-key \
--public-key-material "$(ssh-keygen -y -f ~/.ssh/id_rsa)"
[/code]

create the ecs cluster autoscaling group

In order to create your first cluster of 6 docker Docker Hosts, type the following command:

[code]
aws cloudformation create-stack \
--stack-name ecs-$USER-cluster \
--template-body "$(<ecs.json)" \
--capabilities CAPABILITY_IAM \
--parameters \
ParameterKey=KeyName,ParameterValue=ecs-$USER-key \
ParameterKey=EcsClusterName,ParameterValue=ecs-$USER-cluster
[/code]

This cluster is based upon the firstRun cloudformation definition, which is used when you follow the Amazon ECS wizard.

And wait for completion…

Wait for completion of the cluster creation, by typing the following command:

[code]
function waitOnCompletion() {
STATUS=IN_PROGRESS
while expr "$STATUS" : '^.*PROGRESS' > /dev/null ; do
sleep 10
STATUS=$(aws cloudformation describe-stacks \
--stack-name ecs-$USER-cluster | jq -r '.Stacks[0].StackStatus')
echo $STATUS
done
}
waitOnCompletion
[/code]

Create the cluster

Unfortunately, CloudFormation does (not) yet allow you to specify the ECS cluster name, so need to manually create the ECS cluster, by typing the following command:

[code]
aws ecs create-cluster --cluster-name ecs-$USER-cluster
[/code]

You can now manage your hosts and tasks from the Amazon AWS EC2 Container Services console.

Run the paas-monitor

Finally, you are ready to run any docker image on ECS! Type the following command to start the paas-monitor.

[code]
../bin/ecs-docker-run --run-as-service \
--number-of-instances 3 \
--cluster ecs-$USER-cluster \
--env RELEASE=v1 \
--env MESSAGE="Hello from ECS" \
-p :80:1337 \
mvanholsteijn/paas-monitor
[/code]

Get the DNS name of the Elastic Load Balancer

To see the application in action, you need to obtain the DNS name of the Elastic Load Balancer. Type the following commands:

[code]
# Get the Name of the ELB created by CloudFormation
ELBNAME=$(aws cloudformation describe-stacks --stack-name ecs-$USER-cluster | \
jq -r '.Stacks[0].Outputs[] | select(.OutputKey =="EcsElbName") | .OutputValue')
# Get the DNS from of that ELB
DNSNAME=$(aws elb describe-load-balancers --load-balancer-names $ELBNAME | \
jq -r .LoadBalancerDescriptions[].DNSName)
[/code]

Open the application

Finally, we can obtain access to the application.

[code]
open https:// $DNSNAME
[/code]

And it should look something like this..

host	release	message	# of calls	avg response time	last response time
b6ee7869a5e3:1337	v1	Hello from ECS from release v1; server call count is 82	68	45	36
4e09f76977fe:1337	v1	Hello from ECS from release v1; server call count is 68	68	41	38
65d8edd41270:1337	v1	Hello from ECS from release v1; server call count is 82	68	40	37

Perform a rolling upgrade

You can now perform a rolling upgrade of your application, by typing the following command while keeping your web browser open at https:// $DNSNAME:

[code]
../bin/ecs-docker-run --run-as-service \
--number-of-instances 3 \
--cluster ecs-$USER-cluster \
--env RELEASE=v2 \
--env MESSAGE="Hello from Amazon EC2 Container Services" \
-p :80:1337 \
mvanholsteijn/paas-monitor
[/code]

The result should look something like this:

host	release	message	# of calls	avg response time	last response time
b6ee7869a5e3:1337	v1	Hello from ECS from release v1; server call count is 124	110	43	37
4e09f76977fe:1337	v1	Hello from ECS from release v1; server call count is 110	110	41	35
65d8edd41270:1337	v1	Hello from ECS from release v1; server call count is 124	110	40	37
ffb915ddd9eb:1337	v2	Hello from Amazon EC2 Container Services from release v2; server call count is 43	151	9942	38
8324bd94ce1b:1337	v2	Hello from Amazon EC2 Container Services from release v2; server call count is 41	41	41	38
7b8b08fc42d7:1337	v2	Hello from Amazon EC2 Container Services from release v2; server call count is 41	41	38	39

Note how the rolling upgrade is a bit crude. The old instances stop receiving requests almost immediately, while all requests seem to be loaded onto the first new instance.

You do not like the ecs-docker-run script?

If you do not like the ecs-docker-run script, do not dispair. Below are the equivalent Amazon ECS commands to do it without the hocus-pocus script…

Create a task definition

This is the most difficult task: Manually creating a task definition file called ‘manual-paas-monitor.json’ with the following content:

[code]
{
"family": "manual-paas-monitor",
"containerDefinitions": [
{
"volumesFrom": [],
"portMappings": [
{
"hostPort": 80,
"containerPort": 1337
}
],
"command": [],
"environment": [
{
"name": "RELEASE",
"value": "v3"
},
{
"name": "MESSAGE",
"value": "Native ECS Command Line Deployment"
}
],
"links": [],
"mountPoints": [],
"essential": true,
"memory": 256,
"name": "paas-monitor",
"cpu": 100,
"image": "mvanholsteijn/paas-monitor"
}
],
"volumes": []
}
[/code]

Register the task definition

Before you can start a task it has to be registered at ECS, by typing the following command:

[code]
aws ecs register-task-definition --cli-input-json "$(<paas-monitor.json)"
[/code]

Start a service

Now start a service based on this definition, by typing the following command:

[code]
aws ecs create-service \
--cluster ecs-$USER-cluster \
--service-name manual-paas-monitor \
--task-definition manual-paas-monitor:1 \
--desired-count 1
[/code]

You should see a new row appear in your browser:

host	release	message	# of calls	avg response time	last response time
….
5ec1ac73100f:1337	v3	Native ECS Command Line Deployment from release v3; server call count is 37	37	37	36

Conclusion

Amazon EC2 Container Services has a higher learning curve than using plain Docker. You need to get passed the lingo, the creation of an ECS cluster on Amazon EC2 and most importantly the creation of the cumbersome task definition file. After that it is almost as easy to use as Docker run.
In return you get all the goodies from Amazon like Autoscaling groups, Elastic Load Balancers and multi-availability zone deployments ready to use in your Docker applications. So, check ECS out!

More Info

Checkout more information: