Auto AWS

The true power (and efficiency) of the cloud can only be realised if you orchestrate infrastructure as and when needed. This is usually done at scale through Kubernetes (K8s). However for small projects K8 is expensive and an overkill.

Auto AWS is a lightweight python library that can create your AWS infrastructure, starting with EC2. The code can be found on my Github

Usage

1
2
3
4
5
6
7
8

from ec2 import AutoEC2
ec2 = AutoEC2()

# show instances
ec2.instances 

# create a new instance based on config file
inst = ec2.create("config.cfg")

In case you are wondering the config file looks like this

[EC2]
tag=name
ami=ami-005a087ff43033c77
key_pair=someKey
security_group=ec2-sg
max_bid=1.0
type=r5.2xlarge
region=us-east-1
product_description=Linux/UNIX
username=ubuntu
public_ip_address=x.x.x.x
iam_role=arn:aws:iam::888888888:instance-profile/name
valid_hours=6

To run jobs on start up of the machine, save bash commands in userdata.txt (in the same folder as your script) Eg.

1
2
3
4

#!/bin/bash
aws ecr get-login-password dkr.ecr.us-east-1.amazonaws.com
sleep 5
docker run -d jupyter/datascience-notebook python some_script.py

When you launch it, it will tell you the spot price you are paying, along with helpful logs.

Spot price $0.1956
Spot request created, status: open
Waiting for spot provisioning
Waiting... None
Waiting... None
Instance allocated ,Id:  i-006071b23e23207f4

You can see your running instances too

1

ec2._get_instances()

            InstanceId PublicIpAddress InstanceType    state     name
0  i-099707305b10aa727         x.x.x.x     t2.micro  running  bastion

Once you are done with your job, you can destroy it in the same session.

1
2
3
4
5

job_done = # Some code to monitor if the job has finished. Eg. if a file on S3 was saved

if job_done:
    # destroy the instance once finished
    ec2.destroy(inst["InstanceId"])

Next steps

More EC2 automation.

1. Pass userdata.txt as string in ec2.create() rather than having to save to disk.
2. Automatically stop instance when done. Eg. monitor a S3 file which can signal to AutoEC2 that the job is complete and so it can kill the instance.