Using waxctl
to Run your Dataflow on AWS#
As well as facilitating deployment on Kubernetes, Waxctl provides an easy path to deploy dataflows on AWS EC2 instances.
Installation#
To install Waxctl visit here.
Dataflow Lifecycle#
You can manage the lifecycle of your dataflow with Waxctl across the following phases:
Deployment
Current Status (Running)
Restarting
Deletion
In the following sections we are going to cover each of these phases.
Available AWS Sub-commands#
When using Waxctl with cloud specific resources, you will add the
cloud name in front of the sub-command. The aws
command has the
following sub-commands:
deploy
list
stop
start
delete
Running waxctl aws --help
will show further details for these.
$ waxctl aws --help
Waxctl Pro version 0.12.0
Manage dataflows running on AWS EC2 instances
Usage:
waxctl aws [flags]
waxctl aws [command]
Available Commands:
delete terminate an EC2 instance created by waxctl
deploy create an EC2 instance running a dataflow
list list AWS EC2 instances created by waxctl
start start an EC2 instance created by waxctl
stop stop an EC2 instance created by waxctl
Flags:
-h, --help help for aws
Global Flags:
--debug enable verbose output
Use "waxctl aws [command] --help" for more information about a command.
Waxctl and AWS CLI#
To use Waxctl on AWS you need to have installed and configured the AWS CLI. You can follow the instructions here.
By default, Waxctl is going to use the AWS CLI default region but you
can use another one just by setting the --region
flag in your Waxctl
commands.
Run a Dataflow in an EC2 instance#
To deploy a dataflow you just need to run waxctl aws deploy
passing
the path of your python script as an argument. If left unset, Waxctl
will use the default name for your dataflow, which is bytewax
.
In our example we are going to deploy a dataflow called my-dataflow
:
$ waxctl aws deploy /var/bytewax/examples/basic.py --name my-dataflow
Created policy arn:aws:iam::111111111111:policy/Waxctl-EC2-my-dataflow-Policy
Created role Waxctl-EC2-my-dataflow-Role
Created my-dataflow instance with ID i-040f98b9160d2d158 running /var/bytewax/examples/basic.py script
In the above example, Waxctl used the default values for all of the
flags except for the name
flag. Waxctl allows you to configure a
wide range of characteristics of your dataflow.
As you can see in the output above, Waxctl created an IAM policy and role. That will allow the EC2 instance to store Cloudwatch logs and start sessions through Systems Manager.
We can see the complete list of available flags with the waxctl aws deploy
help command.
$ waxctl aws deploy -h
Waxctl Pro version 0.12.0
Deploy a dataflow to a new EC2 instance.
The deploy command expects one argument, which is the path or URI of your python dataflow file.
By default, Waxctl creates a policy and a role that will allow the EC2 instance to store Cloudwatch logs and start sessions through Systems Manager.
Examples:
# The default command to deploy a dataflow named "bytewax" in a new EC2 instance running my-dataflow.py file.
waxctl aws deploy my-dataflow.py
# Deploy a dataflow named "bytewax" in a new EC2 instance running dataflow.py file.
waxctl aws deploy https://raw.githubusercontent.com/my-user/my-repo/main/dataflow.py
# Deploy a dataflow named "custom" using specific security groups and instance profile
waxctl aws deploy dataflow.py --name custom \
--security-groups-ids "sg-006a1re044efb2d23" \
--principal-arn "arn:aws:iam::1111111111:instance-profile/my-profile"
Usage:
waxctl aws deploy [PATH] [flags]
Flags:
-P, --associate-public-ip-address associate a public IP address to the EC2 instance (default true)
-m, --detailed-monitoring specifies whether detailed monitoring is enabled for the EC2 instance
-E, --environment-variables strings environment variables to set when running the python script. The format must be KEY=VALUE
-e, --extra-tags strings extra tags to apply to the EC2 instance. The format must be KEY=VALUE
-h, --help help for deploy
-t, --instance-type string EC2 instance type to be created (default "t2.micro")
-k, --key-name string name of an existing key pair
-n, --name string name of the EC2 instance to deploy the dataflow (default "bytewax")
-o, --old-entrypoint needed if the version of Bytewax is previous than 0.16.0
-p, --principal-arn string principal ARN to assign to the EC2 instance
--profile string AWS cli configuration profile
-f, --python-file-name string python script file to run. Only needed when [PATH] is a tar file
--python-package use this flag if your tarball file is a python package
--region string AWS region
-r, --requirements-file-name string requirements.txt file if needed
--save-cloud-config save cloud-config file to disk for troubleshooting purposes
-S, --security-groups-ids strings security groups Ids to assign to the EC2 instance
-s, --subnet-id string the ID of the subnet to launch the EC2 instance into
-c, --system-setup-file-name string sh script file to be run before the installation of the requirements
Global Flags:
--debug enable verbose output
We suggest paying special attention to the requirements-file-name
flag because normally you will want to specify a requirements.txt
file with the needed libraries to run your dataflow program.
Default IAM Role#
As we mentioned, Waxctl creates an IAM policy and role to allow your EC2 instance to store CloudWatch logs and to start Systems Manager sessions. In case you need to use a custom IAM role, here we show you what are the permissions that the policy created by Waxctl has:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel",
"ssm:UpdateInstanceInformation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetEncryptionConfiguration"
],
"Resource": "*"
}
]
}
So, your role must have those permissions to keep both features working. We recommend attaching to your Role a Customer managed policy having only those permissions and maybe with an explicit name like “Bytewax-Policy” or “Waxctl-Policy”.
Getting Dataflow Information#
You can query which dataflows are deployed on EC2 instances in your
AWS account using the waxctl aws list
sub-command. By default the
output will be a table with this information:
$ waxctl aws ls
Dataflow Python File Name VM State Launch Time
my-dataflow /var/bytewax/examples/basic.py running 2022-10-03 13:34:02 +0000 UTC
You can use the --verbose
flag to get more details of each dataflow:
$ waxctl aws ls --verbose
[
{
"instanceId": "i-040f98b9160d2d158",
"instanceType": "t2.micro",
"instanceState": "running",
"name": "my-dataflow",
"iamInstanceProfile": "arn:aws:iam::111111111111:instance-profile/Waxctl-EC2-my-dataflow-InstanceProfile",
"keyName": "",
"launchTime": "2022-10-03 13:34:02 +0000 UTC",
"detailedMonitoring": "disabled",
"privateDnsName": "ip-172-31-3-116.us-west-2.compute.internal",
"privateIpAddress": "172.31.3.116",
"pythonFileName": "/var/bytewax/examples/basic.py",
"publicDnsName": "ec2-18-237-167-156.us-west-2.compute.amazonaws.com",
"publicIpAddress": "18.237.167.156",
"securityGroupsIds": [
"sg-1e2c2c27"
],
"subnetId": "subnet-64840439",
"tags": [
{
"key": "bytewax.io/waxctl-version",
"value": "0.5.1"
},
{
"key": "Name",
"value": "my-dataflow"
},
{
"key": "bytewax.io/managed-by",
"value": "waxctl"
},
{
"key": "bytewax.io/waxctl-managed-instance-profile-name",
"value": "Waxctl-EC2-my-dataflow-InstanceProfile"
},
{
"key": "bytewax.io/waxctl-managed-policy-arn",
"value": "arn:aws:iam::111111111111:policy/Waxctl-EC2-my-dataflow-Policy"
},
{
"key": "bytewax.io/waxctl-managed-role-name",
"value": "Waxctl-EC2-my-dataflow-Role"
},
{
"key": "bytewax.io/waxctl-python-filename",
"value": "/var/bytewax/examples/basic.py"
}
],
"consoleDetails": "https://us-west-2.console.aws.amazon.com/ec2/v2/home?region=us-west-2#InstanceDetails:instanceId=i-040f98b9160d2d158",
"logs": "https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Fbytewax$252Fsyslog/log-events/my-dataflow$3FfilterPattern$3Dbytewax"
}
]
As you can see, there are a lot of details including links to access instance information and your dataflow logs in the AWS web console.
This is the help text of the list
command:
$ waxctl aws ls --help
Waxctl Pro version 0.12.0
List EC2 instances created by waxctl.
Examples:
# List EC2 instances running dataflows.
waxctl aws list
# List EC2 instance named "my dataflow" in region us-east-1.
waxctl aws list --name "my dataflow" --region us-east-1
Usage:
waxctl aws list [flags]
Aliases:
list, ls
Flags:
-h, --help help for list
-n, --name string name of the EC2 instance to find.
--profile string AWS cli configuration profile.
--region string AWS region.
-v, --verbose return detailed information of the EC2 instances.
Global Flags:
--debug enable verbose output
Stopping and Starting a Dataflow#
In case you need to pause your dataflow, there are two commands to
manage that: stop
and start
. These control the EC2 instance state
where your dataflow is running.
Following the example, you can stop the dataflow EC2 instance with this command:
$ waxctl aws stop --name my-dataflow
EC2 instance my-dataflow with ID i-040f98b9160d2d158 stopped.
So, if you run the list
command, you will see something like this:
$ waxctl aws ls
Dataflow Python File Name VM State Launch Time
my-dataflow /var/bytewax/examples/basic.py stopping 2022-10-03 13:34:02 +0000 UTC
And a few seconds later, the list
command is going to show the state
stopped
:
$ waxctl aws ls
Dataflow Python File Name VM State Launch Time
my-dataflow /var/bytewax/examples/basic.py stopped 2022-10-03 13:34:02 +0000 UTC
You can use the start
command to start the EC2 instance again:
$ waxctl aws start --name my-dataflow
EC2 instance my-dataflow with ID i-040f98b9160d2d158 started.
You can change any of the flags of your dataflow.
Removing a Dataflow#
To terminate the EC2 instance where your dataflow is running you need
to run waxctl aws delete
while passing the name of the dataflow as a
parameter.
To run a dry-run delete of our dataflow example we can run this:
$ waxctl aws delete --name my-dataflow
EC2 instance my-dataflow with ID i-040f98b9160d2d158 found in us-west-2 region.
--yes flag is required to terminate it.
And if we want to actually delete the dataflow we must add the --yes
flag:
$ waxctl aws rm --name my-dataflow --yes
Role Waxctl-EC2-my-dataflow-Role deleted.
Policy arn:aws:iam::111111111111:policy/Waxctl-EC2-my-dataflow-Policy deleted.
EC2 instance my-dataflow with ID i-040f98b9160d2d158 has been terminated.
Note that we used rm
in the last command, which is an alias of
delete
. Many of the Waxctl sub-commands have an alias and you can
see them in the help.
How it works internally#
As you can imagine, Waxctl uses the AWS API to manage EC2 instances, IAM policies, and roles.
The operating system of EC2 instances created by Waxctl is Ubuntu 20.04 LTS.
Waxctl relies on
Cloud-init, a standard
multi-distribution method for cross-platform cloud instance
initialization. Using Cloud-init, Waxctl configures a Linux service
which is going to run pip install -r /home/ubuntu/bytewax/requirements.txt
and after that run your python
dataflow program.
As we mentioned before, you can specify your own requirements.txt
file using the --requirements-file-name
flag. If you don’t, Waxctl
is going to put only bytewax
as a requirement.
Besides setting the service that runs your dataflow, Waxctl configures settings to push the syslog logs to CloudWatch. With that enabled, you can see your dataflow stdout and stderr in CloudWatch.
Troubleshooting#
You have two ways to see what’s going on with your dataflow program: viewing logs and connect to the EC2 instance.
Logs#
Since Waxctl runs your dataflow program a Linux service and all syslog logs are sent to CloudWatch, you can see your dataflow logs directly in CloudWatch.
When you run waxctl aws ls --verbose
you get a link to CloudWatch
Logs Viewer in AWS Web Console filtering by your EC2 instance and your
dataflow. Like this:
...
"logs": "https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Fbytewax$252Fsyslog/log-events/my-dataflow$3FfilterPattern$3Dbytewax"
...
Connecting to the EC2 Instance#
By default Waxctl will create an IAM Policy and a IAM Role to allow
you to use AWS Systems Manager to connect to the EC2 instance. To do
that, you need to know the instance ID (you can get it running waxctl aws ls --verbose
) and then run this:
$ aws ssm start-session --target i-0a04d5e18c1df4c90
You may want to check:
/home/ubuntu/bytewax - where your requirements and python script are copied.
systemctl status bytewax-dataflow.service
- Linux service that runs your dataflow.df -H /
- File system information.top
- Processes information.
You can install any profiling or debugging tool and use it. Also you
could modify your script and restart the bytewax-dataflow.service
running:
systemctl restart bytewax-dataflow.service
A Production-like Example - Requires Waxctl Pro#
In case your dataflow needs access to other AWS managed services, like MKS, you will probably want to use your own Security Group and IAM configuration.
In a production environment, the EC2 instance should be running in a specific Subnet, commonly a private one so we are going to instruct Waxctl to not associate a public IP address and to create the EC2 in a concrete Subnet.
So, this is an example Waxctl command for the described scenario:
$ waxctl aws deploy my-dataflow.py \
--name=production-dataflow \
--requirements-file-name=requirements.txt \
--instance-type=m5.xlarge \
--security-groups-ids=sg-0eae80fc9370624f7 \
--principal-arn=arn:aws:iam::111111111111:instance-profile/production-profile \
--subnet-id=subnet-03ec94a8d7d22d8e9 \
--debug
The output of that deploy will be like this:
2024/09/03 10:31:05 Analylics - information to send:
{"waxctl_version":"0.12.0","waxctl_tier":"Pro","platform":"amd64","os":"linux","command":"aws","subcommand":"deploy"}
2024/09/03 10:31:05 Analylics - duration: 942.50645ms
2024/09/03 10:31:05 Validating parameters...
2024/09/03 10:31:05 Getting AWS region us-west-2 from default configuration
2024/09/03 10:31:05 Getting AWS CLI configuration...
2024/09/03 10:31:07 Creating cloud-init config file...
2024/09/03 10:31:08 Selected AMI - Name: ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220924 - ImageId: ami-07eeacb3005b9beae
Created production-dataflow instance with ID i-0409c71c2cc1b249f running my-dataflow.py script