AWS - Create your own Cloud Cluster!
Welcome to the very first of our new blog series to teach and educate a greater audience about the hidden world of High Powered Computing (HPC) and the potential it holds for not only yourself as a user, but to the way of life in years to come.
Throughout this series, you will learn the importance of HPC in future applications, how to use a HPC cloud cluster, submitting jobs, and even complete your very first task! Let us begin:
Amazon Web Services (AWS)
AWS is an extensive platform that provides a wide range of applications, everything from databases, machine learning, AI and analytics. For our purposes in HPC, we will be using what is called a cloud cluster, to enable access to lots of very fast computers simultaneously, to help complete complex coding tasks quickly. Now before you get too excited (like I did), you will need to explicitly agree to not use this computer cluster for mining crypto! (Very sad I know).
Within the DeepLearning team, the cloud cluster is used to run deep learning algorithms, effectively teaching a computer human-like tasks such as image recognition. Fun fact the ‘hello world’ of deep learning is teaching a computer to identify hand-written numbers in a range of fonts and neatness. Within HPC, we use cloud clusters for our tasks that range from finding the fastest way to hack a password, to teaching drones how to navigate spaces by themselves.
The first thing you should do to follow along with these blogs is sign up here to an AWS EC2 account at the following link: https://aws.amazon.com/ec2/?did=ft_card&trk=ft_card
Before we begin there are a few key terms you will see in the tutorial:
IAM: Identity and access manager. A tool to allow users access to certain controls within AWS, not so important for us, but we need to assign ourselves all access.
EC2: This is the group of fast computers located in a server room that will run your code virtually!
Instance: Think of an instance as a clone for whatever you would like it to be. For our purposes, our instances will be a collection of computing resources such as CPUs, GPUs and memory. This will effectively create a mini computer in a given language that you will need to code your task in so that it can read it. We often use Ubuntu operating systems for our tasks we submit to cloud servers, and Linux as the coding language.
Now you’re ready to begin!
Setup
AWS SIGN UP
The first step will involve creation of a new AWS account to access services such as EC2.
Open the following link, create an account and login. https://portal.aws.amazon.com/billing/signup
COMPUTER SETUP
The following tutorial requires you to ssh (a method of connecting to a machine over a network) to an instance created by Amazon. If you are on Linux or Mac ssh should be installed however for Windows users installing Windows Subsystem for Linux is recommended.
KEYPAIR CREATION
AWS uses public-key cryptography to secure the login information for your instance. A Linux instance has no password; you use a key pair to log in to your instance securely. You specify the name of the key pair when you launch your instance, then provide the private key when you log in using SSH.
You can create one by using the Amazon EC2 console.
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2
From the top navigation bar, select the Asia Pacific (Sydney) Region for the security group.
3. In the navigation pane, choose Key Pairs.
4. Choose Create key pair.
5. The private key file is automatically downloaded by your browser. The base file name is the name you specified as the name of your key pair, and the file name extension is determined by the file format you chose. Save the private key file in a safe place.
6. If you will use an SSH client on a macOS or Linux computer to connect to your Linux instance, use the following command to set the permissions of your private key file so that only you can read it. “chmod 400 my-key-pair.pem” For the purposes of the tutorial I would do chmod 400 deepneuronblog.pem.
SECURITY GROUP SETUP
To manage connections for an instance we must first set up a security group.
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/, security groups are specific to a Region, so you should select the same Region in which you created your key pair.
In the left navigation pane, choose Security Groups.
3. Choose Create security group.
4. For Basic details, do the following:
Enter a name for the new security group and a description. Use a name that is easy for you to remember.
In the VPC list, leave the default VPC.
5. For Inbound rules, create rules that allow specific traffic to reach your instance. For example, use the following rules for a web server that accepts HTTP and HTTPS traffic. For more examples, see Security group rules for different use cases.
Choose Add rule. For Type, choose HTTP. For Source, choose Anywhere-IPv4.
Choose Add rule. For Type, choose HTTPS. For Source, choose Anywhere.
Choose Add rule. For Type, choose SSH. For Source, do one of the following:
Choose My IP to automatically add the public IPv4 address of your local computer.
Choose Custom and specify the public IPv4 address of your computer or network in CIDR notation. To specify an individual IP address in CIDR notation, add the routing suffix /32, for example, 203.0.113.25/32.
Warning
For security reasons, do not choose Anywhere for Source with a rule for SSH. This would allow access to your instance from all IP addresses on the internet. This is acceptable for a short time in a test environment, but it is unsafe for production environments.
6. For Outbound rules, keep the default rule, which allows all outbound traffic.
7. Choose Create security group.
LAUNCH AN INSTANCE
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
From the console dashboard, choose Launch Instance.
3. The Choose an Amazon Machine Image (AMI) page displays a list of basic configurations, called Amazon Machine Images (AMIs), that serve as templates for your instance. Select an HVM version of Amazon Linux 2. Notice that these AMIs are marked "Free tier eligible."
4. On the Choose an Instance Type page, you can select the hardware configuration of your instance. Select the t2.micro instance type, which is selected by default. The t2.micro instance type is eligible for the free tier.
5. On the Choose an Instance Type page, choose Review and Launch to let the wizard complete the other configuration settings for you.
6. On the Review Instance Launch page, under Security Groups, you'll see that the wizard created and selected a security group for you. You can use this security group, or alternatively you can select the security group that you created when getting set up using the following steps:
Choose Edit security groups.
On the Configure Security Group page, ensure that Select an existing security group is selected.
Select your security group from the list of existing security groups, and then choose Review and Launch.
7. On the Review Instance Launch page, choose Launch.
8. When prompted for a key pair, select Choose an existing key pair, then select the key pair that you created when getting set up.
Warning
Don't select Proceed without a key pair. If you launch your instance without a key pair, then you can't connect to it. When you are ready, select the acknowledgement check box, and then choose Launch Instances.
9. A confirmation page lets you know that your instance is launching. Choose View Instances to close the confirmation page and return to the console.
10. On the Instances screen, you can view the status of the launch. It takes a short time for an instance to launch. When you launch an instance, its initial state is pending. After the instance starts, its state changes to running and it receives a public DNS name. (If the Public IPv4 DNS column is hidden, choose the settings icon ( ) in the top-right corner, toggle on Public IPv4 DNS, and choose Confirm.
11. It can take a few minutes for the instance to be ready so that you can connect to it. Check that your instance has passed its status checks; you can view this information in the Status check column.
CONNECTING TO YOUR LINUX INSTANCE
For Mac and Linux users you can quickly see if you have SSH installed by typing ssh and enter. The following output or similar should show
For Windows users simply install Windows Subsystem for Linux or WSL.
To connect using your instance's public DNS name, enter the following command.
ssh -i /path/my-key-pair.pem ec2-user@my-instance-IPv4-address
An example for my case may be ssh -i deepneuronblog.pem ec2-user@0.12.34.56 where the ssh key is in my current directory
You should see a response like the following:
The authenticity of host 'ec2-198-51-100-1.compute-1.amazonaws.com (198-51-100-1)' can't be established.
ECDSA key fingerprint is l4UB/neBad9tvkgJf1QZWxheQmR59WgrgzEimCG6kZY.
Are you sure you want to continue connecting (yes/no)?
2. (Optional) Verify that the fingerprint in the security alert matches the fingerprint that you previously obtained in (Optional) Get the instance fingerprint. If these fingerprints don't match, someone might be attempting a "man-in-the-middle" attack. If they match, continue to the next step.
3. Enter yes.
You see a response like the following:
Warning: Permanently added 'ec2-198-51-100-1.compute-1.amazonaws.com' (ECDSA) to the list of known hosts.
After you've finished with the instance that you created for this tutorial, you should clean up by terminating the instance.
TO TERMINATE YOUR INSTANCE
In the navigation pane, choose Instances. In the list of instances, select the instance.
Choose Instance state, Terminate instance.
Choose Terminate when prompted for confirmation.
Amazon EC2 shuts down and terminates your instance. After your instance is terminated, it remains visible on the console for a short while, and then the entry is automatically deleted. You cannot remove the terminated instance from the console display yourself.Note: do not terminate the instance if you wish to continue using it for later tasks, instead it can be stopped by clicking Stop instance
Conclusion
You have successfully set up an Amazon EC2 instance and have started on the journey to learn what HPC is about. Future blog posts will build upon this instance utilising our newly created cloud cluster.
