Amazon Elastic Compute Cloud
The Elastic Compute Cloud (EC2) from Amazon provides dynamically scaling of compute resources for developers and enterprises. Clients maintain complete control of their computing resources. Amazon provides a wealth of options to obtain their services – many of which are very cost effect – including a “scale as you like and pay as you go” model. To obtain and boot new server instances, you simply choose from a list of instance types (of which there are quite a few but that is a good thang!).
Many different types of Linux images are available and you can bring your own. I’m currently using a Amazon Linux 2 image (think RHEL 7 type image) that provides a significant amount of processing power and RAM at a cost effective price on a 3 year contract – running this blog. More specifically, the EC2 instance running this blog today 02/07/2018 is a “t2.medium” 2 vCPUs – and 4GB RAM.
AWS allows the separate definition of security groups ( virtual firewalls ) that you can use to safe guard EC2 and other services. AWS EC2 provides default security groups with default attributes that block everything if you don’t define your own when you create an EC2 instance – so it is up to you to open it up by customizing the defaults or creating your own security groups from scratch. Security group definition or customizations are done like almost every other provisioning action from the AWS console via an easy to use GUI in your favorite browser.
Amazon Elastic Block Storage
Elastic Block Storage – Amazon EBS delivers persistent block-level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are automatically replicated within Availability Zones to protect from failure, providing high availability and durability at a reasonable price compared to the competition.
EBS provides SSD based block storage at two tier / pricing levels and spinning disks that are even more cost effective.
EBS / EC2 snapshots provide easy backups of your hard work at a reasonable price. Snapshots are incremental in nature, it is worth noting that purging older snapshots does not effect your ability to restore from newer snapshots.
Amazon Elastic IP and Route 53
Fixed IP for your own purposes / website etc. at a very very small money price. Route 53 provides DNS registration and DNS queries via Amazon DNS servers via and pretty easy to use and well documented web interface.
This blog uses Elastic IP for a static IP and Route 53 to support DNS registration and DNS queries.
Amazon Simple Storage Service
Simple Storage Service (S3) is a Cloud based storage service that offers unlimited data storage space via a straight forward Web services interface that stores and retrieves data from anywhere on the Web. Pricing is quite aggressive compared to the competition.
S3 is secure, durable, and highly scalable cloud storage. Amazon S3 is standalone object storage that you can use to store and retrieve any amount of data from just about anywhere on the web. Generally, S3 should be used for objects that are used frequently or at least have to be accessed quickly independent of EC2.
Amazon S3 greatly mitigates / eliminates the need to capacity plan, reduces the need for capacity constraints by allowing you to pay for only what you use. Of course, reports are available to show current and historical usage. S3 not only scales extremely well from a volume capacity perspective by automatically providing space on demand practically unlimited, S3 also increases IO rates based on demand by automatically partitioning buckets when demand increases.
It is worth noting again that S3 storage can be used independent of any EC2 instance in fact an EC2 instance is not required.
Amazon S3 objects are also called buckets. Bucket names must be unique across all AWS accounts.
Buckets are NOT hierarchical. Each bucket is separate and one bucket cannot contain other buckets. A bucket can contain unlimited objects. Each object is composed of metadata about the object, and data.
Any kind of data can be stored in S3 in any format. Buckets can contain nested folders and or storage objects.
I typically manipulate my S3 buckets from the AWS console but Amazon provides other methods – e.g. an API and URL. The S3 API works by providing http like verbs – e.g. get and put.
Amazon States:
“Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.”
S3 buckets provide what Amazon refers to as “read-after-write” consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES. Meaning new objects are immediately readable as written but updates and deletes will eventually be consistent – so understanding these rules is important to managing expectations as the rules for consistency are quite different than traditional storage. See AWS S3 documentation for the gory details.
S3 pricing seems reasonable – especially if you are in your free trial year – you can get a fair amount of somewhat limited reads and writes and 15GB free. Even after you have to pay, the price is good and goes down per GB as you use more.
If you choose to allow read public permissions on a bucket and an object in that bucket read public – you can access object via URL – S3 will point out the URL for you. Like:
https://s3-us-west-1.amazonaws.com/myuniquebucket-public/MyResume.htm
Take it one step further if you are using Route 53 for DNS name and routing and you can create a DNS CNAME record that uses your domain name instead of the S3 URL. That is another post here.
AWS S3 – supports several types of server side encryption and client side encryption, and logging of activies against S3 objects.
Amazon Glacier is also considered a form of S3 – without the key name (object id only) for cold rarely used data. Glacier data has a 3 to 5 hour retrieval window. S3 Glacier tier is normal the last tier in object life-cycle.
Amazon Glazier can also be used stand alone.
Glacier based objects are stored in zip or TAR format and an archive can be up to 40TB – immutable / read only. Many archives can be stored in a vault, and vaults can be locked.
Amazon Elastic MapReduce (EMR)
Elastic MapReduce allows businesses and developers, among others, to process vast amounts of data using a hosted Hadoop, and Spark frameworks running on Amazon HDFS, EMRFS, EC2 and or S3, for things like data mining and AI.
Amazon CloudFront
CloudFront provides distributed data services through high-speed transfers.
CloudFront works well with EC2 and S3, and delivers static and streaming content using a global network of edge locations.
Amazon SimpleDB
SimpleDB is a non-relational data store that eliminates much of the tedious database administrative task. The service as creates and manages multiple replicas of your data automatically ensuring high availability and durability. The product is not widely used.
Amazon Relational Database Service (RDS)
RDS provides robust, scalable relational databases services in the cloud. RDS supports several database products including mysql, mariadb, Oracle, PostgreSQL & Aurora. Many database administrative tasks are automated for you if you choose at very reasonable prices – like replication and backups.
Amazon DynamoDB
Serverless database technology that given the right use case not only scales massively well with excellent performance but can also be cost effective.
With DynamoDB, you design your tables and queries against them to make the most common and important queries as fast and as inexpensive as possible. Your data structures are customize to very specific requirements of your business use cases. Generally joins don’t work well at all. The number of use cases for which DynamoDB is feasible is limited. Equality queries against one table using the primary key scale extremely well.
Amazon Redshift (and ancillary products like Spectrum, Glue, DMS and many others)
Redshift is a clustered database warehousing purposed technology – that allows you to pick the number of nodes in the cluster, and the types (# of CPUs, Memory etc.) of nodes in the cluster. Data is spread across slices per node. The number of slices per node is roughly equivalent to the # of CPUs per node. Data distribution is controlled on a per table basis by identifying a distribution key.
Distribution keys are of type EVEN, ALL, KEY.
Data access can be tuned using sort keys.
Amazon CloudWatch
CloudWatch enables monitoring for EC2 and other Amazon cloud services. Configurable by EC2 instances and more.
There is a free level and advanced level for a price. Metrics reporting provided include CPU utilization, disk reads and writes and network traffic.
Amazon Simple Queue Service
Simple Queue Service (SQS) is a hosted queueing service and API guarantees message delivery in the cloud.
Amazon Simple Notification Service
Simple Notification Service (SNS), is a Web service that provides a publish and subscribe model for notifications via an API. This service is used by AWS itself Simple WorkFlow Service (SWF).
Amazon Simple WorkFlow (SWF)
SWF allows developers build, run, and scale background jobs that have parallel or sequential steps. And gives end users the ability to process work loads via corporate define rule sets and process flows in the Cloud.
AWS Elastic Load Balancing
Elastic Load Balancing automates the distribution of incoming application traffic across multiple Amazon EC2 instances, containers, and IP addresses in a single Availability Zone or across multiple Availability Zones.
Elastic Load Balancing offers three types of load balancing: Application Load Balancing, Network Load Balancing, and Classic Load Balancing.
Amazon Elastic MapReduce (Amazon EMR)
EMR / Amazon Elastic MapReduce is a managed web service that runs Hadoop clusters to process vast amounts of data by distributing the work and data among one to many EC2 servers. EMR uses Amazon EC2 and Amazon S3 services and built-in scalability and robustness to complement and enhance Apache Hadoop framework in your AWS cloud.
AWS Elastic Container Services ( ECS )
An Amazon managed service to support Docker Deployment.
AWS Elastic Container Registry ( ECR )
An Amazon managed service / repository for Docker images used with ECS.
To be completed…