Mountpoint for Amazon S3 is an open-source file client designed to help your file-aware Linux applications connect directly to Amazon S3 buckets. Initially released in alpha earlier this year, it is now generally available and ready for production use. It’s ideal for large-scale, read-heavy applications such as data lakes, machine learning training, image rendering, autonomous vehicle simulation, and ETL. Mountpoint supports file-based workloads that perform sequential and random reads and writes.
Why Files?
Many AWS customers use S3 APIs and SDKs to build apps that work with S3 buckets. But some have existing tools and workflows that follow UNIX-style file access, like reading directories and files. These customers asked AWS for a reliable and fast way to access S3 at a large scale. After talking to them, AWS found that their main concerns were speed and stability, and they didn’t need full UNIX compatibility.
All About Mountpoint…..!
Mountpoint is very simple. You create a mount point and mount an Amazon S3 bucket at the mount point, and then access the bucket using shell commands (ls, cat, dd, find, and so forth), library functions (open, close, read, write, creat, opendir, and so forth) or equivalent commands and functions as supported in the tools and languages that you already use
The Linux Virtual Filesystem (VFS) translates these operations into calls to Mountpoint, which then translates them into S3 operations like LIST, GET, and PUT. Mountpoint optimizes network bandwidth usage, increasing throughput and reducing compute costs by enabling you to accomplish more work in less time.
You can use Mountpoint from an Amazon EC2 instance, or within Amazon ECS or Amazon EKS containers. It can also be installed on your existing on-premises systems, with access to S3 either directly or over an AWS Direct Connect connection via AWS Private Link for Amazon S3.
Installing and Using Mountpoint for Amazon S3
Mountpoint is available in RPM format and can easily be installed on an EC2 instance running Amazon Linux. I simply fetch the RPM and install it using yum:
$ wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
$ sudo yum install ./mount-s3.rpmFor the last few days I have been regularly fetching images and files from several of the applications and storing them in my New-bucket bucket.
I collect the txt file in order to track the comings and goings of the bucket, with a goal of analyzing them at some point to find the best times to ride. My goal today is to backup the Application data Securely. I start by creating a mount point and mounting the bucket:
$ mkdir eks-s3-addon-bucket
$ mount-s3 eks-s3-addon-bucketI can traverse the mount point and inspect the bucket:
As you can see, I used Mountpoint to access the existing image files and to write the newly created files back to S3. While this is a fairly simple demo, it does show how you can use your existing tools and skills to process objects in an S3 bucket.
Mountpoint for Amazon S3 Facts
Here are a couple of things to keep in mind when using Mountpoint:
- Pricing – There are no new charges for the use of Mountpoint; you pay only for the underlying S3 operations. You can also use Mountpoint to access requester-pays buckets.
- Performance – Mountpoint is able to take advantage of the elastic throughput offered by S3, including data transfer at up to 100 Gb/second between each EC2 instance and S3.
- Credentials – Mountpoint accesses your S3 buckets using the AWS credentials that are in effect when you mount the bucket. See the configuration document ( Configuring Mountpoint for Amazon S3) for more information on credentials, bucket configuration, use of requester pays, some tips for the use of S3 Object Lambda, and more.
- Operations & Semantics – Mountpoint supports basic file operations and can read files up to 5 TB in size. It can list and read existing files, and it can create new ones. It cannot modify existing files or delete directories, and it does not support symbolic links or file locking. To enable deletion of existing files, pass the
--allow-deleteflag to themount-s3command. - Storage Classes – You can use Mountpoint to access S3 objects in all storage classes except S3 Glacier Flexible Retrieval, S3 Glacier Deep Archive, S3 Intelligent-Tiering Archive Access Tier, and S3 Intelligent-Tiering Deep Archive Access Tier.
- Open Source – Mountpoint is open source and has a public roadmap.
Hop On
As you can see, Mountpoint is really cool, and I am guessing that you are going to find some awesome ways to put it to use in your applications. Check it out and let me know what you think!
Referral Link: S3 Mount-point
