If you need a minimal and scalable distributed object server, then your best bet is Minio. You can use it to store all your unstructured data such as photos, videos, container/VM images, log files and archives. Delve into this article to know more.
We usually tend to think of databases in terms of structured data like tables/columns or even attributes. However, modern applications generate lots of data that cannot be easily classified into structures. Instead, it exists in the form of blobs, objects or files. As an example, a blogging server may use a SQL database for posts and authors, but it also has various other media files like videos, images, sound files, etc. Now, imagine a scenario where you are responsible for storing away backup files or log files in a secure manner and making them accessible.
Minio is an Apache licensed open source distributed object server that can help you here, and do a lot more. Written in the Go language, it has a minimal design and the goal is to get you started quickly. It is a great option with which to introduce an Object Store into your application. Minio is designed to be compatible with Amazon S3, which makes it easy to have the same API interface and make it work seamlessly on your local systems and the cloud.
So, let’s get started with setting up Minio on your local machine and writing an application to integrate with it.
Running a Minio server
There are a couple of ways in which you can run Minio. You can either download the binaries for your operating system/architecture from the Downloads page or you can use the Minio Docker image, if you are using Docker. We will go with the former option.
Go ahead and download the Minio Server distribution that applies to you. In my case, I downloaded a Windows build. From the Downloads page, you can get both the server (minio.exe) and its command-line client (mc.exe). Ensure that the binaries are available in the PATH.
While starting the Minio server, the key thing is to provide it a PATH value that points to the directory where Minio can persist the objects. While there are multiple disks and other options that you can provide while starting it, we will go with the basic use case, i.e., use one folder (directory) to store our persistent data. This can be any accessible drive on your machine. In my case, I chose to go with the d:\minio-data folder.
Starting the Minio server is as simple as using the server sub-command for the Minio executable and providing the PATH, as shown below:
D:\minio>minio server d:\minio-data Endpoint: http://169.254.224.241:9000 http://10.21.11.176:9000 http://169.254.68.108:9000 http://169.254.83.68:9000 http://192.168.99.1:9000 http://127.0.0.1:9000 AccessKey: MDMU09WUXO3ZEFDKSA5B SecretKey: 6V/xJLyW1XTL78kq9Jl3kuGoqnZpJ82at1t/wBaR Region: us-east-1 SQS ARNs: <none> Browser Access: http://169.254.224.241:9000 http://10.21.11.176:9000 http://169.254.68.108:9000 http://169.254.83.68:9000 http://192.168.99.1:9000 http://127.0.0.1:9000 Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide $ mc.exe config host add myminio http://169.254.224.241:9000 MDMU09WUXO3ZEFDKSA5B 6V/xJLyW1XTL78kq9Jl3kuGoqnZpJ82at1t/wBaR Object API (Amazon S3 compatible): Go: https://docs.minio.io/docs/golang-client-quickstart-guide Java: https://docs.minio.io/docs/java-client-quickstart-guide Python: https://docs.minio.io/docs/python-client-quickstart-guide JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide Drive Capacity: 64 GiB Free, 304 GiB Total
On successfully starting up, you should note the server access key and the secret key that are provided. You will need these to access the Web user interface that Minio also provides. Additionally, if you are looking to use the Minio API to integrate into your applications, you will need these key values; so, keep them handy and secure.
Accessing the Minio server
In the startup log shown in the console, you will notice multiple URLs. Since we are using the same machine, we can go with localhost for accessing the Minio server. Simply launch a browser and visit the http://localhost:9000 address. This will bring up the page as shown in Figure 1.
Provide the access key and the secret key, after which you are good to go. Minio organises the objects into a collection of buckets. A bucket can be conceptually thought of as a folder that holds a collection of objects.
Click on the + sign at the right bottom to get an option to either create a bucket or upload a file (Figure 2).
Select the Create bucket option and give it a name, e.g., backups. This will create a bucket in the list of buckets shown in the left pane.
You can now use the interface itself to add files to the specific bucket. This structure is required because Minio is also compatible with cloud storage services like Amazon S3; so, it becomes easier for it to synchronise with the same.
Think for a while about how the process of setting up and running Minio is a straightforward exercise. Imagine if you set this up on your local network for your personal use or even for your team/enterprise. You can immediately start accessing the server and working with buckets and files.
You will definitely need to provide the Object Store in your applications. For example, it could be an extension to your existing application, using which people can upload files that Minio will store and manage. Other use cases could be taking backups of existing databases or folders and then storing them in Minio. The possibilities are endless, and I recommend readers take a look at the Minio community projects page to see how Minio has been innovatively integrated as part of larger projects.
Minio client SDKs
If you wish to interface with Minio via one or more programming languages, there are SDKs for Java, Python, JavaScript, Go and others. The SDKs are well designed wrappers around the Minio REST APIs. The pattern of using any Minio client SDK is straightforward—you establish a client connection to the Minio server and then perform one or more commands that are applicable. The commands could include listing/creating buckets, listing/uploading file objects, and so on.
A simple backup utility
Let us think of how we can potentially use Minio as a store to retain multiple backups. These could be particular directories that you want to back up, certain files (log files) or even database dumps that you need to save for restore or audit purposes.
To use the API, as mentioned earlier, you will need to have the hostname and port of your Minio server along with the access and secret key. Once you have all this, we can establish a connection to the Minio client and then perform the operation. In this case, we will simply upload a standard file object into a bucket mybackup that we have pre-created.
We shall use the Minio Python SDK for this purpose. I have modified the code from the official documentation, and the code assumes that you have a bucket named mybackup in the Minio server:
from minio import Minio
from minio.error import ResponseError
minioClient = Minio('localhost:9000',
access_key='YOUR_MINIO_ACCESS_KEY',
secret_key='YOUR_MINIO_SECRET_KEY',
secure=False)
print(minioClient)
try:
minioClient.fput_object('mybackup', 'mydb.log', 'mydb.log')
except ResponseError as error:
print(error)
The code first creates the Minio client object. Remember to use your access details in the parameters. It then assumes that you have a bucketname created in the Minio server named mybackup and a file mydb.log in the current directory that you want to upload. The second and third parameters in the fput_object method stand for object name (when stored in Minio server) and the file path (where the file can be found by the Python program).
To run the Python app, first ensure that you have downloaded the Minio Python library as given below:
pip install minio
Minio client
The Minio client (mc) is a command line utility that provides an alternative to UNIX commands like ls and cp, with the interesting twist that it works for both your local system as well as the Amazon S3 service in a seamless manner. You can check out https://docs.minio.io/docs/minio-client-complete-guide for more details.
Contributing to Minio
Minio prides itself on being open source, and a quick look at the GitHub page shows an active project with multiple contributions from developers. The Minio server and client are written using the Go programming language and, hence, contributors need to be familiar with this language. You can also look at the client SDKs available in multiple languages, if you wish to contribute in that way.
You can also contribute to the growing list of Minio community projects that are listed in the community page. A quick glance at how multiple projects have used Minio is sure to inspire you with some ideas.
The creators of Minio have focused on minimalism and providing well designed APIs that make it easy for you to customise it to your needs. This, coupled with the fact that Minio is available on multiple target platforms, allows you to choose your hardware and scale, as needed.
