Microservices in the cloud is a good modern solution.
At the same time, migration applications to a microservice architecture complicated their infrastructure. This cause the development of many methods and approaches to its deployment and orchestration. To make life easier for developers, cloud providers offer infrastructure management automation as a service, allowing developers to focus on coding.

So how to plan your infrastructure with a Cloud Agnostic approach.
Each cloud provider tends to retain its customers not only with its competitive advantages and favorable offers.

Over time, the part of SaaS-integrated code in the functionality of the microservice application increase, and it works only for a certain provider, there can be more and more such modules. To optimize cloud services, sometimes you need to move some of them to another environment. It is not easy to do with vendor-locked solutions when the software installed on its servers is non-unified. Therefore, for multi-functional resource-intensive applications, it is better to build your infrastructure as Cloud Agnostic. It will be cost-efficient and easier to scale it in the future.

The main idea of Cloud Agnostic is the initial focus on independence from a specific cloud platform and vendor-dependent solutions, it makes it possible to flexibly manage the scaling and distribution of the infrastructure independently of the specificity of a certain cloud. To implement it in practice we choose the tools that suit any cloud and can be installed even on-premise.

Serverbee's Cloud Agnostic Experience - Striking the Balance!

Why did we choose Kubernetes as the orchestrator for Cloud Agnostic infrastructure?

Orchestration is a key mechanism for managing the scaling of a microservice architecture. The scalability of the infrastructure and the efficiency of resource management depends on it.  It also influences how quickly it is possible to expand the necessary functionality with regular means, whether there is compatibility for creating multi-cloud services, how large the support community is, whether there are many ready-made solutions, etc.

Kubernetes is the de facto standard orchestrator in any cloud and is well-documented and studied. A DevOps engineer who worked with K8S can quickly understand its settings and configurations no matter which provider it is installed on. Kubernetes gives the highest degree of compatibility with well-known cloud SaaS tools. In the case of transferring the microservice to another provider, you often can replace dependent services with similar ones installed (or included) in the K8S. All sets of helm charts and operators will work the same everywhere, and in case of migration, only a few changes should be made to configurations and manifests, according to the specifics of a particular cloud.

For example, to use secrets from different cloud providers in one Kubernetes cluster, we use the External secrets operator. It makes it possible to store sensitive data in secure cloud storage and to have painless access to it from other Kubernetes applications.

File storage management

Kubernetes can connect various types of file storage to the cluster, work with stateful objects ( SQL database), or transfer configuration information such as secrets. K8S uses the Persistent Volume (PV) subsystem to work with file storage. PV can connect three types of file storage to the cluster:

• Storages with file systems;
• Object storages;
• Block devices for structured data (databases, file systems of virtual machines, etc.).

According to Kubernetes docs:
"... The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, we introduce two new API resources: PersistentVolume and PersistentVolumeClaim..”

The PersistentVolume allows to get access to a file system or device (eg NFS, iSCSI, RBD, CephFS, etc.), The PersistentVolumeClaim it’s a storage request to PersistentVolumes but with different types of access parameters, such as ReadWriteOnce (read and write to one pod), ReadOnlyMany (read-only for many pods), ReadWriteMany (read and write for many pods). At the same time, one PV can be used by only one PVS, and some types of access are not supported by some storages (RBD or iSCSI do not support ReadWriteMany).

For Cloud Agnostic, the PV subsystem is essential, as it is implemented in every cloud with its tools but allows you to connect storage from another cloud or dedicated servers. Also of note, Kubernetes' Container Storage Interface (CSI) takes file storage management mechanisms outside of its core and makes a unified API available to other orchestration systems and storage vendors.

Orchestration of a distributed data storage system in clusters

However, file storage requires not only a high degree of compatibility and availability but also fault tolerance and scalability. It is not a big deal to take data from AWS S3 and place it somewhere with the help of MinIO (native to Kubernetes, S3-compatible object storage). But it is much more difficult to ensure reliable data storage and replication, as well as load distribution. For this purpose, we at ServerBee use Ceph.

Ceph is a distributed storage orchestrator that can be used as an external cluster to connect to Kubernetes. It has no single point of failure. Ceph programmatically balances the file storage load, replicates data, and easily scales it to petabyte sizes. Since Ceph is a software-defined system, it is possible to combine a stack of servers with different types of disks into its cluster, no matter their size, but with a reliable, fast network connection. In case of failure of some node, a group of nodes, or disks, Ceph will restore access to the lost data without the downtime and transparently for clients. After replacing the disks or nodes with good ones, it will replicate the data back.

I should mention here that some companies also use Rook.
Rook was designed to automate functions, self-manage, self-scaling, and self-healing of file storage. Rook automates deployment, bootstrapping, configuration, provisioning, scaling, upgrades, migrations, disaster recovery, monitoring, and resource management. But there is a nuance - all this Rook provides for data stores that are used in the Kubernetes environment. We know that many prefer not to do this, so we mention the capabilities of this operator just for information. "Advantages and disadvantages of storing data, (or databases), inside or outside K8S" is still a debatable topic that requires a separate article.

A brief review of databases in the context of Cloud Agnostic

Those who often use various cloud services have probably noticed that each cloud provider has a standard set of databases. Some of them are very specific and designed to work in the conditions of a certain cloud (for example, Azure SQL Database, Google Cloud Spanner, Amazon Quantum Ledger Database, etc...) Such databases have their advantages within their cloud. These are optimization for additional features or acceleration of certain functions in serverless applications. But if you want to develop a project as Cloud Agnostic, you better avoid similar SaaS tools, to maintain compatibility and uniformity of infrastructure. In our experience, Cloud Agnostic projects most often use SQL databases — PostgreSQL, MySQL, and MongoDB (a classic example of a NoSQL database) or other open-source databases.

Conclusion:

We can outline some important principles from our experience for the implementation of Cloud Agnostic infrastructure:

• For the orchestration of microservices, it is recommended to use Kubernetes because this solution is independent of any cloud platform. It is worth avoiding vendor-locked services and exotic software. Kubernetes is supported by a solid community, and its functionality is easily extended with available operators and applications.
• Access to file repositories in the cluster should be carried out through the PV(PVC) or CSI interface. These abstractions allow different kinds of access to different kinds of file storage with help of simple API.
• File storage scaling should be cost-efficient and have a reliable mechanism for data storage and replication, as well as load balancing! A software-oriented (en. software-defined) system of Ceph clusters suits this purpose very well in the Cloud Agnostic paradigm.
• It is better to plan the infrastructure as Cloud Agnostic in advance. You should take into account the type of databases used in your application, and avoid the use of specific vendor-dependent ones that are not supported by other cloud providers.

The list can be enlarged but these are the main points which we wanted to highlight.