Digital applications rule our lives today. It’s important to be familiar with the basic principles for building them.
Modern applications are based on digital technologies leveraging the cloud, data and AI. They are available 24/7, providing a rich, seamless and omni-channel user experience. Unlike legacy applications, which are custom built and are usually monolithic in nature, often accumulating a lot of technical debt over time, modern applications are reimagined to be cloud and AI native. They are scalable, resilient, and provide interfaces to connect with external systems. A good data framework is foundational to meet these requirements.
Cloud-native applications fall into various categories:
- A microservices-based web application is a set of small services, with each service doing a particular activity that can be scaled independently. Each of these services can be containerised and managed/deployed separately.
- An event-driven application can typically be triggered by an event and run as functions, pipelines or processes. The patterns are scheduled to meet the event triggering criteria.
- A data application ingests data in real-time/batch and stores it in a data lake. The data is processed further to make it ready for analytical applications, which provide features like real-time dashboards.
- A chat application is a modern use case that can leverage the power of generative AI/agentic AI tools and orchestrated workflows to answer user queries in natural language.
- A mobile application typically runs on user handheld devices that typically have lower system configurations and provide anywhere access.
- Being cloud-native also enables:
- Continuous integration and continuous delivery, leading to faster time to market and fostering a culture of innovation and growth.
- Infrastructure automation and DevOps, which are foundational for ensuring fit-for-use and fit-for-purpose on-demand infrastructure as well as collaboration between development and operational teams.
- Ready-to-use identity and security components, which help to build applications that can handle users across the globe reliably with integrity.
Key principles for creating the data architecture
To meet the demands of modern cloud-native applications, organisations must have a strong data strategy.
The key principles are:
- Build a ‘strong foundation’. Have a clear purpose and estimate the data needs of the applications.
- Adopt the ‘data is gold’ principle. Data from different sources and systems can be used for various purposes. Applications produce structured, semi-structured and unstructured data, and there are tools and technologies to process the various types of data.
- Separate ‘chalk from cheese’. The available data could also contain a lot of junk data that needs to be weeded out. These could be repeatable elements, system generated messages or duplicated data elements.
- Find the ‘Holy Grail’. Process the useful data, and refine and enrich it to make the data readily usable and inferential.
- Make it ‘sustainable’. Create strong data governance standards and build metadata to easily manage the data lifecycle throughout the organisation.
Data architecture for cloud-native applications
Key architectural considerations for cloud-native applications are:
- Create a data-service combo, where a service is fully handled by a database. This ensures that the service can be managed and scaled independently.
- Separate reads and writes by strategically replicating the data so that applications that only need to access the data can independently handle their needs without any cross-dependencies on data updates.
- De-couple and build asynchronous data components so that producers of data are separated from the consumers, and each can process according to their needs.
- Separate operational data stores from analytical stores, and build data pipelines that transform and load data as per the needs of the organisation.
- Adopt data privacy and security principles by properly masking the data and providing access to users as per their needs.
- Always back up your data and restore it based on your recovery point and recovery time needs.
- Avoid data-centre concentration risks by leveraging the power of the cloud and storing data in multiple areas/zones.
- Adopt modern architectural styles like data-mesh, where data is decentralised and made ready for use by the domain teams.
In summary, creating a data architecture is both a top-down and bottom-up exercise. Business owners, data architects, data science/data analytics teams, development teams and governance/security teams need to work together in a cohesive manner to understand the data needs, and then build a data platform and architecture that meets the needs of the organisation in a sustainable manner.
