This article presents a solution for using multiple Azure services to optimize storage performance and cost. Azure Cosmos DB holds recent data and meets availability, performance, and resiliency requirements. Azure Data Factory transports data, and Azure Data Lake Storage provides cost-effective storage.
Architecture
Download a Visio file of this architecture.
Dataflow
- The client authenticates with Azure Active Directory (Azure AD) and is granted access to web applications hosted on Azure App Service.
- Azure Front Door, a firewall and layer-7 load balancer, switches user traffic to a different Azure region if there's a regional outage.
- App Service hosts websites and RESTful web APIs. Browser clients run AJAX applications that use the APIs.
- Web APIs delegate functions to handle background tasks. The tasks are queued in Azure Queue Storage queues.
- The functions that are hosted by Azure Functions perform the background tasks, triggered by the queued messages.
- Azure Cache for Redis caches database data for the function apps. By using a cache, the solution offloads database activity and speeds up the function apps and web apps.
- Azure Cosmos DB holds three to four months of the most recent data used by the web applications.
- Data Lake Storage holds historical data used by the web applications.
- Periodically, Data Factory moves data from Azure Cosmos DB to Data Lake Storage to reduce storage costs.
Components
- Azure AD is a multi-tenant identity and access management service that can synchronize with an on-premises directory.
- Azure DNS is a high-availability hosting service for DNS domains that provides apps with fast DNS queries and quick updates to DNS records. Managing Azure DNS is like managing other Azure services, and uses the same credentials, APIs, tools, and billing.
- Azure Front Door is a secure content delivery network (CDN) and load balancer with instant failover. It operates at the edge close to users, accelerating content delivery while protecting apps, APIs, and websites from cyber threats.
- App Service is a fully managed service for building, deploying, and scaling web apps. App Service provides a framework for building apps by using .NET, .NET Core, Node.js, Java, Python, or PHP. Apps can run in containers or on Windows or Linux. In a mainframe migration, the front-end screens or web interface can be coded as HTTP-based REST APIs. They can be segregated and can be stateless to orchestrate a microservices-based system. For more information about web APIs, see RESTful web API design.
- Functions provides an environment for running small pieces of code, called functions, without having to establish an application infrastructure. You can use it to process bulk data, integrate systems, work with Internet of Things (IoT) devices, and build simple APIs and microservices. With microservices, you can create servers that connect to Azure services and are always up to date.
- Azure Storage is a set of massively scalable and secure cloud services for data, apps, and workloads. It includes Azure Files, Azure Table Storage, Queue Storage, and Data Lake Storage. Azure Files is often an effective tool for migrating mainframe workloads.
- Queue Storage provides simple, cost-effective, durable message queueing for large workloads.
- Azure Cache for Redis is a fully managed in-memory caching service and message broker for sharing data and state among compute resources. It includes both the open-source Redis and a commercial product from Redis Labs as managed services. You can improve the performance of high-throughput online transaction processing applications by designing them to scale and to make use of an in-memory data store such as Azure Cache for Redis.
- Azure Cosmos DB is a globally distributed, multi-model database that enables your solutions to elastically and independently scale throughput and storage across any number of geographic regions. It offers throughput, latency, availability, and consistency guarantees with comprehensive service level agreements (SLAs).
- Data Factory is a managed service that orchestrates and automates data movement and data transformation.
- Data Lake Storage offers secure and massively scalable data lake functionality that's built on Azure Blob Storage.
Alternatives
- Azure Traffic Manager directs incoming DNS requests across the global Azure regions based on your choice of traffic routing methods. It also provides automatic failover and performance routing.
- Azure Content Delivery Network caches static content in edge servers for quick response, and uses network optimizations to improve response for dynamic content. Content Delivery Network is especially useful when the user base is global.
- Azure Kubernetes Service (AKS) is a fully managed Kubernetes service for deploying and managing containerized applications. You can use it to implement a microservices architecture whose components scale independently on demand.
- Azure Container Instances provides a quick and simple way to run tasks without having to manage infrastructure. It's useful during development or for running unscheduled tasks.
- Azure Service Fabric is a platform for scaling and orchestrating containers and microservices.
- Azure Service Bus is a reliable cloud messaging service for simple hybrid integration. It can be used instead of Queue Storage in this architecture. For more information, see Storage queues and Service Bus queues - compared and contrasted.
- Azure Synapse Analytics is a cloud-based data warehouse that uses massively parallel processing to quickly run complex queries across petabytes of data. Azure Synapse Analytics provides serverless or dedicated querying options at scale. Its Spark pools support elastic pool storage.
- Azure Synapse Link for Azure Cosmos DB is a cloud-native hybrid transactional and analytical processing (HTAP) capability that enables you to run near real-time analytics over operational data in Azure Cosmos DB. Azure Synapse Link creates a tight, seamless integration between Azure Cosmos DB and Azure Synapse Analytics.
Scenario details
By using multiple Azure services, this solution optimizes storage performance and cost:
- Azure Cosmos DB holds recent data. This database service also meets web app availability, performance, and resiliency requirements. Native replication capabilities in Azure Cosmos DB replicate its data to other regions.
- Data Lake Storage provides cost-effective storage for older data.
- Data Factory periodically moves older data from Azure Cosmos DB to Data Lake Storage.
Potential use cases
This solution is appropriate for any application that uses a massive amount of data that's always available when it's needed. Examples include apps that:
- Track customer spending habits and shopping behavior.
- Forecast weather.
- Offer smart traffic systems or implement smart traffic systems or use smart technology to monitor traffic.
- Analyze manufacturing IoT data.
- Display smart meter data or use smart technology to monitor meter data.
Considerations
These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that can be used to improve the quality of a workload. For more information, see Microsoft Azure Well-Architected Framework.
- You need to migrate historical data to Data Lake Storage as a one-time activity to ensure the cost effectiveness of the solution.
- Your application developers must implement data migration routines that use Data Factory to move data from Azure Cosmos DB to Data Lake Storage.
- If you're migrating data from an old storage system, you might need to write routines to copy a portion of the old data to Azure Cosmos DB. Make sure that you use time stamps and copy flags to track the progress of the migration.
- You can further optimize the overall architecture by replacing Azure Cache for Redis with an Azure Cosmos DB integrated cache.
Contributors
This article is maintained by Microsoft. It was originally written by the following contributors.
Principal author:
- Nabil Siddiqui | Cloud Solution Architect - Digital and Application Innovation
Next steps
- Web-Queue-Worker architecture style
- Design a geographically distributed application
- Work with Azure Cosmos DB
- Develop solutions that use Azure Cosmos DB
- How to model and partition data on Azure Cosmos DB using a real-world example
- Options to migrate your on-premises or cloud data to Azure Cosmos DB
- Migrate hundreds of terabytes of data into Azure Cosmos DB
- Introduction to Azure Data Factory
- Orchestrate data movement and transformation in Azure Data Factory or Azure Synapse Pipeline