Cross-Storage Device Vertical Tiering (Erl, Naserpour)
How can the vertical scaling of data processing be carried out dynamically?
ProblemIncreasing the processing capacity of data stored on cloud storage devices generally requires the manual vertical scaling of the device, which is inefficient and potentially wasteful.
SolutionA system is established whereby the vertical scaling of data processing can be carried out dynamically across multiple cloud storage devices.
ApplicationUsing pre-defined capacity thresholds, LUN migration is used to dynamically move LUN disks between cloud storage devices with different capacities.
MechanismsAudit Monitor, Automated Scaling Listener, Cloud Storage Device, Cloud Usage Monitor, Pay-Per-Use Monitor
Compound PatternsBurst In, Burst Out to Private Cloud, Burst Out to Public Cloud, Elastic Environment, Infrastructure-as-a-Service (IaaS), Multitenant Environment, Platform-as-a-Service (PaaS), Private Cloud, Public Cloud, Resilient Environment, Software-as-a-Service (SaaS)
After working with a provisioned cloud storage device, a cloud consumer may determine that the device is unable to accommodate its necessary performance requirements. Conventional approaches to vertically scaling the cloud storage device include adding more bandwidth to increase IOPS (input/output per second) and adding more data processing power. This type of vertical scaling can be inefficient and time consuming to implement, and can result in waste when the increased capacity is no longer needed.
This approach is depicted in the following scenario in which a number of requests for access to a red LUN increased, requiring it to be manually moved to a high performance storage device.
Figure 1 - Conventional vertical scaling of a cloud storage device.
- The cloud provider installs and configures a storage device.
- The cloud provider creates the required LUNs for the cloud consumers.
- Storage LUNs are presented to their respective cloud consumers, who begin using them.
- The storage devices start forwarding requests to cloud consumer LUNs.
- The number of requests increases significantly, resulting in high storage bandwidth and performance demands.
- Some of the requests are rejected or time out due to performance capacity limitations.
A system is established that can survive bandwidth and data processing power constraints (thereby preventing time-outs), and that introduces vertical scaling between different storage devices possessing different capacities. As a result, LUNs can automatically scale up and down across multiple devices, allowing requests to use the appropriate level of storage devices to perform the tasks required by the cloud consumer.
Although the automated tiering technology can move data to cloud storage devices with the same storage processing capacity, the new cloud storage devices with increased capacity need to be made available. For example, solid-state drives (SSDs) may be suitable devices for upgrading data processing power.
The automated scaling listener monitors requests sent to specific LUNs. When it identifies that a predefined threshold is reached, it signals the storage management program to move the LUN to a higher capacity device. Interruption is prevented because a disconnection during the transfer never occurs. While the LUN data is scaling up to another device, its original device remains up and running. As soon as the scaling is completed, cloud consumer requests are automatically redirected to the new cloud storage device.
A sample cloud architecture that illustrates the application of the Cross-Storage Device Vertical Tiering pattern is shown in three parts in Figures 2, 3, and 4.
Figure 2 - A cloud architecture resulting from the application of the Cross-Storage Device Vertical Tiering pattern (Part 1).
- The primary storage (also known as the "lower capacity storage") is installed and configured, and responding to cloud consumer storage requests.
- A secondary storage device with higher capacity and performance is installed.
- The LUN migration is configured via a storage management program.
- Thresholds are defined in the automated scaling listener, which monitors the requests.
- The storage management program is installed and configured to categorize the storage based on device performance.
- Cloud consumer requests are sent to the (lower capacity) primary storage device.
Figure 3 - A cloud architecture resulting from the application of the Cross-Storage Device Vertical Tiering pattern (Part 2).
- The number of cloud consumer requests reaches the predefined request threshold.
- The automated scaling listener signals the storage management program that scaling is required.
- The storage management program calls the LUN migration program to move the cloud consumer's red LUN to a higher capacity storage device.
- The LUN migration program initiates the move of the red LUN to a higher capacity storage device.
- Even though the LUN was moved to a new storage device, cloud consumer requests are still being sent to the original storage device.
Figure 4 - A cloud architecture resulting from the application of the Cross-Storage Device Vertical Tiering pattern (Part 3).
- The storage service gateway forwards the cloud consumer storage requests from the red LUN to the new storage device.
- The red LUN is deleted from the lower capacity device (via the storage management and LUN migration programs).
- The automated scaling listener monitors the cloud consumer requests for access to the higher capacity storage for the red LUN.
- Usage and billing data is tracked and stored via the pay-per-use monitor mechanism.
NIST Reference Architecture Mapping
This pattern relates to the highlighted parts of the NIST reference architecture, as follows: