Canonical and Intel Optimize Data Compression in Data Centers with Intel® Quick Assist Technology (QAT) Hardware Acceleration, Reducing CPU Consumption and Enhancing Storage Efficiency in Ceph.
The increasing demand for cloud and data center storage has driven the search for solutions that optimize hardware usage without compromising performance. In this context, Canonical and Intel have developed an integration that allows Ceph to leverage Intel® Quick Assist Technology (QAT) to reduce processor workload in compression tasks, improving efficiency without sacrificing speed.
The Challenge of Large-Scale Storage
The cost of storing large volumes of data is largely defined by the strategy used to protect the information. Traditionally, there are two approaches:
- Multiple replicas, which increases redundancy but also space consumption.
- Erasure Coding, which optimizes storage use but requires higher processing power.
Data compression is another key strategy to reduce disk space consumption, but its implementation without specialized hardware often increases CPU consumption and degrades system performance. The solution to this problem lies in hardware acceleration, allowing these tasks to be carried out without affecting overall processing capability.
Intel® QAT: Reducing CPU Load and Increasing Efficiency
Intel® Quick Assist Technology (QAT) is a hardware acceleration solution that offloads certain compute-intensive tasks from the CPU, including:
- Data encryption and decryption.
- Data compression and decompression.
With the arrival of the 4th (Sapphire Rapids) and 5th Generation (Emerald Rapids) Intel® Xeon® processors, QAT engines no longer require additional PCIe cards but are instead integrated within the processor, facilitating their adoption in enterprise and data center environments.
The impact of this integration on storage with Ceph is significant, as it enables high levels of performance while reducing the amount of storage used, improving efficiency and reducing costs.
Performance Comparison: Compression with and without QAT
To assess the impact of accelerated compression, tests were conducted with a 4-node Ceph cluster and NVMe drives, using MinIO Warp to simulate object storage workloads.
Key Results:
| Metric | Without Compression | With Compression (Without QAT) | With Compression + QAT |
|---|---|---|---|
| BandwidthBandwidth is the maximum transfer capacity for writing | 4.66 GBps | 50% Reduction | 5.05 GBps (+8%) |
| Read Bandwidth | 21.86 GBps | 11.28 GBps (-48%) | 20.88 GBps (-4%) |
| CPU Consumption | Normal | +150% | No Impact |
Space Savings: Compression Assessment
Data compression is especially useful in previously uncompressed formats, allowing for significant storage savings.
| Data Type | Compression Ratio | Space Saving |
|---|---|---|
| Structured Data (CSV, JSON, Text) | 1.33 | 25% |
| Compressed Images (JPEG) | 1.01 | 1% |
| Uncompressed Video (RAW YUV) | 3.13 | 68% |
| Compressed Video (H.264) | 1.00 | 0% |
The results indicate that compression is more effective on structured data and uncompressed multimedia files, while formats like JPEG and H.264 do not benefit from a second compression.
Implementation in Enterprise Environments
For companies interested in trying out this solution, Canonical recommends implementing MicroCeph, an optimized version of Ceph that simplifies setup in testing and production environments.
Steps to Enable Intel® QAT in Ceph
- Install MicroCeph with QAT support:
snap install microceph --channel=squid/edge/qat --devmode - Install the QAT engine on each storage node:
sudo apt-get -y install qatengine - Configure Ceph to enable QAT:
ceph config set client.radosgw.gateway qat_compressor_enabled true - Enable compression on object storage:
radosgw-admin zone placement modify --rgw-zone default --placement-id default-placement --storage-class STANDARD --compression zlib - Verify data compression with:
radosgw-admin bucket stats
Impact on the Data Center Market
The implementation of QAT in Ceph represents a key advancement in storage efficiency in data centers and enterprise cloud solutions. Among the most relevant benefits are:
- Reduction in operational costs by requiring less hardware and less physical space.
- Optimization of performance with greater efficiency in CPU usage.
- Higher storage density, allowing for more workloads on each node.
- Improvement in sustainability by reducing energy consumption compared to traditional systems.
With the growth of data volumes in sectors such as banking, healthcare, telecommunications, and e-commerce, these innovations are essential for maintaining scalability and operational efficiency.
Conclusion: A New Standard in Scalable Storage
The combination of Ceph, Intel® QAT, and Ubuntu opens new possibilities for efficient, cost-effective, and high-performance storage. Companies and cloud service providers can optimize infrastructure without sacrificing speed or processing capacity, making this solution a strategic choice for next-generation storage environments.
via: Ubuntu