Transition from hardware dependency to service dependency
Apache Kafka was born in the era of IDC, designed for physical hardware. In the cloud computing era, software defines hardware and provides high availability and reliability with service level agreements. Building Kafka on cloud services eliminates the need to implement complex distributed multi-replica replication protocols, resulting in a simpler architecture and lower costs.
Transform the reservation model to the pay-as-you-go model
Physical server provisioning typically takes months, leading enterprises to budget annually and design software based on reserved resources from a decade ago. However, in today's cloud environment, all resources can be provisioned via API calls. AutoMQ leverages optimal cloud resource selection for cost-effectiveness.
Storage Cost
257.4 USD/Month
2095.7 USD/Month
Computing Cost
201.5 USD/Month
3054.3 USD/Month
AutoMQ
Apache Kafka
* The data is tested based on workload ranging from 80MB/s to 1200MB/s.
Shared storage, no data replication when scaling up
In shared storage architecture, partition data is fully stored in S3, enabling the cluster to scale out without the need for data replication to handle sudden traffic spikes. In contrast, Apache Kafka requires significant bandwidth for data replication after scaling out, limiting its ability to quickly respond to sudden traffic spikes.
Transition from human governance to autonomous Kafka
In the past, software architecture relied on manual stability protection measures to ensure business continuity, such as capacity assessment, stability plans, and traffic throttling. Today, leveraging cloud-native technologies enables automatic scaling, automatic load balancing, and automatic fault recovery, making systems more autonomous and efficient.
Storage-compute separation is crucial for achieving elasticity. Offloading storage to cloud services significantly reduces operational complexity and enables complete separation of state, leading to a more streamlined architecture. Introducing a distributed storage system would greatly increase operational complexity.
Shared-everything arch is superior to shared-nothing arch
In the past, limitations existed between different hardware types, making Shared Nothing architecture a compromise. However, storage has become highly flexible today, with capacities approaching "infinite". By leveraging shared storage like S3 to achieve storage-compute separation architecture, you can effortlessly perform operations such as partition migration, scaling in/out, in an unprecedentedly simple manner.
Compatible with the entire compute layer, not just limited to API-level compatibility
The definition of a software product is primarily reflected in the compute layer. Over the past decade, Kafka has released multiple major versions, with the producer message API undergoing dozens of version iterations. To achieve full compatibility, it is necessary to reuse Kafka code as much as possible. Leveraging the native Kafka compute layer and cloud-native streaming storage, AutoMQ achieves 100% compatibility with Kafka functionality by identifying minimal storage replacement entry points.
Ensure compatibility with the latest version of Apache Kafka with at most a one-month lag
By re-implementing the core class kafka.log.LogSegment in Apache Kafka, we have successfully migrated Kafka storage to object storage with minimal changes to other Kafka code. This allows AutoMQ to complete the adaptation within one month of the release of a new version of Apache Kafka, ensuring continuous synchronization with the Kafka community.
To reduce operating costs, traditional Kafka services utilize a shared physical cluster for multiple tenants, which necessitates capping each tenant's resource usage to prevent interference with others. AutoMQ, on the other hand, delivers high availability with a lean setup of just three machines, each with 2 CPUs and 8GB RAM, coupled with the capability for on-demand scalability. Hence, each tenant can exclusively command the underlying cluster and scale without constraints as needed.
Minimize blast radius
In traditional multi-tenant scenarios, where multiple tenants share a large Kafka cluster, there is a risk that excessive usage by a few tenants can lead to significant outages, impacting others. AutoMQ leverages a cloud-native architecture to successfully address this issue. By ensuring each application operates on an independent Kafka cluster, AutoMQ achieves optimal cost efficiency while simultaneously reducing the blast radius of system failures.
.png)
.png)
.png)
.png)
.png)
.png)
.png)
%201.png)
.png)
.png)
.png)
