Hardware requirements

Olga Semme

It is preferable to use physical servers, but it is necessary to analyze the power consumption of the equipment you are considering.

Arenadata Streaming works equally effectively both on bare metal and in the cloud.

When providing network availability between servers, the following scenarios are also possible:

  • heterogeneous IT infrastructure;

  • Multi-clouds;

  • distributed installation on different infrastructures.

The sequence of preparation of physical and virtual hosts is given in the article Preparing hosts.

Examples of requirements for different services depending on the load pattern are listed below.

ZooKeeper, Kafka Manager, Schema Registry

  • Minimal

  • Proof of concept

  • Commodity

  • Balanced

Storage CPU RAM Network

400 GB SSD for OS

500 GB SSD for Data

8 cores

32 GB

1 x 10 GbE
Number of nodes Storage CPU RAM Network

3 or more

1 x 500 GB HDD for OS

1 x 2.4 TB SSD for Transaction Logs

1 x 1 TB HDD for Data

8 cores

32 GB

1 x 10 GbE
Number of nodes Storage CPU RAM Network

3 or more

2 x 500 GB HDD RAID1 for OS

2 x 2.4 TB SSD RAID1 for Transaction Logs

2 x 1 TB HDD RAID1 for Data

8 cores

32 GB

2 x 10 GbE
Number of nodes Storage CPU RAM Network

3 or more

2 x 500 GB HDD RAID1 for OS

2 x 2.4 TB SSD RAID1 for Transaction Logs

2 x 1 TB HDD RAID1 for Data

8 cores

32 GB

2 x 10 GbE
IMPORTANT
For guaranteed HA (High Availability), it is necessary to have an odd number of ZooKeeper Server instances.

Kafka Broker

  • Proof of concept

  • Commodity

  • Balanced

  • Compute-intensive

Number of nodes Storage CPU RAM Network

3 or more

1 x 500 GB HDD for OS

2 x 500 GB SSD RAID1 for Data

8 cores

32 GB

2 x 10 GbE
Number of nodes Storage CPU RAM Network

3 or more

2 x 500 GB HDD RAID1 for OS

4 x 500 GB SAS12G SSD RAID10 (PCIe Gen3 x4+) for Data

8 cores

32 GB

2 x 25 GbE
Number of nodes Storage CPU RAM Network

3 or more

2 x 500 GB HDD RAID1 for OS

8 x 500 GB SAS12G SSD RAID10 (PCIe Gen3 x8) for Data

16 cores

64 GB

2 x 25 GbE
Number of nodes Storage CPU RAM Network

6 or more

2 x 500 GB HDD RAID1 for OS

8 x 500 GB SAS12G SSD RAID10 (PCIe Gen3 x8) for Data

24 cores

96 GB

2 x 25 GbE

NiFi

  • Minimal

  • Proof of concept

  • Commodity

  • Balanced

  • Compute-intensive

Storage CPU RAM Network

400 GB SSD for OS

500 GB SSD for Data

8 cores

32 GB

1 x 10 GbE
Number of nodes Storage CPU RAM Network

2 or more

1 x 500 GB HDD for OS

1 x 500 GB SSD for Data

8 cores

32 GB

1 x 10 GbE
Number of nodes Storage CPU RAM Network

2 or more

2 x 500 GB SSD RAID1 for OS

4 x 500 GB SAS12G SSD RAID10 (PCIe Gen3 x4+) for Data

12 cores

64 GB

2 x 25 GbE
Number of nodes Storage CPU RAM Network

2 or more

2 x 500 GB SSD RAID1 for OS

8 x 500 GB SAS12G SSD RAID10 (PCIe Gen3 x8) for Data

24 cores

96 GB

2 x 25 GbE
Number of nodes Storage CPU RAM Network

4 or more

2 x 500 GB SSD RAID1 for OS

16 x 500 GB SAS12G SSD RAID10 (2 controllers PCIe Gen3 x8) for Data

48 cores

192 GB

2 x 25 GbE

KSQL

Load pattern Number of nodes Storage CPU RAM

All

2 or more

SSD (volume depends on the number of requests and aggregations)

4 cores

32 GB

REST Proxy

Load pattern Number of nodes Storage CPU RAM

All

2 or more

HDD (no speed requirement)

16 cores or more (work with producer/

consumer)

1 GB

64 MB per producer

16 MB per consumer
Found a mistake? Seleсt text and press Ctrl+Enter to report it