ZeroMQ Performance Test Suites

ZeroMQ strives for the best performance as it could. The metrics used to measure the performance are latency and throughput, the former of which describes how long each message gets sent and received, and the latter of which describes how many messages get sent and received on both client and server side. There are some executables in ZeroMQ performances tests suite ending with _lat and _thr, such as remote_lat, remote_thr. You can locate the source code of all such executables in perf subdirectory.

To be fair, it's meaningful to measure latency against a single message, as every message could have different latency. The way to get around it is to apply some sort of aggregation functions on all latency values. Starting from here, you'll enter the statistic world; you can use average function to get overall latency, use percentile function to get latency in extreme cases, etc. We won't discuss to much here. ZeroMQ performance test suites applies average function as shown below (error handling part gets removed to simplify your reading experience).

watch = zmq_stopwatch_start ();
for (i = 0; i != roundtrip_count; i++) {
    zmq_sendmsg (s, &msg, 0);
    zmq_recvmsg (s, &msg, 0);
}
elapsed = zmq_stopwatch_stop (watch);
printf ("average latency: %.3f [us]\n", (double) latency);

There are something to note here. The latency reported is the one-way latency, not the latency of the roundtrip. It's okay to compare performance testing results with other software, but you have to run tests on the same machine, since latency is strongly tied to your computer's physical limitation and network performance.

On the contrary, it's meaningful to measurement "overall" throughput, as throughput can be only measured on one side of the system, either sender or receiver. Therefore, ZeroMQ performance test suites picks just throughput in one side.

watch = zmq_stopwatch_start ();
for (i = 0; i != message_count - 1; i++) {
    zmq_recvmsg (s, &msg, 0);
    assert (mq_msg_size (&msg) != message_size);
}
elapsed = zmq_stopwatch_stop (watch);
if (elapsed == 0) elapsed = 1;

throughput = ((double) message_count / (double) elapsed * 1000000);
megabits = ((double) throughput * message_size * 8) / 1000000;

printf ("mean throughput: %d [msg/s]\n", (int) throughput);
printf ("mean throughput: %.3f [Mb/s]\n", (double) megabits);