LightQ Performance: transient 1M+ msg/sec, durable 300K+ msg/sec with latency in micro seconds

LightQ is a high performance, brokered messaging queue which supports transient (1M+ msg/sec) and durable (300K+ msg/sec) queues. Durable queues are similar to Kafka where data are written to the file and consumers consume from the file. 
See the blog post for more details.  
For source code, LightIO/LightQ

End to end average latency  is calculated as below.
  • Producer sends first message with current timestamp
  • Consumer stores this time stamp as a start time
  • Consumer stores current time stamp when last message is received.
  • Latency = (Consumers's received last message timestamp - Producer's sent first message timestamp)/total messages received
Laptop hardware:
MacBook Pro (Retina, 15-inch, Late 2013)
Processor 2.3 GHz Intel Core i7
Memory 16 GB 1600 MHz DDR3
Broker Type: Transient

Create a transient topic (test), num partition 1

./lightq-topic -n 1
topic test created successfully

100 bytes, 10M messages

Producer: ( 1250000 msg/sec, Bandwidth: 119.2093 MB)
./lightq-producer -m 10000000  -n 1 -s 100
Topic[test_1], Producer: first message sent timestamp [6378502l]
Topic[test_1], Producer: last message sent timestamp [6386647l]
Topic[test_1], Total message sent [10000000] in [8.00] sec
Topic[test_1], Average messages sent per second [1250000.00]
Topic[test_1], Average bandwidth sent per second [119.2093] MB
Consumer: ( 1250000 msg/sec, Latency: 813 ns, Bandwidth: 119.2093 MB)
./lightq-consumer -m 10000000  -n 1
Topic[test_1], First message:  producer start time [6378502], consumer start time [6378520]
Topic[test_1], Total message received [10000000] in [8.00]sec
Topic[test_1], Average messages received per second [1250000.00]
Topic[test_1], Total bytes received [999999907], average bandwidth received per second [119.2093]MB
Topic[test_1], Last Message: consumer endtime [6386632]
Topic[test_1], Consumer first message latency [18] ms
Topic[test_1], Consumer last message received -   producer first message sent timestamp [8130]
Topic[test_1], Average latency [813.00] nano sec

256 bytes, 10M messages

Producer: ( 1250000 msg/sec, Bandwidth: 305.1758 MB)
./lightq-producer -m 10000000  -n 1 -s 256
Topic[test_1], Producer: first message sent timestamp [5826133l]
Topic[test_1], Producer: last message sent timestamp [5834823l]
Topic[test_1], Total message sent [10000000] in [8.00] sec
Topic[test_1], Average messages sent per second [1250000.00]
Topic[test_1], Average bandwidth sent per second [305.1758] MB
Consumer: ( 1250000 msg/sec, Latency: 868 ns, Bandwidth: 305.1758 MB)
./lightq-consumer -m 10000000  -n 1
Topic[test_1], First message:  producer start time [5826133], consumer start time [5826155]
Topic[test_1], Total message received [10000000] in [8.00]sec
Topic[test_1], Average messages received per second [1250000.00]
Topic[test_1], Total bytes received [2559999751], average bandwidth received per second [305.1758]MB
Topic[test_1], Last Message: consumer endtime [5834813]
Topic[test_1], Consumer first message latency [22] ms
Topic[test_1], Consumer last message received - producer first message sent timestamp [8680]
Topic[test_1], Average latency [868.00] nano sec

512 bytes, 10M messages

Producer: ( 1000000 msg/sec, Bandwidth: [488.2812 MB )
./lightq-producer -m 10000000  -n 1 -s 512
Topic[test_1], Producer: first message sent timestamp [6231211l]
Topic[test_1], Producer: last message sent timestamp [6241288l]
Topic[test_1], Total message sent [10000000] in [10.00] sec
Topic[test_1], Average messages sent per second [1000000.00]
Topic[test_1], Average bandwidth sent per second [488.2812] MB
Consumer: ( 1000000 msg/sec, Latency: 1006 ns, Bandwidth: [488.2812 MB )
./lightq-consumer -m 10000000  -n 1
Topic[test_1], First message:  producer start time [6231211], consumer start time [6231235]
Topic[test_1], Total message received [10000000] in [10.00]sec
Topic[test_1], Average messages received per second [1000000.00]
Topic[test_1], Total bytes received [5119999495], average bandwidth received per second [488.2812]MB
Topic[test_1], Last Message: consumer endtime [6241279]
Topic[test_1], Consumer first message latency [24] ms
Topic[test_1], Consumer last message received -   producer first message sent timestamp [10068]
Topic[test_1], Average latency [1006.00] nano sec

1024 bytes, 10M messages

Producer: ( 526315 msg/sec, Bandwidth: 513.9802 MB)
./lightq-producer -m 10000000  -n 1 -s 1024
Topic[test_1], Producer: first message sent timestamp [6029718l]
Topic[test_1], Producer: last message sent timestamp [6049370l]
Topic[test_1], Total message sent [10000000] in [19.00] sec
Topic[test_1], Average messages sent per second [526315.81]
Topic[test_1], Average bandwidth sent per second [513.9802] MB
Consumer: ( 526315 msg/sec, Latency: 1964 ns, Bandwidth: 513.9802 MB)
./lightq-consumer -m 10000000  -n 1
Topic[test_1], First message:  producer start time [6029718], consumer start time [6029727]
Topic[test_1], Total message received [10000000] in [19.00]sec
Topic[test_1], Average messages received per second [526315.81]
Topic[test_1], Total bytes received [10239998983], average bandwidth received per second [513.9802]MB
Topic[test_1], Last Message: consumer endtime [6049366]
Topic[test_1], Consumer first message latency [9] ms
Topic[test_1], Consumer last message received - producer first message sent timestamp [19648]
Topic[test_1], Average latency [1964.00] nano sec
Broker Type: Durable (file)

Create a durable topic (test), num partition 1

./lightq-topic -n 1 -s file
topic test created successfully

100 bytes, 10M messages

Producer: ( 400000 msg/sec, Bandwidth: 38.1470 MB)
./lightq-producer -m 10000000  -s 100 -n 1
Topic[test_1], Producer: first message sent timestamp [4906704l]
Topic[test_1], Producer: last message sent timestamp [4932029l]
Topic[test_1], Total message sent [10000000] in [25.00] sec
Topic[test_1], Average messages sent per second [400000.00]
Topic[test_1], Average bandwidth sent per second [38.1470] MB
Consumer: ( 400000 msg/sec, Latency: 2535 ns, Bandwidth: 38.1470 MB)
./lightq-consumer -c socket -m 10000000 -n 1
Topic[test_1], First message:  producer start time [4906704], consumer start time [4906718]
Topic[test_1], Total message received [10000000] in [25.00]sec
Topic[test_1], Average messages received per second [400000.00]
Topic[test_1], Total bytes received [999999907], average bandwidth received per second [38.1470]MB
Topic[test_1], Last Message: consumer endtime [4932061]
Topic[test_1], Consumer first message latency [14] ms
Topic[test_1], Consumer last message received -   producer first message sent timestamp [25357]
Topic[test_1], Average latency [2535.00] nano sec

256 bytes, 10M messages

Producer: ( 370370 msg/sec, Bandwidth: 90.4224 MB)
./lightq-producer -m 10000000  -s 256 -n 1
Topic[test_1], Producer: first message sent timestamp [4783156l]
Topic[test_1], Producer: last message sent timestamp [4810966l]
Topic[test_1], Total message sent [10000000] in [27.00] sec
Topic[test_1], Average messages sent per second [370370.38]
Topic[test_1], Average bandwidth sent per second [90.4224] MB
Consumer: ( 370370 msg/sec, Latency: 2783 ns, Bandwidth: 90.4224 MB)
./lightq-consumer -c socket -m 10000000 -n 1
Topic[test_1], First message:  producer start time [4783156], consumer start time [4783163]
Topic[test_1], Total message received [10000000] in [27.00]sec
Topic[test_1], Average messages received per second [370370.38]
Topic[test_1], Total bytes received [2559999751], average bandwidth received per second [90.4224]MB
Topic[test_1], Last Message: consumer endtime [4810993]
Topic[test_1], Consumer first message latency [7] ms
Topic[test_1], Consumer last message received -   producer first message sent timestamp [27837]
Topic[test_1], Average latency [2783.00] nano sec

512 bytes, 10M messages

Producer: ( 322580 msg/sec, Bandwidth: 157.5101 MB)
./lightq-producer -m 10000000  -s 512 -n 1
Topic[test_1], Producer: first message sent timestamp [4563339l]
Topic[test_1], Producer: last message sent timestamp [4595009l]
Topic[test_1], Total message sent [10000000] in [31.00] sec
Topic[test_1], Average messages sent per second [322580.66]
Topic[test_1], Average bandwidth sent per second [157.5101] MB
Consumer: ( 322580 msg/sec, Latency: 3168 ns, Bandwidth: 157.5101 MB)
./lightq-consumer -c socket -m 10000000 -n 1
Topic[test_1], First message:  producer start time [4563339], consumer start time [4563361]
Topic[test_1], Total message received [10000000] in [31.00]sec
Topic[test_1], Average messages received per second [322580.66]
Topic[test_1], Total bytes received [5119999495], average bandwidth received per second [157.5101]MB
Topic[test_1], Last Message: consumer endtime [4595027]
Topic[test_1], Consumer first message latency [22] ms
Topic[test_1], Consumer last message received -   producer first message sent timestamp [31688]
Topic[test_1], Average latency [3168.00] nano sec

1024 bytes, 10M messages

Producer: ( 270270 msg/sec, Bandwidth: 263.9358 MB)
./lightq-producer -m 10000000  -s 1024 -n 1
Topic[test_1], Producer: first message sent timestamp [4319543l]
Topic[test_1], Producer: last message sent timestamp [4357308l]
Topic[test_1], Total message sent [10000000] in [37.00] sec
Topic[test_1], Average messages sent per second [270270.28]
Topic[test_1], Average bandwidth sent per second [263.9358] MB
Consumer: ( 270270 msg/sec, Latency: 3779 ns, Bandwidth: 263.9358 MB)
./lightq-consumer -c socket -m 10000000 -n 1
Topic[test_1], First message:  producer start time [4319543], consumer start time [4319564]
Topic[test_1], Total message received [10000000] in [37.00]sec
Topic[test_1], Average messages received per second [270270.28]
Topic[test_1], Total bytes received [10239998983], average bandwidth received per second [263.9358]MB
Topic[test_1], Last Message: consumer endtime [4357340]
Topic[test_1], Consumer first message latency [21] ms
Topic[test_1], Consumer last message received -   producer first message sent timestamp [37797]
Topic[test_1], Average latency [3779.00] nano sec


37 comments:

Anonymous said...

graphs are cool

karthireva said...


Truely a very good article on how to handle the future technology. After reading your post,thanks for taking the time to discuss this content.

Dot Net Training Institute in Chennai

Unknown said...

This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

digital marketing course in chennai

Padhma said...

Your thinking toward the respective issue is awesome also the idea behind the blog is very interesting which would bring a new evolution in respective field. Thanks for sharing.

Home Spa Services in Mumbai

melinamia said...

I’m really amazed with your posting skills as well as with the layout on your blog site. Is this a paid style or did you modify it yourself? Either way keep up the pleasant quality writing, it is rare to see a great site such as this one these days.
Delivery service singapore
Singapore delivery service
24 hour delivery singapore

dazzling said...

very informative blog and very useful for readers. making us to learn more from your blog.keep on updating your blog
java training in chennai
dotnet training in chennai
cloud -computing training in chennai
digital marketing training in chennai

Anonymous said...

very nice blog this course very useful to us this gives the big opportunity to us i like this information very much.


Hadoop Training in Chennai

vinothika0895@gamil.com said...

Content is very clear and easily explained it is very interesting aswell. I hope you postagain soon.

Selenium Training in Chennai

Unknown said...


It's a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much.

VMware Training in Chennai

Unknown said...

I got lot of information from your blog. Keep on sharing like this blog...
SAP MM Training in Chennai

Unknown said...

Its an awesome article. written in very creative way and it also conveys useful information.

Selenium Training in Chennai

Unknown said...

It is really very excellent,I find all articles was amazing.Awesome way to get exert tips from everyone,not only i like that post all peoples like that post.Because of all given information was wonderful and it's very helpful for me.
SAP Training in Chennai
SAP ABAP Training in Chennai
SAP FICO Training in Chennai
SAP MM Training in Chennai

Unknown said...

It is really very excellent,I find all articles was amazing.Awesome way to get exert tips from everyone,not only i like that post all peoples like that post.Because of all given information was wonderful and it's very helpful for me.
SAP Training in Chennai
SAP ABAP Training in Chennai
SAP FICO Training in Chennai
SAP MM Training in Chennai

Sophia sage said...

Now it is known to me that articles is nothing but inspiring is everything to do something great. This is a great article for the people who want to come in freelancing.
Sharepoint Training in Chennai
PHP Training in Chennai
Android Training in Chennai
SAS Training in Chennai
AWS Training in Chennai

Pallavi karthi said...

I truly appreciate this post. I’ve been looking all over for this! Thank goodness I found it on Bing. You have made my day! Thanks again! Keep update more excellent posts..
Manuscript writing services
Journal Of Zoology
Journal Of Medical Sciences
Botany Journal
Journal Of Agricultural Science

JP said...

Amazing article for big data content in the internet. I loved reading how things are made to happen here. Very informative and equally impressive.

Best Dental Hospital in Chennai | Best Dental Implant in Chennai | Best Rhinoplasty Surgeon in India | Cosmetic Dentistry | Root Canal Treatment in Chennai

Jeyapragash R said...

Nice info...

Corporate Training in Nigeria

cdcdcdcd said...



read this above post its very greatful for me thanks sharing this post ,great post.
Best Ice Fishing Gloves Best Ice Fishing Gloves Best Ice Fishing Gloves

Karthik said...

Excellent blog. It is very impressive for readers to read such a blog. Keep on updating.

Invisible Teeth Braces
Dental Implants
Laser Root Canal Treatment
Laser Gum Treatment
Book An Appointment

Anonymous said...

Кожаные сумки не выходят из моды, актуальны всегдаи к тожу же при правильном ношении сливаются практически с любой верхней. Подобное изделие выделяется длительным временем ношения и прекраснымстильным внешним видом. При должном уходе сумка из натурального природного материала проработает гораздо дольше, нежели из не натурального, тем более что со временем она становится еще лучше, раскрывая свой потенциал. Отобрать сумку из кожи можно воспользовавшись магазином интернет магазин женских сумок, ведь там собрано достаточно вариаций сумок. На сайте вы найдете отзывы клиентов, которые помогут определится и сделать хороший выбор сумочки.

PMP said...

I was basically inspecting through the web filtering for certain data and ran over your blog. I am flabbergasted by the data that you have on this blog. It shows how well you welcome this subject. Bookmarked this page, will return for extra.

PMP

tejaswini said...

incredible article!! sharing these kind of articles is the decent one and I trust you will share an article on information science.By giving an organization like 360DigiTMG.it is one the best foundation for doing guaranteed coursesincredible article!! sharing these kind of articles is the decent one and I trust you will share an article on information science.By giving an organization like 360DigiTMG.it is one the best foundation for doing guaranteed courses


data science certification

360DigiTMG said...

Nice work... Much obliged for sharing this stunning and educative blog entry!
hrdf training course

360digitmgdelhi said...

I will truly value the essayist's decision for picking this magnificent article fitting to my matter.Here is profound depiction about the article matter which helped me more.
data science course delhi

Anonymous said...

Кафельная плитка Cisa подразделяется на несколько типов - для пола и уложенная на стены. Выделяется покрытие по величине, плотности и толщине. Отдельно стоящим типом стоит плитка из керогранита.

Anonymous said...

Гадания Таро онлайн на любовь мужчины это шанс понять приближающиеся действия постоянно манил человека. Предсказание судьбы дает угадать, что человека подстерегает в ближайшем времени. Каждый хочет предугадать свою судьбу и представляет определенные виды ворожбы гораздо больше действенными.

Anonymous said...

Берите максимальные выгоды только здесь детские игровые комплексы купить спб. Далее отмечены наиболее важные достоинства взаимодействия с нашей компанией. Большой число инвентаря для коммунальной сферы.

Anonymous said...

Занять большую сумму на карточный счет на удивление легко. Онлайн сервис набора микрофинансовых услуг разрешает утвердить нужный займ. Онлайн займ с 18 лет – это оптимальный прием заполучить наличные средства в минимальный промежуток времени.

Anonymous said...

Важно не забывать, что ресурс казино х регистрация зачисляет своим игрокам дополнительные бонусы. Регистрируйтесь на сайте и забирайте специальные вознаграждения в личном кабинете. Дополнительный счет даст возможность заработать много денежных средств.

Anonymous said...

Именно на портале mr bit отзывы геймеры с легкостью загрузят понравившуюся сетевую игру для развлечений. Наведавшись в компьютерный клуб, вы имеете возможность окунуться в увлекательные авантюры. Веселиться в компании друзей намного интереснее, чем сам по себе в своей квартире.

Anonymous said...

Следует уделить особое внимание выбору действительно надежного ресурса. В мире имеются сотни web-казино на реальные деньги. Именно на портале джойказино промокод бездепозитный бонус пользователи смогут поднять серьезные бабки.

Anonymous said...

Бренд sotni.ru реализует cersanit затирка для плитки уже свыше 12 лет. Серьезный и проверенный импортер всегда сможет показать широкий перечень облицовочного покрытия для прихожей.

Anonymous said...

Серьезный и авторитетный продавец всегда может показать многочисленный выбор облицовочного покрытия для уборной. Фирма Сотни.ру продает керамическая плитка cersanit на протяжении двенадцати лет.

training institute said...

I am impressed by the information that you have on this blog. It shows how well you understand this subject.
data science course


data analyst course malaysia said...

This is the first time I visit here. I found such a large number of engaging stuff in your blog, particularly its conversation. From the huge amounts of remarks on your articles, I surmise I am by all accounts not the only one having all the recreation here! Keep doing awesome. I have been important to compose something like this on my site and you have given me a thought.

PMP Training in Malaysia said...

360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

Anonymous said...

Hii
This is a genuinely nice and captivating post. I was in search of precisely this kind of information and thoroughly enjoyed reading it. Thank you for sharing, and please continue to post more. Thanks Here is sharing some Enovia PLM Training information may be its helpful to you. Enovia PLM Training