Building a simple instant messaging application might not be the most complicated task in our tech world. However, building an app that can handle even the fragment of a scale at which the top chat apps like WhatsApp operate requires a well-thought-out messaging infrastructure. Let’s focus primarily on the instant messaging protocols.
There are several options available, most popular of which would be XMPP, MQTT, SIP and WebSockets. Today, we’ll have a look at XMPP implementation – ejabberd specifically (an open-source XMPP server with integrated MQTT broker and SIP service) and how they are being used at enormous scale by giants such as Meta, Nintendo, Epic Games and more.
How WhatsApp uses XMPP protocol and ejabberd
WhatsApp started as a mere text status service built on PHP and MySQL, but when its founders realized people are using it to chat rather than post statuses, they started looking for a chat server. Facebook was using ejabberd at a time (they seem to have switched, at least partially, to using MQTT in their mobile applications), which might have served as an inspiration to choose said server as WhatsApp’s core technology.
Being the most popular instant messaging service in the world with over 2 billion active monthly users in over 100 countries is an enormous magnitude of operations. To support such a scale, WhatsApp uses XMPP (a heavily modified version, to be more precise) on the ejabberd server (which is also heavily modified to improve and optimize performance).
Considering they haven’t switched to anything else, it is safe to assume that, although not perfect out-of-the-box, both solutions provide the necessary flexibility to build exactly what your particular product needs and not worry that you won’t be able to scale it. WhatsApp processes over 100 billion messages daily and still everything works like a charm.
How Nintendo Switch uses XMPP protocol and ejabberd
In 2019, Nintendo shed some light on the Nintendo Switch Push Notification infrastructure (NPNS), showcasing how XMPP and ejabberd are used to power a system that, as of today, serves around 80-90 million active users (according to this Nintendo’s presentation).
Considering NPNS is a large-scale real-time messaging protocol that has to stay constantly connected to the infrastructure, listening for push notifications, it has to be able to handle millions of concurrent connections. The push notifications in questions include user-based (aimed at those who are active, eg playing a game) and system-based (updates, news and announcements, downloads etc).
Similarly to the WhatsApp’s case, XMPP and ejabberd had to be customized to meet the business requirements, especially the scale. Number-wise, as of that 2019 presentation, NPNS was able to handle 10 million simultaneous connections, 2 billion daily sent messages, 100-200 thousand connections per node and 600 messages a second.
To read more about the issues encountered and how Nintendo handled them, feel free to read this article.
How Trans.eu uses XMPP protocol and ejabberd
Trans.eu is one of the largest logistics platforms in Europe and has been our client for more than 20 years now. Considering the magnitude of our cooperation, we had a chance to develop their large-scale enterprise communication system, also powered by ejabberd, amongst a number of other services.
To tell you more about the decision for this particular tech stack, I sat down with Paweł Dorofiejczyk, the Lead Architect who leads the team implementing the protocol. Here’s what he had to say about the reasoning behind the decision.
‘We opted for the XMPP protocol because it is an open, mature and well-established messaging protocol. In our case, the availability of client libraries for various technologies was of critical importance. It meant we were able to quickly implement it within our web application, mobile apps for Android and iOS, and finally a desktop application for Windows.
We also took advantage of the wide possibilities of extending the XMPP protocol with custom functionalities that Trans.eu was asking for.
The ejabberd server, which implements the XMPP protocol, proved to also be a perfect base for the Trans.eu messaging backend. The server is fast, scales perfectly and allows us to expand its functionalities without interfering with the server code.
During our work on the server, we subjected the ejabberd functionality to considerable modifications and added new functions to enable the fulfilment of specific business needs. We have implemented such functionalities like:
- offline messaging;
- message archiving;
- strangers' status subscriptions;
- sending messages in the business context of logistics marketplace;
- offline text status handling;
- sending system messages via the platform; and
- sending push messages for mobile applications.
Stats-wise, at the moment, the communicator handles:
- 70 thousand concurrently chatting users;
- who send about 300 messages a second; and
- 10 thousand statuses each second.’
How Fortnite uses XMPP protocol and ejabberd
I don’t think Fortnite requires any introduction. If you don’t know it (lucky you), then your kids or nieces definitely know of it and most likely danced some Fortnite dances at some point. All-in-all, it’s a vastly popular online game with over 250 million monthly active users and 25-35 million daily active users.
When in 2018 a service outage happened, Epic Games released a detailed postmortem, explaining what exactly happened. The most interesting part for us is the part where they showcased how XMPP was being utilized within the game.
Epic XMPP (apparently that’s what they call it) supports end-to-end encryption over TCP and WebSockets and is powering their:
- online presence service;
- push notifications;
- whispers (in-game private chats); and
- group chats for parties, teams and global rooms.
Number-wise, if we were to extrapolate the usage statistics they have shown in 2018, as of 2022, the systems handles something along the lines of the below numbers:
- Total throughput in packets:: ~6 million a second (including aux traffic, forwarding and broadcast effect);
- Presences: ~1.8 million a second;
- Notification: ~500 thousand a second; and
- Messages: ~400 thousand a second.
More can be found in the aforementioned postmortem.
How League of Legends uses XMPP protocol and ejabberd
League of Legends. Slightly less popular than Fortnite (only about 11 million concurrently active users), it still runs at a scale larger than many popular applications. Unfortunately, not much is available publicly apart from what was shown in 2014. It could mean that no drastic changes were made, and the system we’ll talk about in a moment was only further developed and improved rather than discontinued.
Let’s have a look at a presentation by Michał Ptaszek, Technical Lead at Riot Games.
As of 2014, LoL’s player base consisted of 70 million active users (125 million as of now, so 2x scale) and they needed a messaging system that works flawlessly.
Can you imagine a multiplayer game with a laggy chat system? No, thank you. Especially a MOBA-type game. Riot Games started with an ejabberd server as a foundation and built on top of it, customizing and extending its functionalities.
The end product provided support for the following scale of usage (once again, as of 2014):
- 67 million unique monthly players (not including other services that use a chat service);
- 27 million daily players;
- 7.5 million concurrent online players;
- 1 billion daily events routed per server, which utilized only 20-30% of available CPU and RAM;
- 11 thousand messages a second;
- a few hundred chat servers deployed around the world and managed by only 3 people; and
- 99% uptime.
According to this series of articles Riot published on their dev blog, a set of XMPP alternatives were also looked into at a time of their preliminary research for a technology that would power their communications, although none seem to outperform ejabberd in the following six areas:
- support for 1:1 private messages, group chats, and offline messages;
- built-in presence mechanisms;
- open specifications for transparency and auditability that would allow for Riot-specific tuning;
- proven security mechanisms;
- customizability;
- availability of open source projects.
How EVE Online uses XMPP protocol and ejabberd
Yet another interesting case comes from the gaming industry. EVE Online is a popular space-based MMORPG with roughly over 250 thousand daily active players. In 2018, they introduced a completely new backend system for their chat services. The decision to switch to ejabberd came to EVE developers when they started breaking up the game’s monolith architecture into smaller microservices.
When it comes to the in-game chat, not only did CCP Games want to separate the chat from the game servers, but to also move it completely outside the game cluster, hosting it on a dedicated cloud cluster.
Why?
- Text messaging services took up a significant portion of CPU usage on their game server's proxy nodes.
- Running it as a standalone service, powered by an industry-standard XMPP protocol offered them a range of future development possibilities.
- Chat logs were much easily accessible to the Customer Support team.
Interestingly, the prototype for this new backend was first mentioned in a Philip Pettersson’s Bachelor thesis back in 2009. It looks like EVE’s previous system was an LSC system that had an unwanted impact on the game servers’ performance, as it would pent-up with time and eventually became a problem.
The thesis gives an in-depth perspective, so I highly recommend checking it out.
Other companies that use XMPP protocol and ejabberd
Not every company shares details of how and where their protocols are being used, so we’re left with a basic indication that they do use XMPP and ejabberd to power their communications. Here’s a list of such organizations:
- Neverwinter: ~16 million users;
- EA Origin: ~40 million users;
- Star Trek Online: ~900 thousand users;
- Champions Online: ~345 thousand users;
- Kik Messenger: ~300 million users;
- Zoom: ~200 million users;
- Grindr: ~4 million users;
- Google Cloud Print: precise number of users unknown;
- Firebase Cloud Messaging : precise number of users unknown;
- Logitech Harmony Hub: precise number of users unknown;
- GitHub: precise number of users unknown;
- Google Push Notifications: ~1.5 billion users;
- Apple Push Notifications: ~500 million users;
- GMX: precise number of users unknown;
- Orange: precise number of users unknown; and
- Facebook: precise number of users unknown.
Customizing XMPP and ejabberd
As you can see from the examples above, XMPP and ejabberd are great at handling communication at an incredibly large scale. But, to achieve sufficient performance and fit a specific use case, customization is required.
Considering our extensive experience with developing custom ejabberd servers and XMPP-based communicated services, we are happy to help you optimize your own ejabberd-powered solution or develop a new one from scratch.
If you’re interested in a free consultation with our Messaging Lead Architects, drop me a line at ross@rst.software, and I’ll make it happen.
.png)
