How Ninja Van uses a dedicated micro-service to facilitate retrying of Kafka messages at scale.


Failures from consuming Kafka messages are inevitable. Fortunately, Kafka handles most causes like network failures, uncommitted messages with its fault-tolerant architecture.

Unfortunately, errors can happen on the application side — things like database constraints being violated or dirty data could cause a message consumption to fail. These errors are terminal, and we should not bother retrying them as they are doomed to fail again.

Some errors are “fixable”. For example, a temporary outage of a dependency, or concurrency issues, can cause an error. …

Instead of hard-coding your data, why not load them from a JSON file?

Using this system, you’ll be able to quickly and easily tweak your game behaviors via JSON. Your non-programmer teammates can also help without touching any code!

Building a bamboo plan to restore archived logs on demand

Introduction to Logs & Archival at Ninja Van

At Ninja Van, we use the Elasticsearch/Fluentd/Kubernetes (EFK) for our logging purposes. If you wish to know more about how we implemented it, you might want to read about it here.

Delivering more than 2 million parcels per day at peak, we experience immense traffic volumes and logs from our different micro-services. At the moment, we have more than 5TB worth of logs per day. …


“Want to join a competition and make a game? It’s a $1 million dollars prize pool!” — asked Timothi when we met up when he came back to Singapore for a visit during his term break in March 2019.

Without giving much thought, I agreed. What I did not expect was a wild ride over the subsequent seven months, where we had the privilege of being one of the few people in the world to develop a game using Niantic’s proprietary technologies.

Niantic Beyond Reality Developer Contest

Within a span of a week, we had roped in three more people into the group, bringing our…

Baron Chan

Software Engineer (Backend) in Singapore

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store