Our learnings from the biggest night of the year

To say that NYE is a busy time for us at Zomato is an understatement. Every year, we begin preparing for high order volumes keeping the previous year as our baseline. In 2020, we hit over 4,000 orders per minute (OPM) on NYE. In 2021, we delivered almost double that number per minute, surpassing our highest expectations. But there’s still a long way to go as India trusts us to deliver their favourite food all year round.

There are many pieces that stitch the experience of a single order together — right from when you run a search, put in filters, build your order, make payment, get an estimate of the delivery time, to tracking the delivery partner and making sure the order reaches you on time.

We make sure our customers have the best experience possible. But when the traffic actually hits, even the best-laid plans can go haywire. Thinking on our feet and keeping the experience smooth for the customers is our main priority, but monitoring all the metrics to ensure we don’t drop the ball for each order delivered on NYE is like Keanu Reeves dodging those bullets in The Matrix. 

Which is why we thought, what better than to share more about our experience with the tech community out there:

NYE is one of the most important days for us. Every system is stretched to its limit, be it services, databases, or logistics. Preparations start weeks in advance. From preparing projections to latency optimisation to system scale-up, NYE then becomes a waiting game as to when we will break our previous highs of throughput and OPM.

by Sumanshu Dwivedi – Data Platforms and Logistics Tech Team

Even my two monitors and a laptop screen were not enough for the numbers I was monitoring that day. The focus was on ensuring the error rate remained below the threshold. The chills from a small abnormality in data, or the happiness of breaking multiple records, all are one hell of a learning experience.

Due to the impeccable preparation by the android team before NYE the statistics ended up even better than we expected.

by Asheesh Sharma – Android Team 

On NYE, we wanted to serve as many customers as we could, while maintaining a good ordering experience. For that, we had to throttle demand at places where the delivery partner availability and the restaurant kitchen bandwidth were quickly getting consumed as orders started flooding in. Having an automated demand management system wasn’t enough, we took manual control of this at times.

by Praveen Omar – Delivery Partner Experience Team 

The over 2.5 million orders we serviced on NYE came from across the globe. People from different parts of the world were ordering food for loved ones in India. So, our customer support was dealing with a global audience. The high order volume meant we had to steer the ship without the help of machines at times. But given the level of preparation we had made, it was all par for the course.   

The iOS team was well-prepared for NYE with a stable release on the App Store, and setting up alerts for order flow in case something goes wrong. We were continuously checking our crashes, out-of-memory sessions, API failures, and business metrics to ensure our customers have the best ordering experience.

by Aditya Jain – iOS Team 

It was like the final exam for which you have been preparing for weeks. If you prepare well, the numbers thrill you, not scare you. It was quite exhilarating for me, but I had to pay attention to all the customer feedback. Listening to our customers’ feedback in real-time while helping our customer support team opened my eyes to so many unsolved things.

by Ritesh Dalal – Restaurant Partner Experience Team 

On NYE, we were tense with questions; will mongo remain performant? Can MQTT handle the load? We tested our systems at 10x the load that we expected during NYE. We had been capturing the production traffic after amplification and randomization of the data. We wanted to load the system in exactly the same manner as anticipated on NYE. We tested by replaying the same data on a production clone. Our in-house platform ‘One Support’ performed exactly the same way as anticipated on NYE.

Thankfully, all our planning and efforts paid off, and we saw the highest ever order volume.

by Kumar Gyanendra- Customer Experience Team 

It was all about monitoring cities driving the largest volume for various business metrics and making real-time configuration changes wherever we noticed the system breaking. The increased order volume also had me worried about spikes in memory or processing time leading to unexpected crashes. Being in a state of “all systems alert” is a big part of running NYE smoothly.

by Tushar Dudani – Order Fulfilment Team

Preparation for the scale we achieved on NYE and the non-existent margin of error helped us flush out a lot of kinks from our systems. Tuning all circuit breakers, having kill switches for each feature, finding bottlenecks for each database and their mitigation strategies were just a few among the many tasks we had to do to be ready for the NYE.

by Yasoob Haider – Food Delivery Search Team

We were all hands on deck to monitor the order-level metrics of our restaurant partners. To run a large operation like this, we are dependent on multiple external service providers too. So while it is easier to plan for things in our direct control, it is equally important to anticipate possible breakpoints at the end of our service providers. We faced disruption when our third-party call provider’s services were impacted, however, we were able to minimise the impact on our orders by quickly switching to an alternate service provider. 

Because there was no room for error, we prepared for, well, the apocalypse. We had a code to crack every foreseeable and unforeseeable problem. 

by Satyaprakash Dwibedi – Restaurant Partner Engineering Team 

One must always be prepared for the unexpected on NYE. As a developer, you should be always prepared to have a fallback mechanism on what to do if anything goes wrong. Every line of code has its own importance in solving problems, so think about that while building products.

by Praneel Rathore – Customer Experience Team

NYE is a big day for our restaurant partners; not only do all services have to be provided timely, but every product needs to operate seamlessly. On such a day, real-time logging and system monitoring is key – keeping a close eye on timed out orders and requests, the orders per minute rate, and the successful relay of orders to our POS vendors, kept us on our toes. Given the scale we were operating at, many things could break, but the anxiety is all worth it when we see improved statistics every year.

by Tushar Mohan – Restaurant Partner Engineering Team 

On NYE there is a massive traffic surge for payments which is why we keep our systems scaled to handle peak loads. The major challenge during NYE dinner peak is to handle the downtimes at different payment gateways, providers and networks routing traffic with minimal damage so as to provide a seamless payment experience across multiple payment methods for our customers.

by Sahil Arya – Payments Engineering Team

We strive to keep the systems up and running flawlessly 100% of the time. NYE reinforces that building feature toggles and kill switches should be a habit and not an event prep toolkit. Database, cache queries and other IO operations should also be metricised. 

While it is always important to code the “right way” without compromising on customer experience, on such important and high volume days, the equation changes a bit for hot-fixes. You need to prioritise experience rather than building stuff the right way. 

by Gaurav Gupta – Restaurant Partner Experience Team

A look at the last day of 2021

What goes on during the last day of the year at Zomato is a different rush altogether. Every year, our customers surpass our expectations. It is these challenges we thrive on. Because nothing gives us greater pleasure than solving real problems for our customers.

So keep the orders, and the challenges, coming!