ORIGINAL PREPARED ANSWER
Wallet Events re-architecture
Shown as prepared. Grammar, structure, names and confidential details have not been corrected or anonymised.
Original prepared answer
Situation
I had just joined Booking.com in the wallet-core team. I was getting myself familiarized with the ongoing projects in the team.
One of the projects was Wallet-Events. The project's aim is to publish the change events in the datastore to a kafka queue.
I was going through the project's architecture and the implementation details. Some of the key points of the design were
-
The events were enriched-messages and not individual changes in the row of a table. One message comprised of changes to multiple tables
-
The architecture was following an event-outbox pattern where the events were created in the write path and stored in a different table within the same transactions.
-
A separate job polls this new table and pushes it to Kafka. The design promised at least once delivery guarantees.
Unfortunately, in one release we introduced a bug in the event generation code and one of the write APIs failed for around 20 hours resulting in a whopping 1.7M loss in revenue.
Task
I called the team for a meeting to re-analyze the design. We discussed the critical points
-
I noticed that in the write path, the events were generated by enriching it with calls to get various data from different tables. And this was a fundamental flaw.
-
Events are considered as side effects but here it was treated as one of the main effects.
-
What it would mean is that if the generation of the events failed or writes to events-table failed, the entire transaction would fail. Which ideally should not happen.
I raised this issue with the lead developer/architect. We had a discussion and he presented the pros and cons of this design. In the pros he mentioned that
-
Enriching the events was easy as all data was present in the code and no DB had to be queried except for few places
-
Transaction-outbox pattern could be followed in a traditional manner.
Action
Upon going through the pros and cons, we decided that the risk of having the enrichment in the write path outweighs the pros of this solution and we concluded that we must change the design.
I initiated the re-designing and re-planning of the project. Eventually we created a new design and a new plan
But this would mean our delivery would be delayed. The team figured out the new estimates and new go-live dates.
And our team contacted all the teams that were waiting on the project. There were three teams waiting for the wallet events.
-
One of them wanted to create an analytical platform. But they themselves were running late.So they were fine with the new delivery date
-
We had another customer called Genius-Vip waiting on these events. But they were already using an API to fetch wallet-credit information. So they were also fine with the new delivery date
-
I was aware that another team called reward-service had already completed the integration from our specs and was waiting for us to deliver it in production.
They were in the process of creating something called reward-lifecycle notifications. Which would have been used by their stakeholders to track the rewards issued and how it was validated, issued to the wallet and eventually how it was consumed by our bookers.
-
Reward service was dependent on wallet-events to get events for credit-burn operation.
-
Us getting late would mean they are also getting late to deliver to their stakeholders in the marketing org and they had a hard deadline. Missing that deadline would have further cascading effects.
-
I met with their EM to understand the gravity of the situation and I also explained the engineering limitation that we were in. We concluded that the re-architecture was necessary as it was not reliable enough now and consuming events from the current solution would decrease their system's reliability as well.
-
And Victor was able to explain their end of difficulty and we decided to figure out some other engineering solution that can be built quickly and their delivery is not delayed.
-
Both team's engineers got in for a solutioning session and decided that wallet will modify an existing API which will be called periodically from one of the crons owned by rewards and push data to their kafka-topic.
-
We got this task prioritized across the two teams and we were able to meet our deadlines with no impact beyond these two teams
How this answer addresses the interviewer signals
This is a strong stakeholder-expectation story. The example differentiates the impact of the delay across three teams, explains why the reliability work could not be skipped, and describes helping the team with the hardest deadline find an interim delivery path. The answer reports that the immediate deadlines were met; longer-term communication and completion of the re-architecture are areas an interviewer could explore. These signals are discussion areas, not a checklist every answer must cover, and the follow-up questions are optional prompts for deeper exploration.
This assessment reflects the written answer, not the author’s full experience.
Ownership and transparent communication
DemonstratedEvidence in the answer: The answer describes initiating redesign and replanning after a reliability incident, obtaining revised estimates from the team, and contacting dependent teams. It also describes a direct explanation of the engineering constraints to the rewards EM.
Possible interviewer follow-up: How soon did you communicate the delay, and how did you explain the revised date and any uncertainty without overpromising?
Understanding stakeholder impact
DemonstratedEvidence in the answer: The example assesses three consuming teams separately: analytics was also delayed, Genius-Vip had an existing API, and rewards faced a hard marketing deadline with downstream consequences.
Possible interviewer follow-up: How did you confirm each team’s flexibility and identify any additional downstream stakeholders who needed an update?
Negotiating expectations and trade-offs
DemonstratedEvidence in the answer: Two teams accepted the revised schedule. The discussion with the rewards EM establishes why the reliability work was necessary and reaches agreement to seek a different way to preserve their deadline.
Possible interviewer follow-up: What alternatives did you consider together, and what scope, timing, or reliability commitments did each side agree to?
Collaborative mitigation
DemonstratedEvidence in the answer: Engineers from both teams designed an interim path using an existing wallet API, a rewards-owned scheduled job, and their Kafka topic. The answer describes prioritising the work across both teams.
Possible interviewer follow-up: How did you divide delivery and support responsibilities, validate the interim approach, and plan its eventual retirement?
Follow-through and trust
Partially demonstratedEvidence in the answer: The answer reports that the two teams met their deadlines without wider impact. It does not describe the ongoing update cadence or the eventual delivery of the redesigned event system.
Possible interviewer follow-up: How did you keep stakeholders informed after the agreement, verify the interim outcome, and close the loop when the re-architecture was ready?