ORIGINAL PREPARED ANSWER
Hikari pool issue
Shown as prepared. Grammar, structure, names and confidential details have not been corrected or anonymised.
Original prepared answer
Situation
We observed high latency in our APIs , found numerous errors in the flog related to connection unavailability and request timeouts.
These problems appeared for some time and it affected only one set of pods of our BKS cluster.
The problem got mitigated by disabling one of the regions which also removed the affected pod.
Initially we thought of some issues with one region's connection to the mysql cluster.
But the issue reappeared in a few days with similar patterns. So we decided to deep dive.
Action
-
We observed that the requests were either very high latency or were timing out.
-
One level deeper, we saw that the error was due to mysql timeout.
-
And it was timing out for all types of queries. So we knew it was not application code.
-
Then further digging showed us that the timeout was during application trying to get a connection object from HikariConnectionPool. The error was that the application was waiting to fetch any new connection from the connection pool and was getting timed out.
This was one stage of investigation.
Now we had to figure out why the getConnection call was timing out. We started hunting for MySql cluster metrics for wallet schema.
Most of the metrics were normal.
-
Then we found some HikariPool metrics and found that it showed a spike in pending-connections.
-
We researched more on it and learned that spike in pending connections mean that the Hikari library was not able to establish new connections with the mysql cluster
-
So we studied further on how Hikari connections work. Hikari has 3 threads.
-
One is responsible for expiring connections constantly due to their maximum lifetime (2 minutes), Hikari simply closes them without taking the remaining number of connections into account.
-
2nd thread spins up new connections in another thread and its a blocking process
-
3rd thread is responsible for orchestrating the connections from the pool
-
During the connection creation, threads continue to be closed due to max lifetime. Normally this isn't an issue since it's relatively fast (< 1s) but due to network / MySQL issues the TCP connection can get "stuck"
-
We ended up in a stage where the getConnection method kept waiting for a connection object but the pool was empty as existing connections were all dead and no new connection object was getting created because of some issues.
We then reached out to the MySql team to check if they knew about this issue.
-
Then they pointed us to a workplace post about an ongoing issue where mysql boxes were getting flipped quite frequently and this rested in HikariCP new connection creation issue. And as mitigation it asked application developers to add socket-timeout parameter in our mysql settings
-
So the way booking works is. In every pod, there is a roster file which has all the ips of the mysql boxes. And the application code looks into this roster file to pick ips and connect to the cluster
-
And when a box is flipped this roster is updated with a new ip
-
Now due to frequent box flips, the roster would change
-
Imagine a situation when the application just reads the rooster and the Hikari is trying to create a connection to a box. But the box had by then flipped and the ip does not exist anymore
-
The Hikari connection creation step stays in a hung state.
-
But if you add SocketTimeOut, any socket connection will time out after the threshold and not stay in hung state
How this answer addresses the interviewer signals
The answer shows a systematic investigation of a recurring production issue: the example moves from API symptoms to connection acquisition and draws on database-team knowledge to identify a plausible mechanism. Its strongest signals are persistence, diagnostic narrowing, and collaboration. The exact causal confirmation and post-change outcome remain areas for follow-up; the narrative ends with a recommended timeout mitigation rather than a verified production result. These signals are discussion areas, not a checklist every answer must cover. Follow-up questions are optional prompts to explore the story further.
This assessment reflects the written answer, not the author’s full experience.
Problem framing and persistence
DemonstratedEvidence in the answer: The answer describes high latency and timeouts affecting a subset of pods, an initial regional mitigation, and recurrence that prompted a deeper investigation.
Possible interviewer follow-up: What was the user impact, and what did the recurrence tell you about the original regional-connectivity hypothesis?
Evidence-led investigation
DemonstratedEvidence in the answer: The investigation progresses from API latency to database-related timeouts, then to waiting for a Hikari pool connection. The example compares database and pool metrics to narrow the focus.
Possible interviewer follow-up: Which logs or traces showed that time was spent acquiring a connection rather than executing a query, and how did you rule out application-side contention or connection leaks?
Root-cause reasoning
Partially demonstratedEvidence in the answer: The answer connects depleted pool capacity and stalled connection creation to changing database hosts, supported by information from the MySQL team. It does not describe direct confirmation of the blocked connection stage or how competing explanations were excluded.
Possible interviewer follow-up: What evidence confirmed the roster-change race, what were the active/idle/total pool counts, and which timeout applied to the blocked stage: pool acquisition, socket connection establishment, or socket operations?
Collaboration and technical ownership
DemonstratedEvidence in the answer: The answer describes tracing the failure through the application and pool layers, then contacting the MySQL team, who connected the symptoms to a known issue and recommended a timeout setting.
Possible interviewer follow-up: Which parts of the investigation did you personally lead, and how did you and the MySQL team confirm that the known issue matched your incident?
Validation and prevention
Not demonstratedEvidence in the answer: The answer explains the recommended timeout mitigation but does not state that it was deployed or describe subsequent recovery, monitoring, or recurrence checks.
Possible interviewer follow-up: Did you deploy the setting, how did you choose and test the value, and what did latency, pool metrics, and later database-host changes show afterwards?