Following an HR call, the process consisted of a high level interview with a lead followed by three technical interviews centered on ML theory, ML system design, and pair programming. The calls with the recruiter and lead were professional and engaging. The ML theory interview was a masterclass in conducting interviews -- at some point in the process you have to be asked a laundry list of theory questions -- but the interviewer made this too into an engaging back and forth conversation instead of the more typical Q followed by A.
After this positive experience, the pair programming was disappointingly a mess. First, I was supposed to receive a GitHib invite ahead of time, which never came and which the interviewer seemed to implicitly blame the recruiter for. Of course it's the interviewer's responsibility to send a GitHub invite, not the recruiter's. It was rescheduled with the Github link rectified. Not a big deal by itself and not something noteworthy if it weren't a predictor of the code itself and the interview that followed.
The code the exercise was built on was surprisingly amateurish and the goal was explicitly not to fix the existing code, but to add additional functionality on top of what was already there. First, the logic for the main unit test was highly interwoven with the feature itself -- an obvious no-no even if it's your first day on the job. In order to write a unit test for the new feature I added, I had to hack something similar and add an additional hack on top of it due to this structure. When I explained to the interviewer why it was necessary to do things this way, he was surprised it was set up as it was.
The exercise was related to a simulated embedding database. They implemented the basic functionality by using a pandas DataFrame and adding a new row every time an item was inserted. This is another blatant no-no that anyone that has spent any time working with data would know -- adding a row to a DataFrame requires re-allocating memory for the whole existing DataFrame together with the new row. It is so inefficient that pandas in fact removed the public facing function to do this at some point, but that didn't stop this platform team: They hacked it together using a private member function anyway. It would have been both conceptually simpler and more computationally efficient to just use a list of dicts or another similar structure.
The interviewer decided to ask a couple of basic "theory" questions at random while I was coding. This was after the in-depth theory interview given by the ML Scientist. However, he himself did not know the answers to the questions he was asking. He seemed surprised to learn that Euclidean distance and L2 distance are synonymous and had to Google another more subtle but well known property about distance metrics.
After coding up a vector similarity search, the interviewer asked if I was happy with the efficiency of the solution. I explained that in a real database you would implement a graph or metric-based index rather than doing an exhaustive linear search, but this was the best we could do here, and I went over the big-O run time of each section. After he still appeared unsatisfied, I realized he wanted a matrix-vector multiplication instead of a series of vector-vector multiplications. I tried to engage him in a conversation about the pros and cons of this approach by pointing out that while it is true that this would enable additional optimization, there was also non-trivial overhead from creating a new matrix in memory from vectors stored in individual pandas cells (i.e. one vector per cell..). He didn't bite. At the outset I was told that the interviews would measure how well the candidate collaborated with the interviewers, which is typical, and this seemed to be especially important for "pair programming". However, the interviewer seemed to want to throw weird roadblocks into an otherwise simple exercise instead of engaging. I created the matrix and moved on to the next part.
The system design round was more in neutral territory. The interviewer gave an abstracted version of a problem he actually faced, but it would have been better if he simply stated the real world problem itself. In addition, the solution involved data augmentation using an LLM (i.e. generating synthetic data with an LLM to train another model). It sounded like this worked well in this particular instance, but it doesn't work well in the general case, since otherwise the Singularity is nigh and LLMs can generate data to further train themselves. Whether this approach yields useful results for a particular problem typically has to be tested explicitly and is not something one should assume during a brief conversational (no code) interview.
