Chat with recruiter, chat with hiring manager, take-home assignment. "We had stronger coding submissions" and no feedback provided. I spent about 4 hours on the assignment using a lot of boilerplate code from my cookiecutter templates. I have some serious doubts that the person that reviewed it understood what they were looking at. I implemented a fastAPI app that allowed ingesting json files into a SQLite DB with the sqlite-vec extension. Supporting full CRUD operations. Ready to be instrumented with OTEL, with alembic to manage migrations, an LLM completions protocol and an embeddings protocol with mock and concrement implementations to be able to mock all interactions with a query param for dev / testing, all based on LiteLLM for a single interface to swap models, automated retries etc. Exception handlers to obfuscate internal errors, logging middleware. CI/CD mock for unit tests, image builds, deployment to k8s (with helm) and automation of notifications and smoke tests. Yes the RAG implementation was very basic but the focus was on MLOps and production focused engineering which is where my strengths lie and I thought we were in alignment with the hiring manager from the initial conversation.

Initial 45m talk with a senior engineer followed by a take home exercise. I would also like to echo a lot of the negative experience comments. The task per se was quite easy: an LLM-based conversational agent over some financial data. As instructed, it should take a few hours, solution should not be complicated, and also they stated that not all ideas need to be implemented. After all, they instructed to spend 25% of time on writing a report that included solution description, 'Future work' and Error analysis. Putting everything together to have a well structured code, run small scale eval to analyse results, do prompt improvements, re-run eval, and conclude with detailed report easily adds up to a few hours. However, the feedback seemed to imply that far more were expected than either necessary for this problem (LLMs+tools or Agentic style solution), or requested in the description or could be finished in a few hours. The feedback also implied using CoT which was again unnecessary for the complexity of the problem and which I also described in detail in the Future work section, but apparently it was ignored. Other negative aspects: - It is the first time I am asked to use LLMs for home exercise and they don't provide with a temp api_key. - In their orig. instructions email, they mentioned it would be possible that I send them one of my projects as opposed to doing the exercise. However, when I offered to opensource a project of mine that included RAG, LLMs, prompting, they declined. In retrospect it seems clear they want you to work on their problem.