I can see that you have messed up with the question and did not focus on what the interviewer wanted to know more. Maybe he asked you this question, to know how you are going to use your money. So, answer these questions like "Money is important for everyone and you should not reject its importance. But you should not make it your first priority. As you will come to learn in your career, opportunities for growth and advancement, in the long run, are far more important for your career development.
So, here neither you are rejecting the value of money nor neglecting your work.