MATLAB is a language built on top of a library.
Python (with NumPy & numba) is a language with a library built under it.
Neither is ideal. Like all languages, both have a few quirks, due to their history.
My suggestion: Door Number 3, Julia.
In either case (MATLAB, Python, Julia), you should ask yourself:
- Is your immediate goal to master the math, or to master how to program it?
- Is your long-term career goal academia, or industry?
- Is it important to you that other people can run your code (e.g. github repo)?
- Is your code all short, or might it grow into a large (many 1000-line, 100's of functions) code?
If your answers are: 1: math, 2: academia, 3: no, 4: all short, then MATLAB is fine. It is an excellent tool. I love it for doing something fast and not fussing with things; it's an industry standard in DSP and radar and other problems that rely very heavily on linear algebra. There is a lot of code written in it; it will be around for a long time to come.
Nevertheless, if those are not your answers, then between the two, I definitely suggest Python over MATLAB. You will become a much better programmer, and your job prospects will be much better, too. If your code gets long, beware that managing a large (many 1000-lines) MATLAB code is a nightmare IMO (namespaces, anyone?). If you find you have to use MATLAB at some later date, it will be easy. Conversely, if you learn MATLAB and later on find you need to learn Python + NumPy + numba, then you will probably find this very difficult.
This is all the more true if we are talking about optimization, which is at least tangential to machine learning (ML). MATLAB is trying to compete in this space, but if you look online for job ads, they are almost all asking for Python, not MATLAB. In fact, in ML, you will find (in my experience) more job ads asking for Julia than for MATLAB.
But again (and yes it is just my personal opinion), I would suggest you take a look at Julia. I have not been as excited about a language after trying Julia since.... well, since I started learning Python 18 years ago. I use all three at work, and Julia is my first choice most of the time; I think the language is truly going places. But, I'm a risk-taker. It's not the safe move. If you want safe, the safe move is Python.
(Disclosure: I have no interests, financial or otherwise, in MATLAB, Python, or Julia, other than my own experience using them for work/research.)