Sounds like you are intending to build solvers. You will be better off in MATLAB. As you noted, matrix and vector operations are part of the language. You'll need matrix decompositions and maybe eigenvalue solvers to build your solvers, probably for dense and sparse matrices, and MATLAB has all of those. If you find that there's a library you need in some other language, you can call it from MATLAB.
There are lots of solvers written in MATLAB, for example, SDPT3 and SeDuMi, the solvers inside of CVX. Besides github, look on the MATLAB File Exchange to see more.
Full disclosure: I work at MathWorks. But was using MATLAB a long time before that!