What are the mainstream models for portfolio optimization? We have Markowitz mean-variance model and CVaR-based models (e.g., max return subject to a CVaR constraint). What else is out there in terms of risk measures or formulations?
Here's what is not really mainstream now, but should be. The mean and especially the covariance matrix of returns is not known. Treating estimates of then as though they are known with certainty can lead to very suboptimal results.
Just to start vectoring yourself in the right direction, you can start by looking at
MEAN–VARIANCE PORTFOLIO OPTIMIZATION WHEN MEANS AND COVARIANCES ARE UNKNOWN, TZE LEUNG LAI, HAIPENG XING, and ZEHAO CHEN, Annals of Statistics, 2011, Vol. 5, No. 2A, 798–823.
Improving Portfolios Global Performance with Robust Covariance Matrix Estimation:Application to the Maximum Variety Portfolio, Emmanuelle Jay, Eugenie Terreaux, Jean-Philippe Ovarlez, and Frederic Pascal.
You may also find of interest methods to identify financial risk factors using large data sets.
Identifying Financial Risk Factors with a Low-Rank Sparse Decomposition, Lisa Goldberg and Alex Shkolnik. This decomposes covariance as a sum of a rank-one factor component and a diagonal security specific return component
Here is a semi-classic paper advising you NOT to use the sample covariance matrix for portfolio optimization. "Shrinking" it toward a better conditioned matrix. even though producing a biased estimator of the covariance matrix, can improve the results of portfolio optimization (note that the condition number of the sample covariance matrix is a very biased estimator of the condition number of the true covariance matrix, and is infinite when the number of vector data points is less than the number of variables).
Honey, I Shrunk the Sample Covariance Matrix, Olivier Ledoit and MichaelWolf, The Journal of Portfolio Management Summer 2004, 30 (4) 110-119 (link is to free version of the paper)
Another aspect of portfolio optimization which is often important in practise is sparsity, i.e., requiring that the portfolio invests in at most $k$ non-zero stocks to cut down on transaction fees and avoid functioning as an index fund. This can be achieved by imposing a sparsity constraint $\Vert x \Vert_0 \leq k.$
The classical reference on this topic is this 1996 paper by Bienstock.
I'm no expert on the topic, but I found the textbook Optimization Methonds in Finance really accessible. The authors teach optimization modeling and solving motivated by applications in finance, including portfolio optimization.
In particular they describe a linear model using mean absolute deviation as a linear proxy for variance.
For the sake of having a simple example, we decided to go with expected return (1 year), standard deviation risk and asset correlation. The example xlsx data came from Yahoo Finance IIRC.