For an optimization problem, there are multiple-type variables should be optimized. Can we use the convex optimization method to solve a subproblem of partial variables, and then, with the obtained results of the subproblem, solve the remaining subproblem of other variables by reinforcement learning?
Not really, but approximately. By OR standards, a problem is "solved" once we manage to satisfy the KKT conditions. There is no machine learning algorithm to date that can consistently satisfy constraints. ML is designed to give pretty good approximations, and that's about it.
For instance, image recognition can be posed as a convex optimisation problem, and we all know how well ML works on those problems. That doesn't mean however that it will always work (and it doesn't), unlike an optimisation algorithm.
You can use RL in any step. But problem is optimality check of solution which is explained above. Also you can solve your problem directly using RL such as RL for VRP. And you can read this blog which is about RL usage. By the way your question is too broad to give detailed answer.