RL (and Machine Learning in general) often use specialized forms of continuous optimization problems that enable one to use elegant solving techniques (e.g. training/learning on the GPU or even special hardware (TPU and friends)). See here for a nice overview of the underlying optimization problems in Machine Learning and Deep Learning.
As @albert-schroteboer already wrote in the comments, there are some activities to bring ML/DL/RL and combinatorial problems together. There have been some efforts with graph neural nets to tackle some problems traditionally solved via combinatorial algorithms. See here for an overview and some more papers here (TSP, Knapsack), here (VRP variants), here (MaxCut, TSP) or here (Quadratic Assignment).
Note that ML/DL/RL algorithms usually do not give a guarantee for the solution quality whereas traditional combinatorial algorithms might have this property (sometimes it is inherently in the algorithms, sometimes extra effort is taken to prove optimality). For large problem instances when the traditional algorithms take a long time to produce a solution or cases where a "good" (but not necessary optimal) solution is sufficient, these new approaches might be a good addition to the other algorithms.