I was watching the following video (https://www.youtube.com/watch?v=-GWze-wtu60 @ 9:30) and came across the following graph:
This graph is meant to show the number of iterations required to achieve similar results on the same function for different optimization algorithms. As we can see in this graph, it appears that "Gradient Free Algorithms" (e.g. Genetic Algorithm) seems to require far more iterations than "Gradient Based Algorithms" - especially as the number of variables increase.
I have always heard the opposite remark being made. Especially in higher dimension problems (i.e. where there are more variables), evaluating the derivatives of complicated loss functions (e.g. the loss functions in machine learning models, neural networks) can become very expensive. This fact is supposed to be one of the main advantages of Gradient Free Algorithms - Gradient Free Algorithms make a sacrifice in terms of their ability to converge, but instead do not require as much computational costs when compared to Gradient Based Algorithms. However, the above graph seems to be negating this fact.
In general, are Gradient Free Algorithms really said to scale as poorly to higher dimensional problems as indicated in the above graph?