So here is my experience (and also some wisdom of other ppl i read) with setting lower bounds for minimization problems: Depending on the complexity of the model or your strategy for cutting invalid solutions of (tightness of the cuts) it might indeed be worse to set a lower bound, because it messes with the internal pricing gurobi uses. All the nodes that are still to be visited, get truncated to the same obj-value and there is no way to prioritize the more promising nodes, which mostly leads to vising random nodes, which then in turn also branch with the same obj-value and before you can derive any reasonable cuts you end up with a much bigger tree.
Now back to your issue when setting a good starting solution (or an upper bound in general): It is possible that setting such good starting solution (or even an optimal one) cuts off branches of the tree, which would otherwise be examinated and might produce better cuts or direction for moving that upper bound further up or something of this sorts. But in contrary to setting lower bounds, which seems to generally perform worse, this is more of an accident. In general it should be a good idea to set better initial solutions - it might just be worse in your specific case (problem instance)