I am doing some output analysis on a discrete event simulation model and trying to interpret the results. My first step was to run the baseline model to get the response variable, let's say it's 10 with a 95% confidence interval(CI) of 8-12.
I then run three more simulations, adjusting, one at a time, input Factor A, B, and C. This gives me the mean response and CI, showing the response output is significantly different:
Baseline Factor A Factor B Factor C
10(8-12) 14(13-15) 20 (16-24) 15(13-18)
I then run a full factorial DOE, setting my factors to low (baseline value), or high (same input values as before). The DOE has no significant P-values when I run an ANOVA on the 8 different runs. Does this mean my simulation is incorrect? Or is this the difference of testing the simulation against a control (baseline), versus the impact of each factor as other factors change?