6
$\begingroup$

I am doing some output analysis on a discrete event simulation model and trying to interpret the results. My first step was to run the baseline model to get the response variable, let's say it's 10 with a 95% confidence interval(CI) of 8-12.

I then run three more simulations, adjusting, one at a time, input Factor A, B, and C. This gives me the mean response and CI, showing the response output is significantly different:

Baseline    Factor A     Factor B      Factor C
10(8-12)   14(13-15)    20 (16-24)    15(13-18)

I then run a full factorial DOE, setting my factors to low (baseline value), or high (same input values as before). The DOE has no significant P-values when I run an ANOVA on the 8 different runs. Does this mean my simulation is incorrect? Or is this the difference of testing the simulation against a control (baseline), versus the impact of each factor as other factors change?

Improve this question
$\endgroup$
  • 1
    $\begingroup$ Seems like a $2^3$ factorial without replication. What anova did you run? one-way, that is, without interactions? This designs (without replications) is not really meant for hypothesis testing, they have too few degrees of freedom. Can you show us all of the effect estimates, including all the interactions? That comes from a saturated model, so no df for a variance estimate. $\endgroup$ – kjetil b halvorsen Oct 25 '19 at 19:21
  • $\begingroup$ I ran 5 reps of each 8 runs, and did a two way, with interaction. Shouldn't my individual factors come through as significant as they did when I compared them to baseline? $\endgroup$ – Yolo_chicken Oct 27 '19 at 12:53
  • $\begingroup$ Baseline is all factors at low value? Sounds strange, but it is difficult to say much without more details. What are the factors? Continuous or discrete? Is the noise level (variance) constant? Maybe the variance is higher when factors is at high level? Maybe present some plots? Generally it is better to visualize data before doing anovas and such. Can you post (a link to) the data? $\endgroup$ – kjetil b halvorsen Nov 2 '19 at 21:34
  • $\begingroup$ One-at-a-time variation is incapable of telling you about interactions, while a good DOE will help identify and estimate them. In my decades of experience, virtually every simulation response surface I've seen has had significant interactions. However, you need to explicitly add those interaction terms to parametric models such regressions. $\endgroup$ – pjs Apr 26 '20 at 4:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.