Each sim could have 100k rows with 5 columns. From each model fit I would save a row with 10 model stats. Simulation volume varies but typically 4000 to 10000. Python and R are the languages.

There are scenarios I would tackle that involved fitting models to orders of magnitude more datasets, but I'm running into the limits of my mental model on how simulation studies effectively scale, and that prompted the initial post.

I simulate 10s of thousands of datasets using likelihood functions from numpy or base R, and then I fit competing models to each dataset and save key model stats. I use the sampling distribution of model stats to compare the business impact of adopting different models for decision making.

@phdemetri.bsky.social great post! You set a parameter for true_probability for A & B in simulating future samples. So you're computing the Pr(change your mind | a 3% real effect). Why not base the future simulations and Pr(change your mind) upon a prior distribution rather than a parameter?

yes

Thank you both, I will check these out!

@jordannafa.bsky.social or @solomonkurz.bsky.social can you provide any leads?

# create a .Rprofile in your project root directory & add this:
.First <- function() {
r_files <- list.files("R", pattern = "\\.R$", full.names = TRUE)

# Source each file
for (file in r_files) {
source(file)
}

cat("All scripts in", "R", "have been executed.\n")
}

I’m curious if anything came out of this line of inquiry

Both books have this theme about the futility of non-strategic action. Perhaps cementing that belief is step one.

I tried to pursue this question during my PhD. It’s a tough nut for a variety of reasons, but I really enjoyed Richard Rumelt’s Good Strategy Bad Strategy and Ericcson’s Peak. Strategy is hard work and most people avoid it. Expertise is pattern recognition and that takes a lot of time and feedback.

Have others seen stats that try to express type M and type D errors simultaneously?

Defining practical accuracy as % of posterior that is directionally accurate but less than a 50% overestimate was pure shooting from the hip. For small effects e.g., 0.25% lift, this would be rather narrow (0>theta>0.375%) and for large effects e.g., 6% lift, the range would be broader (0>theta>9%).

Appreciate everyone’s feedback!