We know that P-values (within t-test context as an example..) is highly sensitive to sample size. A larger sample will yield a smaller p-value remaining everything else constant. On the other hand, Cohen´s d effect size remains the same.

Sample size and P values

I'm inspired in this code here, but I´ve changed some parts to make the difference between means constant, instead of creating a random variable based on a normal distribution.

Although everything is working, I do imagine that some of the experts in this community could improve my syntax.


ctrl_mean <- 8
ctrl_sd <- 1

treated_mean <- 7.9
treated_sd <- 1.2

sample <- numeric() #criar vetor para grupar resultados
nsim <- 1000 #criar variavel
t_result <- numeric()

for (i in 1:nsim) { 
  t_result[i] <- (mean(ctrl_mean)-mean(treated_mean))/sqrt((ctrl_sd^2/(i))+(treated_sd^2/(i))) #manual t test
  sample[i] <- i # number of participants
ds <- data.frame(
  sample = sample, #assign the sample size
  t_result = round(t_result,3), #get the t test result
  degrees = sample*2-2) #compute the degrees of freedom

ds %>% 
  filter(sample>1) %>% 
  mutate(P_Value = 2*pt(abs(t_result), df=degrees,lower.tail=FALSE)) %>% 
  left_join(ds,.) -> ds

ggplot(ds, aes(x=sample, y=P_Value)) +
  geom_line() +
  annotate("segment", x = 1, xend=sample, y = 0.05, yend = 0.05, colour = "purple", linetype = "dashed") +
  annotate("segment", x = 1, xend=sample, y = 0.01, yend = 0.01, colour = "red", linetype = "dashed") +
  annotate("text", x = c(1,1), y=c(.035,.001), label = c("p < 0.05", "p < 0.01"))

In this code you do not need the loop:

sample <- 1:nsim
t_result <- (mean(ctrl_mean)-mean(treated_mean)) /
# OR:
t_result <- (mean(ctrl_mean) - mean(treated_mean)) /
  sqrt((ctrl_sd^2 + treated_sd^2) / sample)

why the set seed?

  • \$\begingroup\$ Thank you. Easier than I was imagining. Set.seed, in this case, was an error, thanks for highlighting that. \$\endgroup\$ – Luis Mar 23 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.