Statistical significance and effect size tests in SEM: Common method bias and strong theorizing

The article below provides evidence in support of a few very important methodological propositions: (a) we should not do away with classic statistical significance tests, but should combine them with effect size tests, and tests of common method bias; (b) high quality theorizing is very important if we are to profitably use a combination of classic statistical significance, effect size, and common method bias tests; and (c) the full collinearity VIF threshold in common method bias assessment for factor-based PLSF-SEM should be 10, as opposed to the 3.3 number used with classic composite-based PLS algorithms.

Kock, N., & Dow, K. E. (2025). Statistical significance and effect size tests in SEM: Common method bias and strong theorizing. Advances in Management Accounting, 37(1), 95–105.

Link to full-text file for this article:

Statistical significance and effect size tests in SEM: Common method bias and strong theorizing.

Abstract:

We generally acknowledge the problematic nature of classic statistical significance tests based on P-values or confidence intervals. In fact, we demonstrate based on an illustrative model for which we created simulated data, that with low and high statistical power, path coefficients in structural equation modeling whose true values are zero, routinely end up being found to be significantly different from zero at the P < .05 level. However, we argue that we should not do away with classic statistical significance tests, and that these tests can be useful but should be complemented by other methodological tools, including effect size tests, and tests of common method bias. We also argue that high quality theorizing is very important if we are to profitably use a combination of classic statistical significance, effect size, and common method bias tests.

Important note for PLSF-SEM users (repeated below for emphasis):

The full collinearity VIF threshold in common method bias assessment for factor-based PLSF-SEM should be 10, as opposed to the 3.3 number used with classic composite-based PLS algorithms.

Best regards to all!

Do employees care about diversity and inclusion? Why academic research should not be politically biased

The article below provides a good example of a robust path analysis employing WarpPLS. It suggests, among other things, that the degree to which a company promotes diversity and inclusion has a negligible effect on how an employee rates the company (which reflects job satisfaction). This is in fact not the main theme of the article, but it is something that received plenty of pushback from reviewers. The analysis was conducted a while ago, when research results that did not strongly support diversity and inclusion were typically viewed rather unfavorably by review panels in many academic journals. We thank the prestigious journal Personnel Review for their academic integrity.

Kock, N., Haddoud, M.Y., Onjewu, A.-K., & Yang, S. (2025). Unveiling workplace dynamics: Insights from voluntary disclosures on business outlook and CEO approval. Personnel Review, 54(2), 474–497.

Links to full-text versions of the article:

https://scriptwarp.com/pubs/Kock_etal_2025_PR_WorkplaceDynamics.pdf

https://pure-oai.bham.ac.uk/ws/portalfiles/portal/253297641/KockN2025Unveiling_AAM.pdf

https://www.emerald.com/insight/content/doi/10.1108/pr-03-2024-0251/full/html

Abstract:

Purpose: This inquiry extends the discourse on job satisfaction and employee referral. It aims to examine the moderating effects of perceived business outlook and CEO approval in the dynamics of job satisfaction and employee referral. A model predicting job satisfaction and employee referral through the lens of Herzberg’s two-factor theory is developed and tested. Design/methodology/approach: To remedy the overreliance on self-reported surveys, impeding generalization and representativeness, this study uses large evidence from 14,840 voluntary disclosures of US employees. A structural equation modeling technique is adopted to test the hypotheses. Findings: The inherent robust path analysis revealed intriguing findings highlighting culture and values as exerting the most substantial positive impact on job satisfaction, while diversity and inclusion played a relatively trivial role. Moreover, employees’ view of the firms’ outlook and their approval of the incumbent CEO were found to strengthen the job satisfaction–referral nexus. Originality/value: The study revisits the relationship between job satisfaction and employee referral by capturing the moderating effects of perceived business outlook and CEO approval. We believe that this investigation is one of the first to capture the impact of these two pivotal factors.

The figure below summarizes the results of the study. The overall rating variable reflects satisfaction with one’s job at a particular company, which predictably influences the probability that a person will recommend the company to a friend as a potential employer. If we had relied only on statistical significance tests, the effect of diversity and inclusion on job satisfaction would actually be negative and statistically significant. But based on the small effect size, we felt that it would be more scholarly to report the effect in question as indistinguishable from zero. With large samples, the likelihood of type I errors (false positives) increases dramatically in statistical significance tests, whether P values or confidence intervals are used.

Shiyu got us the awesome Glassdoor dataset, while Yacine and Adah-Kole did most of the theory development and later discussion work (thank you, my talented co-authors). The curious thing is that I did the analyses for this article, using WarpPLS and double-checking with other analysis tools, and was not only surprised but rather displeased with the results. But why was I displeased with the results? Well, as an academic, I work in a very diverse environment, and find that diversity stimulating. In particular, I am very interested about countries and regions (domestically and abroad), their cultures, and histories. Furthermore, as someone with a diverse background, I have lived in Brazil and New Zealand, before settling in the US. While in the US, Belgium was like a second home for several years, as I travelled there often to consult for the European Commission.

Yet, regardless of personal background, and for the sake of societal credibility, academics must report research results as they are, to the best of their ability. Furthermore, they must report research results independently from political orientation and how they personally feel about those results. Finally, they have to resign themselves to the fact that all empirical studies provide incomplete views of the world, and usually call for more research using different approaches and epistemologies.

Best regards to all!

PS: I thank Nadya Larumbe for her comments on a previous version of this post.

Minimum sample size in PLS-SEM, regression, and path analyses

Based on Monte Carlo simulations, the minimum sample size in PLS-SEM can be reliably and conservatively estimated based on the inequality below:

N > ( 2.48 / Abs(bm) ) ^ 2

Extensive tests suggest that this also applies to multiple regression, and path analyses. In the latter, only single-indicator variables are included in the model, even though it looks a lot like an SEM model.

The inequality above is discussed in the article titled: "Minimum sample size estimation in PLS‐SEM: The inverse square root and gamma‐exponential methods" (). It refers to the inverse square root method. The gamma‐exponential method, also discussed in the article, is a refinement of the inverse square root method that relies on equations that are much more complex.

In the inequality above, N is the required sample size, and Abs(bm) is the absolute value of the path coefficient with the minimum expected magnitude in the model. This inequality assumes that:

- One-tailed P values are used for hypothesis testing. A previous post discusses this issue in more detail ().

- The threshold for P values is .05. That is, P values should be equal to or lower than .05.

- Effect sizes (ESs), as calculated by WarpPLS, are also used for hypothesis testing ().

- The threshold for ESs is .02. That is, ESs should be equal to or greater than .02.

- Acceptable statistical power is equal to or greater than .8.

- The latent variables in the model are not collinear, when both lateral and vertical collinearity are considered. That is, the full collinearity VIFs calculated by WarpPLS for all latent variables are equal to or lower than 3.3 ().

This inequality highlights the fact that path coefficient strength is a much stronger determinant of statistical power in Monte Carlo simulations than the configuration of the structural model.

The inequality is proposed as an alternative to the widely used (and discredited) "10 times rule". It yields minimum sample sizes that are consistent with Cohen's power tables for multiple regression.

For example, let us say one has a model where the path coefficient with the minimum expected magnitude is .3. Then the required sample size is:

N > ( 2.48 / .3 ) ^ 2 = 68.34

The minimum required sample size is thus:

Nm = 69

The above assumes a pre-analysis minimum sample size estimation, where the path coefficient with the minimum expected magnitude is set prior to the analysis.

A post-analysis minimum sample size estimation, on the other hand, would be based on the results of a full PLS-SEM analysis. Generally pre-analysis estimation is recommended over post-analysis estimation.

The latter, post-analysis estimation, can only confirm that an appropriate sample size was used.