Statistical significance and effect size tests in SEM: Common method bias and strong theorizing

The article below provides evidence in support of a few very important methodological propositions: (a) we should not do away with classic statistical significance tests, but should combine them with effect size tests, and tests of common method bias; (b) high quality theorizing is very important if we are to profitably use a combination of classic statistical significance, effect size, and common method bias tests; and (c) the full collinearity VIF threshold in common method bias assessment for factor-based PLSF-SEM should be 10, as opposed to the 3.3 number used with classic composite-based PLS algorithms.

Kock, N., & Dow, K. E. (2025). Statistical significance and effect size tests in SEM: Common method bias and strong theorizing. Advances in Management Accounting, 37(1), 95–105.

Link to full-text file for this article:

Statistical significance and effect size tests in SEM: Common method bias and strong theorizing.

Abstract:

We generally acknowledge the problematic nature of classic statistical significance tests based on P-values or confidence intervals. In fact, we demonstrate based on an illustrative model for which we created simulated data, that with low and high statistical power, path coefficients in structural equation modeling whose true values are zero, routinely end up being found to be significantly different from zero at the P < .05 level. However, we argue that we should not do away with classic statistical significance tests, and that these tests can be useful but should be complemented by other methodological tools, including effect size tests, and tests of common method bias. We also argue that high quality theorizing is very important if we are to profitably use a combination of classic statistical significance, effect size, and common method bias tests.

Important note for PLSF-SEM users (repeated below for emphasis):

The full collinearity VIF threshold in common method bias assessment for factor-based PLSF-SEM should be 10, as opposed to the 3.3 number used with classic composite-based PLS algorithms.

Best regards to all!

Do employees care about diversity and inclusion? Why academic research should not be politically biased

The forthcoming (pre-published online) article below suggests, among other things, that the degree to which a company promotes diversity and inclusion has a negligible effect on how an employee rates the company (which reflects job satisfaction). This is in fact not the main theme of the article, but it is something that received plenty of pushback from reviewers. The analysis was conducted a while ago, when research results that did not strongly support diversity and inclusion were typically viewed rather unfavorably by review panels in many academic journals. We thank the prestigious journal Personnel Review for their academic integrity.

Kock, N., Haddoud, M. Y., Onjewu, A. K., & Yang, S. (forthcoming). Unveiling workplace dynamics: Insights from voluntary disclosures on business outlook and CEO approval. Personnel Review.

Links to full-text versions of the article:

https://www.emerald.com/insight/content/doi/10.1108/pr-03-2024-0251/full/html

https://pure-oai.bham.ac.uk/ws/portalfiles/portal/253297641/KockN2025Unveiling_AAM.pdf

Abstract:

Purpose: This inquiry extends the discourse on job satisfaction and employee referral. It aims to examine the moderating effects of perceived business outlook and CEO approval in the dynamics of job satisfaction and employee referral. A model predicting job satisfaction and employee referral through the lens of Herzberg’s two-factor theory is developed and tested. Design/methodology/approach: To remedy the overreliance on self-reported surveys, impeding generalization and representativeness, this study uses large evidence from 14,840 voluntary disclosures of US employees. A structural equation modeling technique is adopted to test the hypotheses. Findings: The inherent robust path analysis revealed intriguing findings highlighting culture and values as exerting the most substantial positive impact on job satisfaction, while diversity and inclusion played a relatively trivial role. Moreover, employees’ view of the firms’ outlook and their approval of the incumbent CEO were found to strengthen the job satisfaction–referral nexus. Originality/value: The study revisits the relationship between job satisfaction and employee referral by capturing the moderating effects of perceived business outlook and CEO approval. We believe that this investigation is one of the first to capture the impact of these two pivotal factors.

The figure below summarizes the results of the study. The overall rating variable reflects satisfaction with one’s job at a particular company, which predictably influences the probability that a person will recommend the company to a friend as a potential employer. If we had relied only on statistical significance tests, the effect of diversity and inclusion on job satisfaction would actually be negative and statistically significant. But based on the small effect size, we felt that it would be more scholarly to report the effect in question as indistinguishable from zero. With large samples, the likelihood of type I errors (false positives) increases dramatically in statistical significance tests, whether P values or confidence intervals are used.

Shiyu got us the awesome Glassdoor dataset, while Yacine and Adah-Kole did most of the theory development and later discussion work (thank you, my talented co-authors). The curious thing is that I did the analyses for this article, using WarpPLS and double-checking with other analysis tools, and was not only surprised but rather displeased with the results. But why was I displeased with the results? Well, as an academic, I work in a very diverse environment, and find that diversity stimulating. In particular, I am very interested about countries and regions (domestically and abroad), their cultures, and histories. Furthermore, as someone with a diverse background, I have lived in Brazil and New Zealand, before settling in the US. While in the US, Belgium was like a second home for several years, as I travelled there often to consult for the European Commission.

Yet, regardless of personal background, and for the sake of societal credibility, academics must report research results as they are, to the best of their ability. Furthermore, they must report research results independently from political orientation and how they personally feel about those results. Finally, they have to resign themselves to the fact that all empirical studies provide incomplete views of the world, and usually call for more research using different approaches and epistemologies.

Best regards to all!

PS: I thank Nadya Larumbe for her comments on a previous version of this post.

Will PLS have to become factor-based to survive and thrive?

The article below provides an overview of various SEM approaches. It argues that minimization of type I and II errors, or false positives and negatives respectively in hypothesis testing, can only happen if latent variables are implemented as factors (and not as composites). It is argued that this requires the use of modern, factor-based PLS methods (known as PLSF methods), which have some advantages not only over classic PLS implementations, but also over covariance-based SEM approaches. We discussed a PLSF type in the article; namely type CFM3.

Kock, N. (2024). Will PLS have to become factor-based to survive and thrive? European Journal of Information Systems, 33(6), 882-902.

Link to full-text file for this article:

Will PLS have to become factor-based to survive and thrive?

Abstract:

Structural equation modelling (SEM) is a general method that aims at estimating models with latent variables (LVs), where the LVs are measured indirectly and with some imprecision via questionnaires. This is done usually employing question-statements answered on Likert-type scales. In this paper we discuss various forms of SEM, and demonstrate that composite-based models, common in classic partial least squares (PLS) implementations, are poorly aligned with the very idea of SEM. We argue that minimisation of type I and II errors, or false positives and negatives respectively in hypothesis testing, can only happen if LVs are implemented as factors (and not as composites). This requires the use of modern, factor-based PLS methods, which have some advantages not only over classic PLS implementations, but also over covariance-based SEM approaches. Our main goal with this paper is to stimulate debate, whether pro or against our views. If we are generally correct in our thinking, the impact on how quantitative research is conducted in the field of information systems, as well as many other fields, could be quite dramatic. The reason for this is the widespread use of SEM in information systems, business, and the behavioural sciences.

Note: Some readers of this blog have brought to our attention that a critique of the article above is already out, and with a number of mistakes and incorrect statements, such as that: they (the critics) used CFM1 because this is the only PLSF type documented in the WarpPLS User Manual (untrue and very easy to check); and that the algorithm that they analyzed (PLSF-CFM1) is a slow version of Dijkstra’s PLSc technique (CFM1 does not use PLSc at all); among other easy-to-avoid mistakes and incorrect statements. We are aware of this critique.

Best regards to all!

Conducting a difference-in-differences analysis with PLS-SEM: The classic 2x2 approach

The article below shows how one can conduct a difference-in-differences analysis employing the classic 2x2 approach for this type of analysis, using structural equation modeling via partial least squares (PLS-SEM).

Kock, N. (2024). Conducting a difference-in-differences analysis with PLS-SEM: The classic 2x2 approach. Data Analysis Perspectives Journal, 5(5), 1-8.

Link to full-text file for this and other DAPJ articles:

https://scriptwarp.com/dapj/#Published_Articles

Abstract:

Difference-in-differences analyses often employ a classic 2x2 scenario, which involves two conditions, control and treatment; and two points in time, before and after an intervention that may be tied to one of the conditions. In our analysis, we assess the impact on labor productivity of being in a more technology-intensive US state, instead of a more manufacturing-intensive one. Consistently with the difference-in-differences analysis scenario, we also assess the full latent growth effect of a government-driven age discrimination crackdown, in the technology-intensive state, on the previous effect – of being in a technology-intensive state on labor productivity. We do this by employing a model analyzed in the context of structural equation modeling via partial least squares (PLS-SEM). We also discuss advantages of using PLS-SEM in this scenario; which include assessments of causality, common method bias, and endogeneity.

Best regards to all!

Combining composites and factors in PLS-SEM models: A multi-algorithm technique

The article below presents a multi-algorithm technique for combining latent variables estimated as composites or factors into a single model, in the context of structural equation modeling via partial least squares (PLS-SEM).

Kock, N. (2024). Combining composites and factors in PLS-SEM models: A multi-algorithm technique. Data Analysis Perspectives Journal, 5(4), 1-8.

Link to full-text file for this and other DAPJ articles:

https://scriptwarp.com/dapj/#Published_Articles

Abstract:

A multi-algorithm technique is presented for combining latent variables estimated as composites or factors into a single model, in the context of structural equation modeling via partial least squares. The multi-algorithm technique consists of three key steps: selecting composite-based or factor-based outer model analysis algorithms to be used for latent variable estimation; adding the latent variables estimated with the chosen composite-based or factor-based algorithms as new standardized variables; and creating and estimating a final model with the new variables added as single indicators of latent variables.

Best regards to all!

A comparison of data analyses with WarpPLS and Stata: A study of trust and its role regarding internet use and subjective well-being

The article below provides a comparative assessment of analyses using the software packages WarpPLS and Stata, in the context of structural equation modeling via partial least squares (PLS-SEM), based on an illustrative study of trust and its role regarding internet use and subjective well-being.

Samak, A., Islam, M. R., & Hanke, D. (2024). A comparison of data analyses with WarpPLS and Stata: A study of trust and its role regarding internet use and subjective well-being. Data Analysis Perspectives Journal, 5(3), 1-6.

Link to full-text file for this and other DAPJ articles:

https://scriptwarp.com/dapj/#Published_Articles

Abstract:

This study investigates the mediating roles of social and institutional trust in the relationship between internet use and subjective well-being, using partial least squares (PLS)-based structural equation modeling (SEM). We compare WarpPLS 8.0 and Stata’s PLS-SEM package, utilizing data from the European Social Survey (ESS), round 8. Our results show consistent model fit and path coefficients across both tools, confirming the significant mediating effects of trust. WarpPLS stands out for its advanced model diagnostics, while Stata’s PLS-SEM excels in integrating with Stata’s broader data management and statistical analysis tools. This comparative analysis contributes to the SEM methodological literature.

Best regards to all!

A comparison of multiple regression analyses in Stata and WarpPLS

The article below provides a comparative assessment of multiple regression analyses using the software packages WarpPLS and Stata.

Tarkom, A., & Gopal, P. (2024). A comparison of multiple regression analyses in Stata and WarpPLS. Data Analysis Perspectives Journal, 5(2), 1-8.

Link to full-text file for this and other DAPJ articles:

https://scriptwarp.com/dapj/#Published_Articles

Abstract:

This paper illustrates a comparative analysis of multiple regression analysis using two different software. The software packages used are WarpPLS 8.0 and Stata 17. Multiple regression analyses performed with both software produce the same results. WarpPLS 8.0 has the added advantage over Stata owing to its graphic user interface that aids in model specification and visualization. Furthermore, it provides users with additional tools to visualize moderating effects. Both software have equal accuracy in terms of the results but differences in terms of what they offer users.

Best regards to all!