One-tailed or two-tailed P values in PLS-SEM?

Should P values associated with path coefficients, as well as with other coefficients such as weights and loadings, be one-tailed or two-tailed? This question is addressed through the publication below.

Kock, N. (2015). One-tailed or two-tailed P values in PLS-SEM? International Journal of e-Collaboration, 11(2), 1-7.

PDF file:

http://cits.tamiu.edu/kock/pubs/journals/2015JournalIJeC2/Kock_2015_IJeC_OneTwoTailedPLSSEM.pdf

Abstract:

Should P values associated with path coefficients, as well as with other coefficients such as weights and loadings, be one-tailed or two-tailed? This question is answered in the context of structural equation modeling employing the partial least squares method (PLS-SEM), based on an illustrative model of the effect of e-collaboration technology use on job performance. A one-tailed test is recommended if the coefficient is assumed to have a sign (positive or negative), which should be reflected in the hypothesis that refers to the corresponding association. If no assumptions are made about coefficient sign, a two-tailed test is recommended. These recommendations apply to many other statistical methods that employ P values; including path analyses in general, with or without latent variables, plus univariate and multivariate regression analyses.

Hypothesis testing with confidence intervals and P values

While P values are widely used in PLS-based SEM, as well as in SEM in general, the statistical significances of path coefficients, weights and loadings can also be assessed employing T ratios and/or confidence intervals. These can be obtained in WarpPLS through the menu option “Explore T ratios and confidence intervals”, which also allows you to set the confidence level to be used. This menu option becomes available after Step 5 is completed.

Related YouTube video: Explore T Ratios and Confidence Intervals in WarpPLS

https://youtu.be/Xao0T2MxJZM

An article is also available explaining how WarpPLS users can test hypotheses based on confidence intervals, contrasting that approach with the one employing P values. A variation of the latter approach, employing T ratios, is also briefly discussed. Below are the reference, link to PDF file, and abstract for the article.

Kock, N. (2016). Hypothesis testing with confidence intervals and P values in PLS-SEM. International Journal of e-Collaboration, 12(3), 1-6.

PDF file

Abstract:
E-collaboration researchers usually employ P values for hypothesis testing, a common practice in a variety of other fields. This is also customary in many methodological contexts, such as analyses of path models with or without latent variables, as well as simpler tests that can be seen as special cases of these (e.g., comparisons of means). We discuss here how a researcher can use another major approach for hypothesis testing, the one building on confidence intervals. We contrast this approach with the one employing P values through the analysis of a simulated dataset, created based on a model grounded on past theory and empirical research. The model refers to social networking site use at work and its impact on job performance. The results of our analyses suggest that tests employing confidence intervals and P values are likely to lead to very similar outcomes in terms of acceptance or rejection of hypotheses.

Note 1:
On Table 1 in the article, each T ratio and confidence interval limits (lower and upper) are calculated through the formulas included below. Normally a hypothesis will not be supported if the confidence interval includes the number 0 (zero).

T ratio = (path coefficient) / (standard error).

Lower confidence interval = (path coefficient) - 1.96 * (standard error).

Upper confidence interval = (path coefficient) + 1.96 * (standard error).

Note 2:
Here is a quick note to technical readers. The P values reported in Table 1 in the article are calculated based on the T ratios using the incomplete beta function, which does not assume that the T distribution is exactly normal. In reality, T distributions have heavier tails than normal distributions, with the difference becoming less noticeable as sample sizes increase.

Multi-group analysis with WarpPLS: Comparing path coefficients for two or more group samples

Starting in version 6.0 of WarpPLS, the menu options “Explore multi-group analyses” and “Explore measurement invariance” allow you to conduct analyses where the data is segmented in various groups, all possible combinations of pairs of groups are generated, and each pair of groups is compared. In multi-group analyses normally path coefficients are compared, whereas in measurement invariance assessment the foci of comparison are loadings and/or weights. The grouping variables can be unstandardized indicators, standardized indicators, and labels. These types of analyzes can  also be conducted via the new menu option “Explore full latent growth”, which presents several advantages (as discussed in the WarpPLS User Manual).

Related YouTube videos:

Explore Multi-Group Analyses in WarpPLS

Explore Measurement Invariance in WarpPLS

The text below is for an older version of this post. It discusses other approaches to multi-group analysis that are only partially automated, and that are thus more time-consuming to implement.

***

I previously discussed on this post multi-group analysis with WarpPLS from the perspective of comparing means of two or more groups. This is also explored in an article titled: Using WarpPLS in e-collaboration studies: What if I have only one group and one condition?

A different type of multi-group analysis would be one in which the same model is analyzed for two or more different samples, where each sample refers to a data group.

For example, a researcher could test the same model with data from the USA and Mexico. In this case, two project files would be used, and the goal of the multi-group analysis would be to assess whether the path coefficients differ significantly across groups.

An approach to conduct this type of multi-group analysis, employing the pooled and Satterthwaite standard error methods, is discussed in a recent publication (Kock, 2014). This approach is implemented through an Excel spreadsheet available from the “Resources” area of the WarpPLS.com site, under “Excel files”.

The Excel spreadsheet above takes as inputs coefficients generated by WarpPLS, including path coefficients and their standard errors. The outputs are T and P values for each pair of coefficients being compared. The formulas used in it are discussed in a recent publication (Kock, 2014).

Reference

Kock, N. (2014). Advanced mediating effects tests, multi-group analyses, and measurement model assessments in PLS-based SEM. International Journal of e-Collaboration, 10(3), 1-13.

Multi-group analysis with WarpPLS: Comparing means of two or more groups

Comparing means of two groups is something that behavioral researchers often do. In fact, this is the most widely used quantitative analysis approach. This is also a form of multi-group analysis, where the number of groups is two. Common tests for comparing means are the t and and one-way ANOVA tests.

There is an arguably much better way of doing comparison of means test, with WarpPLS. Follow the steps below. This is a two-group test. Multi-group tests can be done in a similar way, through multiple two-group tests where conditions (i.e., groups) are compared pair by pair.

- Create a dummy variable (G) with numbers associated with each of the two groups - e.g., 0 for one group, and 1 for the other group. This dummy variable should be implemented as a LV with 1 indicator.

- Define your dependent (or criterion) construct (T) as you would normally do; in this case, I think that would be as a reflective LV with 3 indicators.

- Create a link between G and T, with G pointing at T.

- Estimate the model parameters with WarpPLS; this will calculate the beta and P values for the link. The P value is analogous to the P value you would obtain with a t test.

The following article goes into some detail about this procedure, contrasting it with other approaches:

Kock, N. (2013). Using WarpPLS in e-collaboration studies: What if I have only one group and one condition?  International Journal of e-Collaboration, 9(3), 1-12.

This type of WarpPLS test has a number of advantages over a standard t test or a one-way ANOVA test (which are essentially the same thing). For example, it allows for the use of LVs as dependent variables; and it is a robust test, which does not require that the dependent variables be normally distributed.

Multi-group tests, with more than two groups, can also be conducted by assigning different values to each of the groups. The key here is to decide what values to assign to each group. This choice is often somewhat arbitrary in exploratory analyses. The “Save grouped descriptive statistics into a tab-delimited .txt file” option may be helpful in this respect. This is a special option that allows you to save descriptive statistics (means and standard deviations) organized by groups defined based on certain parameters. For more details, see the user manual for WarpPLS, which is available from Warppls.com.

Starting in version 6.0 of WarpPLS, the menu options “Explore multi-group analyses” and “Explore measurement invariance” allow you to conduct analyses where the data is segmented in various groups, all possible combinations of pairs of groups are generated, and each pair of groups is compared. In multi-group analyses normally path coefficients are compared, whereas in measurement invariance assessment the foci of comparison are loadings and/or weights. The grouping variables can be unstandardized indicators, standardized indicators, and labels. These types of analyzes can now also be conducted via the new menu option “Explore full latent growth”, which presents several advantages (as discussed in the WarpPLS User Manual).

Related YouTube videos:

Explore Multi-Group Analyses in WarpPLS

Explore Measurement Invariance in WarpPLS

Field studies, small samples, and WarpPLS

Let us assume that a researcher wants to evaluate the effectiveness of new management method by conducting an intervention study in one single organization.

In this example, the researcher facilitates the use of a new management method by 20 managers in the organization, and then measures their degree of adoption of the method and their effectiveness.

The above is an example of a field study. Often field studies will yield small datasets, which will not conform to parametric analysis (e.g., ANOVA and ordinary multiple regression) pre-conditions. For example, the data will not typically be normally distributed.

WarpPLS can be very useful in the analysis of this type of data.

One reason is that, with small sample sizes, it may be difficult to identify linear relationships that are strong enough to be statistically significant (at P lower than 0.05, or less). Since WarpPLS implements nonlinear analysis algorithms, it can be very useful in the analysis of small samples.

Another reason is that P values are calculated through resampling, a nonparametric approach to statistical significance estimation. For small samples (i.e., lower than 100), jackknifing is the recommended resampling approach. Bootstrapping is recommended only for sample sizes greater than 100.

Bootstrapping or jackknifing (or both) in WarpPLS?

The blog post below refers to resampling methods, notably bootstrapping and jackknifing, which are often used for the generation of estimates employed and hypothesis testing. Even though they are widely used, resampling methods are inherently unstable. More recent versions of WarpPLS employ "stable" methods for the same purpose, with various advantages. See the most recent version of the WarpPLS User Manual (linked below) for more details.

http://www.scriptwarp.com/warppls/#User_Manual

***

Arguably jackknifing does a better job at addressing problems associated with the presence of outliers due to errors in data collection. Generally speaking, jackknifing tends to generate more stable resample path coefficients (and thus more reliable P values) with small sample sizes (lower than 100), and with samples containing outliers. In these cases, outlier data points do not appear more than once in the set of resamples, which accounts for the better performance of jackknifing (see, e.g., Chiquoine & Hjalmarsson, 2009).

Bootstrapping tends to generate more stable resample path coefficients (and thus more reliable P values) with larger samples and with samples where the data points are evenly distributed on a scatter plot. The use of bootstrapping with small sample sizes (lower than 100) has been discouraged (Nevitt & Hancock, 2001).

Since the warping algorithms are also sensitive to the presence of outliers, in many cases it is a good idea to estimate P values with both bootstrapping and jackknifing, and use the P values associated with the most stable coefficients. An indication of instability is a high P value (i.e., statistically insignificant) associated with path coefficients that could be reasonably expected to have low P values. For example, with a sample size of 100, a path coefficient of .2 could be reasonably expected to yield a P value that is statistically significant at the .05 level. If that is not the case, there may be a stability problem. Another indication of instability is a marked difference between the P values estimated through bootstrapping and jackknifing.

P values can be easily estimated using both resampling methods, bootstrapping and jackknifing, by following this simple procedure. Run an SEM analysis of the desired model, using one of the resampling methods, and save the project. Then save the project again, this time with a different name, change the resampling method, and run the SEM analysis again. Then save the second project again. Each project file will now have results that refer to one of the two resampling methods. The P values can then be compared, and the most stable ones used in a research report on the SEM analysis.

References:

Chiquoine, B., & Hjalmarsson, E. (2009). Jackknifing stock return predictions. Journal of Empirical Finance, 16(5), 793-803.

Nevitt, J., & Hancock, G.R. (2001). Performance of bootstrapping approaches to model test statistics and parameter standard error estimation in structural equation modeling. Structural Equation Modeling, 8(3), 353-377.

How many resamples to use in bootstrapping?

The default number of resamples is 100 for bootstrapping in WarpPLS. This setting can be modified by entering a different number in the appropriate edit box. (Please note that we are talking about the number of resamples here, not the original data sample size.)

Leaving the number of resamples for bootstrapping as 100 is recommended because it has been shown that higher numbers of resamples lead to negligible improvements in the reliability of P values; in fact, even setting the number of resamples at 50 is likely to lead to fairly reliable P value estimates (Efron et al., 2004).

Conversely, increasing the number of resamples well beyond 100 leads to a higher computation load on the software, making the software look like it is having a hard time coming up with the results. In very complex models, a high number of resamples may make the software run very slowly.

Some researchers have suggested in the past that a large number of resamples can address problems with the data, such as the presence of outliers due to errors in data collection. This opinion is not shared by the original developer of the bootstrapping method, Bradley Efron (see, e.g., Efron et al., 2004).

Reference:

Efron, B., Rogosa, D., & Tibshirani, R. (2004). Resampling methods of estimation. In N.J. Smelser, & P.B. Baltes (Eds.). International Encyclopedia of the Social & Behavioral Sciences (pp. 13216-13220). New York, NY: Elsevier.

Welcome to the WarpPLS blog!

Welcome to the WarpPLS blog! WarpPLS is a powerful structural equation modeling (SEM) software. It is commercialized by ScriptWarp Systems: www.scriptwarp.com.

Among other things, WarpPLS identifies nonlinear (or “warped”, hence the name of the software) relationships among latent variables and corrects the values of path coefficients accordingly. WarpPLS is arguably the first SEM software to do this.

Since most relationships between numeric variables are nonlinear, one could argue that WarpPLS finds the "real" relationships between latent variables in an SEM analysis. Typically path coefficients are increased, in some cases going from non-significant to significant at the P lower than 1 percent level.

The underlying algorithm employed by WarpPLS as its outer model default is partial least squares (PLS) regression, whose main characteristic is its ability to minimize multicollinearity among latent variables (even in the presence of overlapping manifest variables, or indicators). Other PLS-based outer model algorithms are also available, including PLS modes A and B.

Additionally, WarpPLS offers the following features, which are largely absent from most, if not all, PLS-based SEM software packages available today:

• It estimates P values for path coefficients automatically, instead of providing only standard errors or T values, and leaving the user to figure out what the corresponding P values are.
• It estimates several model fit indices, which have been designed to be meaningful in the context of PLS-based SEM analyses.
• It automatically builds the indicators’ product structure underlying moderating relationships, and goes a little further. It shows those moderating relationships, related path coefficients, and related P values in a model graph as they should be shown – that is, as links between latent variables and direct links. The latter connect pairs of latent variables, while the former connect latent variables and direct links between pairs of latent variables.
• It allows users to view scatter plots of each of the relationships among latent variables (when they are connected through arrows in the model), together with the curves that best approximate those relationships, and save those plots as .jpg files for inclusion in research reports.
• It provides a variety of graphs from which users can choose, including zoomed 2D graphs and 3D graphs; the latter for moderating effects. Both multivariate and bivariate relationship graphs are provided, for linear and nonlinear relationships, using standardized and unstandardized scales.
• It allows users to segment curves based on increments in the first derivative of the predictor latent variables on each of their criteria latent variables. This provides an alternative to data segmentation approaches such as FIMIX-PLS, without any reduction in sample.
• It calculates variance inflation factor (VIF) coefficients for latent variable predictors associated with each latent variable criterion. This allows users to check whether some predictors should be removed due to multicolinearity (this feature is particularly useful with latent variables that are measured based on only 1 or a few indicators).
• It calculates effect size coefficients analogous to Cohen’s f-squared coefficients for all paths. These are calculated as the absolute values of the individual contributions of the corresponding predictor latent variables to the R-square coefficients of the criterion latent variable in each latent variable block.
• It calculates indirect effects for paths with 2, 3 etc. segments; as well as total effects. The corresponding P values, standard errors, and effect sizes are also calculated. Indirect and total effects can be critical in the evaluation of downstream effects of latent variables that are mediated by other latent variables, especially in complex models with multiple mediating effects along concurrent paths.
• It calculates a variety of causality assessment coefficients, all of which are reported. These can be used in the assessment of the plausibility and direction of hypothesized cause-effect relationships.

These are only a few of the new features offered by WarpPLS.

Ned Kock

WarpPLS developer