One-tailed or two-tailed P values in PLS-SEM?

Should P values associated with path coefficients, as well as with other coefficients such as weights and loadings, be one-tailed or two-tailed? This question is addressed through the publication below.

Kock, N. (2015). One-tailed or two-tailed P values in PLS-SEM? International Journal of e-Collaboration, 11(2), 1-7.

PDF file:

http://cits.tamiu.edu/kock/pubs/journals/2015JournalIJeC2/Kock_2015_IJeC_OneTwoTailedPLSSEM.pdf

Abstract:

Should P values associated with path coefficients, as well as with other coefficients such as weights and loadings, be one-tailed or two-tailed? This question is answered in the context of structural equation modeling employing the partial least squares method (PLS-SEM), based on an illustrative model of the effect of e-collaboration technology use on job performance. A one-tailed test is recommended if the coefficient is assumed to have a sign (positive or negative), which should be reflected in the hypothesis that refers to the corresponding association. If no assumptions are made about coefficient sign, a two-tailed test is recommended. These recommendations apply to many other statistical methods that employ P values; including path analyses in general, with or without latent variables, plus univariate and multivariate regression analyses.

Using WarpPLS in E-Collaboration Studies: Mediating Effects, Control and Second Order Variables, and Algorithm Choices

A new article discussing WarpPLS is available. The article is titled “Using WarpPLS in E-Collaboration Studies: Mediating Effects, Control and Second Order Variables, and Algorithm Choices”. It has been recently published in the International Journal of e-Collaboration. A full text version of the article is available here as a PDF file. Below is the abstract of the article.

This is a follow-up on two previous articles on WarpPLS and e-collaboration. The first discussed the five main steps through which a variance-based nonlinear structural equation modeling analysis could be conducted with the software WarpPLS (Kock, 2010b). The second covered specific features related to grouped descriptive statistics, viewing and changing analysis algorithm and resampling settings, and viewing and saving various results (Kock, 2011). This and the previous articles use data from the same e-collaboration study as a basis for the discussion of important WarpPLS features. Unlike the previous articles, the focus here is on a brief discussion of more advanced issues, such as: testing the significance of mediating effects, including control variables in an analysis, using second order latent variables, choosing the right warping algorithm, and using bootstrapping and jackknifing in combination.

Geographically distributed collaborative SEM analysis using WarpPLS

I am currently conducting a geographically distributed collaborative SEM analysis using WarpPLS. The analysis involves a few people in different states of the USA, and two people outside the country. The collaborators are not only separated by large distances, but also operate in different time zones.

Yet, we have no problems collaborating. The collaboration is asynchronous – one person does some work one day, and shares it with the others, who review the work in the next few days and respond.

Since we all have WarpPLS installed on our computers, we exchange different versions of a WarpPLS project file (extension “.prj”) with the same dataset. This way we can do analyses in turns, and discuss the results on emails.

Each slightly different project file is saved with a different name – e.g., W3J_InfoOvld_2010_03_02.prj, W3B_InfoOvld_2010_03_02.prj, W2J_InfoOvld_2010_03_02.prj etc.

In the examples above, the first three letters indicate the SEM algorithm used (W3 = Warp3 PLS Regression; W2 = Warp2 PLS Regression), and the resampling method used (J = jackknifing; B = bootstrapping). The second part of the name describes the dataset, and the final part the date.

This is just one way of naming files. It works for our particular project, but more elaborate file names can be used in more complex collaborative SEM analyses.

This geographically distributed collaborative SEM analysis highlights one of the advantages of WarpPLS over other SEM software: all that is needed for the analysis is contained in one single project file.

Moreover, the project file will typically be only a few hundred kilobytes in size. In spite of its small size, the file includes the original data, and all of the results of the analysis.

One member of our team asked me how the project file can be so small. The reason is that all of the SEM analysis results are stored in a format that allows for their rendering every time they are viewed.

Plots of nonlinear relationships, for example, are not stored as bitmaps, but as equations that allow WarpPLS  to re-create those plots at the time of viewing.

Bootstrapping or jackknifing (or both) in WarpPLS?

The blog post below refers to resampling methods, notably bootstrapping and jackknifing, which are often used for the generation of estimates employed and hypothesis testing. Even though they are widely used, resampling methods are inherently unstable. More recent versions of WarpPLS employ "stable" methods for the same purpose, with various advantages. See the most recent version of the WarpPLS User Manual (linked below) for more details.

http://www.scriptwarp.com/warppls/#User_Manual

***

Arguably jackknifing does a better job at addressing problems associated with the presence of outliers due to errors in data collection. Generally speaking, jackknifing tends to generate more stable resample path coefficients (and thus more reliable P values) with small sample sizes (lower than 100), and with samples containing outliers. In these cases, outlier data points do not appear more than once in the set of resamples, which accounts for the better performance of jackknifing (see, e.g., Chiquoine & Hjalmarsson, 2009).

Bootstrapping tends to generate more stable resample path coefficients (and thus more reliable P values) with larger samples and with samples where the data points are evenly distributed on a scatter plot. The use of bootstrapping with small sample sizes (lower than 100) has been discouraged (Nevitt & Hancock, 2001).

Since the warping algorithms are also sensitive to the presence of outliers, in many cases it is a good idea to estimate P values with both bootstrapping and jackknifing, and use the P values associated with the most stable coefficients. An indication of instability is a high P value (i.e., statistically insignificant) associated with path coefficients that could be reasonably expected to have low P values. For example, with a sample size of 100, a path coefficient of .2 could be reasonably expected to yield a P value that is statistically significant at the .05 level. If that is not the case, there may be a stability problem. Another indication of instability is a marked difference between the P values estimated through bootstrapping and jackknifing.

P values can be easily estimated using both resampling methods, bootstrapping and jackknifing, by following this simple procedure. Run an SEM analysis of the desired model, using one of the resampling methods, and save the project. Then save the project again, this time with a different name, change the resampling method, and run the SEM analysis again. Then save the second project again. Each project file will now have results that refer to one of the two resampling methods. The P values can then be compared, and the most stable ones used in a research report on the SEM analysis.

References:

Chiquoine, B., & Hjalmarsson, E. (2009). Jackknifing stock return predictions. Journal of Empirical Finance, 16(5), 793-803.

Nevitt, J., & Hancock, G.R. (2001). Performance of bootstrapping approaches to model test statistics and parameter standard error estimation in structural equation modeling. Structural Equation Modeling, 8(3), 353-377.

How many resamples to use in bootstrapping?

The default number of resamples is 100 for bootstrapping in WarpPLS. This setting can be modified by entering a different number in the appropriate edit box. (Please note that we are talking about the number of resamples here, not the original data sample size.)

Leaving the number of resamples for bootstrapping as 100 is recommended because it has been shown that higher numbers of resamples lead to negligible improvements in the reliability of P values; in fact, even setting the number of resamples at 50 is likely to lead to fairly reliable P value estimates (Efron et al., 2004).

Conversely, increasing the number of resamples well beyond 100 leads to a higher computation load on the software, making the software look like it is having a hard time coming up with the results. In very complex models, a high number of resamples may make the software run very slowly.

Some researchers have suggested in the past that a large number of resamples can address problems with the data, such as the presence of outliers due to errors in data collection. This opinion is not shared by the original developer of the bootstrapping method, Bradley Efron (see, e.g., Efron et al., 2004).

Reference:

Efron, B., Rogosa, D., & Tibshirani, R. (2004). Resampling methods of estimation. In N.J. Smelser, & P.B. Baltes (Eds.). International Encyclopedia of the Social & Behavioral Sciences (pp. 13216-13220). New York, NY: Elsevier.

Viewing and changing settings in WarpPLS 1.0 and 2.0

The blog post below refers to version 1.0 - 2.0 of WarpPLS. See this YouTube video on how to view and change settings for version 3.0. For more recent versions, see the WarpPLS User Manual and YouTube videos available from warppls.com.

***

The view or change settings window (see figure below, click on it to enlarge) allows you to select an algorithm for the SEM analysis, select a resampling method, and select the number of resamples used, if the resampling method selected was bootstrapping. The analysis algorithms available are Warp3 PLS Regression, Warp2 PLS Regression, PLS Regression, and Robust Path Analysis.

Many relationships in nature, including relationships involving behavioral variables, are nonlinear and follow a pattern known as U-curve (or inverted U-curve). In this pattern a variable affects another in a way that leads to a maximum or minimum value, where the effect is either maximized or minimized, respectively. This type of relationship is also referred to as a J-curve pattern; a term that is more commonly used in economics and the health sciences.

The Warp2 PLS Regression algorithm tries to identify a U-curve relationship between latent variables, and, if that relationship exists, the algorithm transforms (or “warps”) the scores of the predictor latent variables so as to better reflect the U-curve relationship in the estimated path coefficients in the model. The Warp3 PLS Regression algorithm, the default algorithm used by the software, tries to identify a relationship defined by a function whose first derivative is a U-curve. This type of relationship follows a pattern that is more similar to an S-curve (or a somewhat distorted S-curve), and can be seen as a combination of two connected U-curves, one of which is inverted.

The PLS Regression algorithm does not perform any warping of relationships. It is essentially a standard PLS regression algorithm, whereby indicators’ weights, loadings and factor scores (a.k.a. latent variable scores) are calculated based on a least squares minimization sub-algorithm, after which path coefficients are estimated using a robust path analysis algorithm. A key criterion for the calculation of the weights, observed in virtually all PLS-based algorithms, is that the regression equation expressing the relationship between the indicators and the factor scores has an error term that equals zero. In other words, the factor scores are calculated as exact linear combinations of their indicators. PLS regression is the underlying weight calculation algorithm used in both Warp3 and Warp2 PLS Regression. The warping takes place during the estimation of path coefficients, and after the estimation of all weights and loadings in the model. The weights and loadings of a model with latent variables make up what is often referred to as outer model, whereas the path coefficients among latent variables make up what is often called the inner model.

Finally, the Robust Path Analysis algorithm is a simplified algorithm in which factor scores are calculated by averaging all of the indicators associated with a latent variable; that is, in this algorithm weights are not estimated through PLS regression. This algorithm is called “Robust” Path Analysis, because, as with most robust statistics methods, the P values are calculated through resampling. If all latent variables are measured with single indicators, the Robust Path Analysis and the PLS Regression algorithms will yield identical results.

One of two resampling methods may be selected: bootstrapping or jackknifing. Bootstrapping, the software’s default, is a resampling algorithm that creates a number of resamples (a number that can be selected by the user), by a method known as “resampling with replacement”. This means that each resample contains a random arrangement of the rows of the original dataset, where some rows may be repeated. (The commonly used analogy of a deck of cards being reshuffled, leading to many resample decks, is a good one, but not entirely correct because in bootstrapping the same card may appear more than once in each of the resample decks.) Jacknifing, on the other hand, creates a number of resamples that equals the original sample size, and each resample has one row removed. That is, the sample size of each resample is the original sample size minus 1. Thus, the choice of number of resamples has no effect on jackknifing, and is only relevant in the context of bootstrapping.

Change resampling method in WarpPLS: YouTube video

The blog post below refers to resampling methods, bootstrapping and jackknifing, which are often used for the generation of estimates employed and hypothesis testing. Even though they are widely used, resampling methods are inherently unstable, as illustrated in the post. More recent versions of WarpPLS employ "stable" methods for the same purpose, with various advantages. See the most recent version of the WarpPLS User Manual (linked below) for more details.

http://www.scriptwarp.com/warppls/#User_Manual

***

A new Youtube video is available for WarpPLS:

http://www.youtube.com/watch?v=Hf-t70r7NKo

This video shows how one can conduct an SEM analysis using WarpPLS, save that analysis with a different project name, change the resampling method (from bootstrapping to jackknifing), and then redo the analysis.

At the end, the user has two project files, one with all of the P values calculated through bootstrapping, and the other with all of the P values calculated through jackknifing.

As noted in the WarpPLS User Manual, bootstrapping and jackknifing provide a good complement to each other in the context of warped PLS-based SEM.

Thus, some users may want to run two analyses of the same model, one with each resampling method,  and use the results that are associated with the most stable resample path coefficients. These will typically be the ones with the lowest P values, since P values go up as the standard errors in the resample set go up. High resample standard errors are associated with instability. The instability itself often comes from outliers, which may drastically change the shape of a warped relationship in each resample.

Well, moving from statspeach to plain English, there are good theoretical reasons to recommend that users choose the most stable results (i.e., with the lowest P values) as the results that they will use in research reports, whether they are obtained with bootstrapping or jackknifing. The choice may be made individually, for each path coefficient. This should be disclosed to the readers of the report; a sentence like this would probably be enough: "Both bootstrapping or jackknifing were used in the analyses. The results reported here are those associated with the most stable resample estimates."