Resources: TOP II Guidelines, FAQ/Guidance

Prepared by the SPSP Top II Task Force

FAQ for Data and Code Sharing: Troubleshooting Common Complexities
Preregistration Reporting: Sample Text
Calculating and Reporting Power
Trusted Repositories

FAQ for Data and Code Sharing: Troubleshooting Common Complexities

How do I know if some of my variables might contain identifying information?

HIPAA has some useful guidance here about how to de-identify data.

What are my options if I have some identifying/potentially identifying variables in my dataset that are used in my analyses (e.g. birthday, postal code)?

If the identifying variables were used to derive less identifying variables (e.g., if you used participants' date of birth to derive their age), post the derived variable (e.g., age) rather than the raw variable (e.g., date of birth). You can also do this for variables that are identifying only because they are very specific (e.g., remove some decimal places from GPS coordinates, or categorize the coordinates into larger and less identifying divisions).

Work with a repository that allows gated access (e.g. ICPSR) to share non-identifying data publically, but restrict access to identifying variables. If you go this route, clearly describe in the public repository and the manuscript the steps that are necessary to gain access to the non-public data (e.g., obtaining ethical board approval for secondary data analysis and emailing proof to the corresponding author). Note: The full dataset should still be uploaded to the repository (just not shared publicly) and the author is responsible for maintaining and sharing any private data in a timely manner with researchers who follow the required steps. Depending on the level of sensitivity of the data, researchers may need to seek out specific repositories to privately house data.

If it is impossible to share much (or any) of the data publicly, explain why in the cover letter of the submission (see section on Data Sharing in the Submission Guidelines). In such cases, we strongly recommend considering what aggregated statistics you can provide in your manuscript to improve transparency. For example, including a correlation matrix between all variables allows reproduction of many key statistics.

Further reading and examples of (partial) sharing of identifying datasets:

Gilmore, R. O., Kennedy, J. L., & Adolph, K. E. (2018). Practical solutions for sharing data and materials from psychological research. Advances in Methods and Practices in Psychological Science, 1 (1), 121-130.
Levenstein, M. C., & Lyle, J. A. (2018). Data: Sharing is caring. Advances in Methods and Practices in Psychological Science, 1 (1), 95-103.
Joel, S., Eastwick, P. W., & Finkel, E. J. (2018). Open sharing of data on close relationships and other sensitive social psychological topics: Challenges, tools, and future directions. Advances in Methods and Practices in Psychological Science, 1 (1), 86-94.

What if the data/code/materials I used are not something I can freely share due to legal reasons (e.g., materials are copyrighted and I obtained them through a licensing agreement)?

If you do not have the rights to share the raw data/code/materials, you must indicate in the manuscript what steps the reader can follow to access them (e.g., what entity the reader should contact to enter into their own licensing agreement).

See also the options for datasets with identifying information (above).

What if my dataset is very large?

Many open data storage solutions can handle individual files up to 2-5 GB in size. If your dataset exceeds this, consider whether you can save it as a more compressed file type.

For data from neural studies (e.g., MRI, MEG, EEG, iEEG, or ECoG), consider OpenNeuro.org, which can handle massive file sizes.

How can I ensure my data/code are usable and ready to be shared?

In addition to following the Submission Guidelines, it can be helpful to ask a colleague to go through your data and code and ensure they can make sense of it. Can they understand the dataset? Are they able to reproduce your key analyses? Can they access everything they need without your help?

What if someone finds an error in my data or analysis code?

Errors will always be handled on a case-by-case basis. In general, if errors are identified during review, authors can address the error during the revision process. Errors identified after publication will typically result in an erratum to fix the issue, unless the error substantially affects the conclusions or meaning of the paper.

Do I need to get consent from my participants to share their data? What if my consent form did not include a data sharing provision?

Consent forms should indicate that anonymized data will be shared with other researchers. If the consent form did not specify this, you may need to check with your local ethics board before sharing data.

For specific advice on language to include in your consent form and what to do if you already collected data without getting participants' explicit consent to share, consult this tutorial:

Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 1, 131-144.

What if I don't have analysis code because I used point-and-click options in SPSS to analyze my data?

We highly recommend re-running your analyses and using the "paste" option to create a record of the code that was used to conduct the analyses. You can verify that this script reproduces your numbers and then save and share this file. Alternatively, or if you're using a program that does not provide a way to save syntax, you can provide very precise instructions such that someone else can reproduce your steps exactly (perhaps including screenshots or a video recording of the entire analysis).

Return to top

Preregistration Reporting: Sample Text

Examples of how to report which elements of a study were or were not preregistered:

In Study 1, our preregistration (https: //link) included the study design, planned sample size, inclusion/exclusion criteria, and planned primary analyses.
All of the studies reported in this manuscript were preregistered (Study 1: https: //link; Study 2: https: //link) and all preregistrations included the study design, a pre-planned stopping rule, and inclusion/exclusion criteria. No planned analyses were included in the preregistrations.
No studies in this manuscript were preregistered.

Examples of how to report whether there were any deviations:

In Studies 1 and 2, we report all preregistered analyses in the main body of the paper, and we clearly indicate all deviations from the preregistered analysis plan (none in Study 1; two in Study 2).
All preregistered analyses in Study 1 are reported in the main text of the manuscript and in the supplement. There were no deviations from our preregistered stopping rule, exclusion criteria, or analyses.

Return to top

Calculating and Reporting Power

The power of a statistical test (the likelihood that a test will detect an effect if one exists in the population) is a function of sample size and the expected effect size (as well as other factors, as discussed in a later section). At the most basic level, power calculations typically involve inputting two of these quantities to calculate the third. Which quantities you want to input versus calculate will depend on what is most helpful for planning or understanding a statistical test in your particular study.

Tools for calculating power

There are many tools available (many for free) and we only cover a few here. For basic power calculations, try G*Power, JAMOVI (through an add-on library), or SPSS. R has many packages available that can handle simple and complex designs, for example the ‘pwr’ package. For calculating power to detect parameter estimates in SEM, try pwrSEM. For multilevel modeling, a data simulation method can be useful (Lane & Hennes, 2017).

If you are inputting an effect size (often called an a priori power analysis):

The effect size estimate must come from outside the study (in other words, don’t calculate “observed power,” which is when you use the effect size estimate from a statistical test to calculate the power of that same statistical test, and which is not a new piece of information but rather just a monotonic mathematical transformation of the test’s p-value).
Provide a rationale in the manuscript for where the effect size estimate (or range of estimates) came from. For example, you may use an effect size estimate from a prior meta-analysis that is unlikely to have been affected by publication bias, or an effect size estimate from a prior large study (see Schönbrodt & Perugini, 2013). Or, if you have no idea what effect size to expect, you may want to use Richard et al. (2003)’s estimates of the average effect size in published personality and social psychology articles based on a large meta-analysis of meta-analyses (i.e., Mdn correlation = .18, M = .21, SD = .15). Richard et al. (2003) also offer hundreds of meta-analytically-derived, topic-specific effect sizes in their Table 1 and Appendix B.
If you planned your sample size based on an a priori power analysis but your actual sample size ended up being substantially different (e.g., you planned to collect N = 500 but were able to collect only N = 400), please also report a sensitivity power analysis that computes the minimum sample size you were powered to detect with 80% power given your sample size.

If you have no rationale for what effect size to expect:

Conduct a sensitivity power analysis to determine the minimum effect size you could detect with 80% power given your sample size.

If your power analysis focuses on the test of an interaction:

Note that effect sizes for interactions involving continuous (vs. categorical) variables are often substantially smaller than effect sizes for simple effects. The sample size required to detect an interaction may therefore be considerably larger than the sample size required to detect one of the constituent effects (see Da Silva Frost & Ledgerwood, 2020, pp. 3-4; McClelland & Judd, 1993).

For all power analyses:

Describe all information necessary for an independent researcher to reproduce the results of your power analysis, including the program you used, the specific test for which you were calculating power (e.g., was it one of the main effects, an interaction, or a follow-up pairwise comparison?), and all input values (e.g., effect size estimate, desired power, mean or median correlation among repeated measures).
Consider including, in your Discussion section, a discussion of power and how it informs your confidence or tentativeness in your conclusions (see Da Silva Frost & Ledgerwood, 2020).

Other factors that influence power:

Several methodological factors can increase power such as using a within-subjects design instead of between-subjects manipulation (Greenwald, 1976; Rivers & Sherman, 2018) as well as strengthening a manipulation or improving the reliability of your measures (Ledgerwood & Shrout, 2011). For experimental studies, it may be possible to pre-determine a planned covariate that correlates strongly with the dependent measure of interest (e.g., measuring age as a covariate for a study that examines health as the primary dependent variable; Wang, Sparks, Gonzales, Hess, & Ledgerwood, 2017). Power can also be affected by the ranges and variances of predictor variables (e.g., optimal designs) and by using non-redundant predictors (e.g., orthogonal contrasts; McClelland, 1997; McClelland & Judd, 1993).

Examples of how to write up a power analysis in a method section:

Example 1: A priori power analysis (calculate N given effect size estimate and desired power)

To determine the sample size for our primary outcome (the correlation between X with Y) we conducted an a priori power analysis in the ‘pwr’ 1.3-0 R package (Champley, 2020). We aimed for 80% power assuming a two-sided test and alpha level of .05. For our effect size estimate, we used the meta-analytic estimate for the correlation between X and Y across all ten previous studies conducted by our lab (r = .3). This analysis indicated that we would need a total sample size of N = 84 in our new study. Given that we had the resources to collect a larger sample and that we expected we would have to drop a few participants based on our a priori exclusion criteria, we decided to recruit 100 participants.

Example 2: Sensitivity analysis (calculate effect size that could be detected with a given level of power and N)

We decided to collect data until the end of the semester, which resulted in a total sample size of N = 140 (n = 70 per condition). We performed a sensitivity power analysis in G*Power 3.1 for our primary outcome comparing the mean of the control and experimental groups with an independent samples t-test, assuming a two-tailed test and an alpha of .05. A sample of this size would provide 80% power to detect an effect of Cohen’s d = .48 and 60% power to detect an effect of Cohen’s d = .38. For reference, some estimates place the median effect size in social psychological research around d = .38 (Lovakov & Agadullina, 2017).

Example 3: Estimated power for a range of effect size estimates (Calculate power provided by a given sample size to detect a range of effect sizes)

We set an a priori target sample size of N = 200 (n = 100 per condition). A recent meta-analysis suggests that the effect of X on Y is in the range of d = .27 to d = .38 (citation). Power analyses in G*Power 3.1, assuming a two-tailed test and an alpha of 0.5, indicated that a sample of N = 200 provided 48% power to detect an effect as small as d = .27 and 76% power to detect an effect as large as d = .38.

References

Da Silva Frost, A., & Ledgerwood, A. (2020, March 4). Calibrate Your Confidence in Research Findings: A Tutorial on Improving Research Methods and Practices. https://doi.org/10.31234/osf.io/6uxkb

Greenwald, A. G. (1976). Within-subjects designs: To use or not to use? Psychological Bulletin, 83(2), 314–320. https://doi.org/10.1037/0033-2909.83.2.314

Ledgerwood, A., & Shrout, P. E. (2011). The trade-off between accuracy and precision in latent variable models of mediation processes. Journal of Personality and Social Psychology, 101(6), 1174–1188. https://doi.org/10.1037/a0024776

McClelland, G. H. (1997). Optimal design in psychological research. Psychological Methods, 2(1), 3–19. https://doi.org/10.1037/1082-989X.2.1.3

McClelland, G. H. (2000). Increasing statistical power without increasing sample size. American Psychologist, 55(8), 963–964. https://doi.org/10.1037/0003-066X.55.8.963

McClelland, G. H., & Judd, C. M. (1993). Statistical difficulties of detecting interactions and moderator effects. Psychological Bulletin, 114(2), 376–390. https://doi.org/10.1037/0033-2909.114.2.376

Richard, F. D., Bond, C. F., Jr., & Stokes-Zoota, J. J. (2003). One Hundred Years of Social Psychology Quantitatively Described. Review of General Psychology, 7(4), 331–363. https://doi.org/10.1037/1089-2680.7.4.331

Rivers, A. M., & Sherman, J. (2018, January 19). Experimental Design and the Reliability of Priming Effects: Reconsidering the "Train Wreck". https://doi.org/10.31234/osf.io/r7pd3

Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47(5), 609–612. https://doi.org/10.1016/j.jrp.2013.05.009

Wang, Y. A., Sparks, J., Gonzales, J. E., Hess, Y. D., & Ledgerwood, A. (2017). Using independent covariates in experimental designs: Quantifying the trade-off between power boost and Type I error inflation. Journal of Experimental Social Psychology, 72, 118–124. https://doi.org/10.1016/j.jesp.2017.04.011

Return to top

Trusted Repositories

What to look for in a data archive: “A proper, trustworthy archive will: (1) assign an identifier such as a “handle” (hdl) or “digital object identifier” (doi); (2) require that you provide adequate documentation and metadata; and (3) manage the “care and feeding” of your data by employing good curation practices” (Goodman et al., 2014).

When sharing data, code, materials, and/or pre-registrations during the peer-review process, please do so in a way that retains the double-masked nature of peer review. Some repositories have specific features for anonymizing content; others require workarounds.

Trusted data repositories and example capabilities

	Free to researchers	Restricted data access capabilities for sensitive data	Private URLs for access to non-public content	Professional curation services	DOI minting	Usage statistics	Accepts non data files(e.g. materials, code)	Additional notes
Dataverse	✓	✓	✓		✓	✓	✓	Files can be anonymized for peer review using workarounds.
Dryad	See note		✓	✓	✓	✓		Free to individuals from member institutions (e.g., all UCs) and from low to middle income economies.
Figshare	✓	✓	✓		✓	✓	✓	Private urls can anonymize project content for masked peer review; private storage with ability to grant access to certain individuals. Partnership with SAGE enables authors to track views and downloads.
ICPSR	See note	✓		✓	✓	✓	✓	Users may have to pay for curation services in order to make the dataset freely available to everyone.
openICPSR	See note	✓			✓	✓	✓	Free to deposit data. Researchers may have to pay to access restricted access data.
OpenNeuro	✓					✓		Specialized for neurodata, can accommodate very large individual file sizes (+200GB)
OSF	✓		✓		✓	✓	✓	Private urls can anonymize project content for masked peer review; private storage with ability to grant access to certain individuals
Zenodo	✓	✓			✓	✓	✓	Private storage with ability to grant access to certain individuals

Return to top