Toolbox

Resources, software, tutorials, coding projects, and tools developed by us or by other research groups.

🦉 denotes resources developed by members of our lab

Data analysis & scientific computing guides 📊💻🔍#

Data sci & statistics#

• R for Data Science
  Free online book on how to do data science with R (largely in tidyverse). Learn how to get your data into useful structures, visualize it, and analyze it.

• Statistical inference via data science
  Online tutorial on data science in R, larging using tidyverse.

• Tidyverse skills for data science
  Another online tutorial about the tidyverse ecosystem of R packages, with practical examples.

• Statistical methods for behavioral and social sciences
  Free online materials for a statistics course designed by Tobias Gerstenberg. Last updated 2024/08/16. Best used in combination with course notes.

• Design of experiments: Advanced statistics
  Course materials for Elizabeth Page-Gould’s graduate class on advanced statistics.

• General linear models
  Course materials and video lectures for Patrick S. Forscher’s class on general linear models.

• Data science: Theories, models, algorithms, and analytics
  Class notes developed by Sanjiv Ranjan Das for his course on machine learning with R.

• Understanding mixed-effects models though data simulation
  Tutorial on simulating data for mixed-effects models in R, by Lisa Debruiine and Dale Barr. Includes extended examples with cross-classified models and binomial data structures. See associated paper.

• An introduction to data analysis
  By Michael Franke. Last updated 2023/02.

• Probabilistic models of cognition

• 10 common statistical mistakes to watch out for when writing or
  reviewing a manuscript
  

• The Good Research Code handbook
  For grad students, postdocs, & PIs who do a lot of programming as part of their research.

• R best practices
  Suggested list of best practices for coding in R and organizing projects.

• R code for meta-analyses
  Introduction to meta-analyses with focus on statistical methods. Last updated 2023/10.

• Learn R 4 free
  More R resources and tutorials.

• Simulation for power analysis

Data visualization#

• Data visualization with ggplot2 🦉
  Paper on data visualization (philosophy and tutorial using ggplot2 in R), with commented code for a wide variety of common data viz situations (means, proportions, relationships). Co-authored with Eric Hehman.

• Modern data visualization with R
  Free online guide by Robert Kabacoff.

• Data visualization with R
  Tutorial by Martin Schweinberger, aimed at beginners and intermediate users of R.

• Data visualization course
  A full data visualization course by Andrew Heiss.

• The R graph gallery
  A collection of charts made using R.

• Choosing what type of data visualization to use
  Helpful data viz cheatsheet found in a twitter thread.

• From Data to Viz
  Super clean and useful resource for deciding what type of visualization to use with your data, and the R code to implement it.

• 10 simple rules for better figures

• Multiple views on how to choose a data visualization

• Beautiful plotting in ggplot2: A cheatsheet
  

• 10 simple rules for a good poster presentation

Machine learning#

• Open-source video lectures from the Summer Institutes in
  Computational Social Science
  Boot camp materials.

• Google Colab tutorials | Python
  Coding tutorials for Python, data science, and machine learning, with a few hands-on examples in Google Colab.

• TensorFlow tutorials | Python
  Coding tutorials for ML and deep learning methods in Google Colab.

• HuggingFace course | Python
  Another Coding tutorial for ML and deep learning methods.

• Lightweight machine learning classics with R
  Online textbook on machine learning with R.

• Intro to machine learning in Python

• Course notes on optimization for machine learning

• CalTech 2022 data science and AI for neuroscience summer school
  Videos and slides.

• Neuromatch computational neuroscience course
  Good introduction to theoretical modeling and data-driven analyses. The substantive focus is less relevant for our lab but the tutorials are helpful.

Tools for collecting data & running experiments 📦🛢️#

• ExperienceSampler app extension for jsPsych 🦉
  Open-source smartphone app framework for building your own experience-sampling study app, integrated into jsPsych (7.3). App development code and documentation builds on the ExperienceSampler framework developed by Sabrina Thai & Liz Page-Gould. Built with Cordova, an open-source library that allows JavaScript to access native device functions on smartphones. Last updated 2023/09.

• jsPsych
  Open-source Javascript library for running experiments on the internet in a web browser, with numerous contributors. Currently managed by Josh de Leeuw, Becky Gilbert, and Björn Luchterhandt.

• Experiment Maker
  Drag-and-drop, “WYSIWYG” browser-based editor for behavioural experiments, created by Tamara Schembri and Peter Budziszewski. No coding required, free templates.

• PsyToolkit
  Free and easy-to-use toolkit for generating experiments, programming, and running psychological studies. Part experiment generator, part teaching aid.

• How to maintain data quality when you can’t see your participants
  

• SoPHIE Labs platform
  For online experiments, behavioral research, and games.

• Chatplat
  Platform that allows you to design, administer, record, and analyze real chats between participants.

• Overview of social media research tools
  Last updated 2019/06.

Theory construction & metascience 💡#

• Theory before the test: How to build high-versimilitude
  explanatory theories in psychological science
  By Iris van Rooij and Giosuè Baggio.

• Beyond Newton: Why assumptions of universality are critical
  to cognitive science, and how to finally move past them.
  By Ivan Kroupin, Helen Davis, and Joseph Henrich.

• A contextualist theory of knowledge: Its implications for
  innovation and reform in psychological research
  By William J McGuire.

Some background reading on the various replication/theory/generalizability/validation crises in our field:

• Theory in social psychology: Seeing the forest and the trees
  By Yaacov Trope. See the article by Tal Yarkoni below for a kind of counterargument.

• Choosing prediction over explanation in psychology:
  Lessons from machine learning
  By Tal Yarkoni and Jacob Westfall arguing for a shift away from parsimonious explanations and toward prediction.

• Integrating explanation and prediction in computational social science
  By computational social scientists and metascientists arguing for the field’s approach to resolving the trade-off between parsimony/explanation and complexity/prediction in social science.

• Dynamical minimalism: Why less is more in psychology
  By Andrzej Nowak, on a non-computational approach to reconciling the trade-off between parsimony and complexity in psychological theories.

• Replicability, robustness, and reproducibility in psychological science
  Metascientists’ reflection on the replication crisis.

• The generalizability crisis
  By Tal Yarkoni.

• The validation crisis in psychology
  By Ulrich Schimmack. So many crises!

• Addressing the theory crisis in psychology
  By Klaus Oberauer and Stephan Lewandowsky.

Some approaches to theory-building from psychological scientists:

• Making a theory useful: Lessons handed down
  By Tory Higgins.

• Dynamical minimalism: Why less is more in psychology
  By Andrzej Nowak, on one approach to reconciling the trade-off between parsimony and complexity in psychological theories.

• A perspectivist approach to theory construction
  By William J McGuire, on a ‘perspectivist’ approach which “assumes that all hypotheses and theories are true–as all are false–depending on the perspective from which they’re viewed.”

• Collaboration: The social context of theory development
  By John M Levine and Richard L Moreland.

• Mind the gap: In praise of informal sources of formal theory
  By Susan Fiske.

• Creative hypothesis generating in psychology: Some useful heuristics
  By William J McGuire.

• The replication crisis in psychology: An overview for theoretical
  and philosophical psychology
  By Bradford Wiggins and Cody Christopherson.

Software & packages 🛠️#

• Bootstrapping confidence intervals around ICCs 🦉
  R script & tutorial to boostrap 95% confidence intervals around intraclass correlation coefficients (ICCs) and variance components in linear mixed-effects models (based on lme4). The 95% CIs of ICCs can then be examined for overlap to determine if they differ. Tested on two-level and cross-classified multilevel models (but should accommodate any number of clusters and their interactions). Last updated 2020/02.

• Estimate ICCs in linear mixed-effect models 🦉
  Convenience function to estimate intraclass correlation coefficients from lmer models (lme4) in R. Accommodates any number of clusters and their interactions. Formulas for calculating ICCs based on Raudenbush & Bryk (2002). Last updated 2021/10.

• Sampling from data to assess when averages are stable 🦉
  R script & tutorial to assess how many observations are needed for an average to become stable. Co-authored with Gabe Nespoli, Eric Hehman, Eugene Ofosu.

Other reading lists & book recommendations 📚#

• Cognitive modeling greatest hits
  

• Gantman lab library
  Topics include moral cognition, moral values in institutional settings, and gender and morality.

• Computing & modeling reading list
  Topics include network analysis and dimensionality reduction.

• IMPACT lab reading list
  

• Introductory statistics