Learning Goals and Objectives

1.1. Understand where data come from

1.1.1. Understand and apply concepts of measurement

Quantify variation in data by assigning numbers or categories to attributes (2)

Differentiate quantitative and categorical levels of measurement (2)

Explain the inherent issue of measurement error (2)

Recognize the limitations of measurement, including measurement bias (2)

1.1.2. Understand sampling methodology

Explain that samples are the result of a sampling process and a data generating process (DGP) (2, 3)

Define independent, random sampling and explain the consequences of violating independent, random sampling (2)

Explain and apply the definition of sampling variation and consider it a possible explanation for observed group differences (2, 4)

Recognize that samples (and even random samples) are not perfectly representative of the entire population (2)

Recognize that samples are studied in order to find out about the population and the DGP (2)

Recognize the limitations of sampling, including sampling bias (2)

1.1.3. Understand elements of research design

Distinguish outcome variables versus explanatory variables (2,3, 4)

Differentiate observational and experimental studies and recognize the limitations of each (4)

Differentiate random sampling and random assignment and explain the advantages of each (4)

1.2. Understand the organization of data

1.2.1. Understand data frames

Interpret the structure of data: rows (i.e. observations) and columns (i.e. variables) (R) (2)

Differentiate and interpret variables versus values in a data set (2)

Differentiate and interpret levels of variables (2)

1.2.2. Manipulate data in a data frame

Write code to perform basic R commands (R) (1 - 12)

Create summary variables (R) (2)

Aggregate variables across rows to create a new data frame (R) (2)

Recode quantitative variables into categorical variables e.g., using ntile() (R) (2)

Identify and decide how to handle missing data (R) (2)

Filter, organize, and manipulate data in a data frame; interpret the results (e.g. arrange, select; R) (2)

1.3. Understand the purpose of data

1.3.1. Imagine data that could answer a question

Generate questions that could be answered with a given data set (2)

1.3.2. Generate questions you could ask of data

Evaluate appropriateness of data to specific questions and purposes (2)

2.1. Understand sources of variation

2.1.1. Categorize sources of variation

Recognize that variation can be divided into explained and unexplained; unexplained variation into real (i.e. a product of the system) or induced; induced variation into measurement error, sampling error, or mistakes (4)

2.1.2. Understand what it means to explain variation

Explain and apply an intuitive definition of ‘explaining variation’ (4)

Identify within and between group variations in a graph (4)

Apply the definition of explaining variation to graphs (e.g. scatterplots, faceted histograms) (4)

2.1.3. Understand correlation, causation, and confounding

Identify and apply the concepts of confounding variables and the problem of directionality (4, 7)

Differentiate experimental from observational designs; understand which can lead to causal conclusions (4, 7, 8, 11)

Recognize that randomness can cause an apparent relationship in data (4)

2.2. Describe distributions of data

2.2.1. Understand concept of distribution

Define a distribution as the pattern of variation in a variable or combination of variables (3)

Describe a distribution using shape, center, and spread, and reason about the possible causes of a distribution's characteristics (3)

Reason about measures of central tendency (5)

2.2.2. Understand that distributions of data are generated by DGP, which is usually unknown

Recognize that populations are the long-run result of a host of causal processes we call the DGP; (3)

Simulate samples of data from a DGP; anticipate what the data may look like ® (3, 4, 9, 10)

Generate hypotheses about the DGP based on distributions of data; recognize limitations (3, 4)

Recognize the role of randomness and the law of large numbers in interpreting distributions of data (3)

2.2.3. Represent univariate distributions in tables and graphs

Create frequency tables and relative frequency tables (e.g., tally) (R) (3)

Create and interpret box plots, histograms, and bar graphs to visualize univariate data (R) (3)

Choose the appropriate visualization to represent a given variable (3)

Create and interpret a five-number summary (i.e., favstats()) (R) (3)

Find and interpret the interquartile range (IQR) (4)

Identify outliers and decide how to handle them (3)

2.2.4. Interpret tables and graphs of univariate distributions

Interpret the axes of a histogram: identify that frequency is represented on the y axis; variable is represented on the x axis (3)

Interpret bar graphs and understand why shape, center, and spread are not meaningful characteristics for these visualizations (3)

Recognize and explain the differences and similarities between a frequency histogram and a relative frequency histogram (3)

2.3. Describe relationships in data

2.3.1. Represent bivariate relationships in tables and graphs

Create two-way contingency tables (e.g. using tally function) with both frequencies and proportions (4) (R)

Create scatterplots, faceted histograms, box plots, jitter plots to visualize bivariate data (4) (R)

Choose the appropriate visualization to represent a bivariate relationship (4)

2.3.2. Interpret tables and graphs of bivariate relationships

Interpret bivariate relationships presented in tables, scatter plots, faceted histograms (4)

Visually evaluate the strength of relationships in scatter plots, faceted histograms, box plots (8)

2.3.3. Represent relationships in word equations

Create path diagrams to represent bivariate relationships (4)

Write word equations to represent relationships between explanatory and outcome variables (4)

3.1. Understand data = model + error

3.1.1. Understand concept of statistical models

Map DATA = MODEL + ERROR in different contexts (4, 5, 7, 8)

Define and use the concept of a statistical model as a function that produces a predicted score for each observation (5, 7, 8)

Differentiate between the empty model and a more complex model (5, 7, 8)

Identify the most appropriate model to use based on how variables are measured (e.g., group versus regression models) (5, 7, 8)

Distinguish between a model of data and a model of DGP (5, 6, 7, 8)

3.1.2. Understand concept of error

Visually intuit which models or distributions show more error (4, 5, 7)

Define, calculate, and reason about residuals from models (5, 6, 7, 8)

Recognize that residuals aggregate to measures of error (7)

Recognize that error from the empty model is all of the unexplained variation (6)

3.1.3. Understand partitioning error

Recognize that sum of squares from the empty model (SS total) can be partitioned into error and model sums of squares (6)

Recognize that SS total is determined by the outcome variable and so total variation for an outcome variable will be the same regardless of explanatory variables included in the model (7,8)

Interpret visual representations of SS error, SS model, and SS total from empty, group, and regression models (8)

3.1.4. Understand how transforming variables can help build models

Use difference scores to model paired samples (7)

Use standardized variables (e.g., z-scores) to compare the strengths of linear relationships between different sets of variables (6, 8)

Calculate and interpret Pearson’s R, and estimate correlation coefficients from scatterplots (8)

Link transformed variables to visualizations (e.g., scatterplots, linear representations of models) (6, 8)

3.2 Specify models

3.2.1. Write models using GLM notation

Use and interpret GLM notation to represent models of data, the DGP, and sampling distributions (5, 7, 8)

Flexibly use GLM notation to represent group and regression models in different contexts (7, 8)

Differentiate between variables and parameters in GLM equations (5, 7, 8)

Use dummy codes to model multi-group data in GLM (7)

3.2.2. Interpret parameter estimates in context

Map GLM components to contexts, graph, and word equations (5, 7, 8)

Interpret GLM components computationally (5,7,8)

3.3. Fit models

3.3.1. Understand what it means to fit a model

Recognize that fitting a model means to calculate best fitting parameter estimates (5,7,8)

Recognize that the best fitting model minimizes the appropriate measure of error (in this course, sum of squares) (6, 7, 8)

Understand the concept of “overfitting” a model (7)

3.3.2. Estimate parameters

Differentiate between parameters and statistics; explain why a statistic is our best estimate of a (usually) unknowable parameter (5, 7, 8)

Estimate parameters for empty models, group models, and regression models (i.e., fit models) (5, 7, 8) (R)

Represent and interpret visual representations of models (e.g. vline() (5, 7, 8)

3.3.3. Interpret estimates

Interpret the output of lm() for empty, group and regression models (R) (5,7,8)

Interpret parameter estimates for group and regression models in context (8)

Write best fitting model based on lm() output (5,7,8)

3.4. Assess model fit

3.4.1. Use residuals to assess model fit

Recognize that the empty model is made up of all unexplained variation (5)

Recognize that a residual from the empty model can be partitioned into error and model (7, 8)

Identify and interpret residuals on a graph of data for empty, group, and regression models (6, 7, 8)

Plot and interpret distributions of residuals (R) (7, 8)

Generate and analyze predictions and residuals from a model (R) (6, 7, 8)

3.4.2. Quantify aggregate error around a model

Quantify aggregate error as Sum of Absolute Deviations (SAD), Sum of Squares (SS), variance, and standard deviation, and interpret these measures (R) (6)

Explain how sums of squares are constructed from residuals (6)

Recognize the pros and cons of different methods of quantifying error (6)

Calculate and interpret z-scores within a distribution or across distributions (6)

3.4.3. Compare the fit of two models

Use SS, PRE, and F statistic to compare models (7, 8)

Calculate (e.g., find ANOVA tables; R) and interpret PRE and F statistic (R) (7, 8)

Interpret a t distribution and relate the critical t to critical z (10)

Interpret PRE similarly across ANOVA models and regression models: recognize that these two measures express the same thing (8)

3.4.4. Understand and calculate different measures of effect size

Describe and determine appropriate measures of effect size such as difference of means, PRE, Cohen’s d (7, 8)

3.5 Use models to generate predictions and probabilities

3.4.5. Use the normal distribution to model variation

Explain why the normal distribution may be used to model variation in the DGP/population (6)

Identify the features of a normal distribution and use data to construct best fitting normal model of variation (e.g., defined by mean and standard deviation) (6)

3.4.6. Make point predictions based on parameter estimates

Use parameter estimates in functions to make point predictions from the empty, group, and regression models (R) (6)

3.4.7. Make likelihood predictions based on probability distributions

Explain the concept of probability distribution as a model of a random DGP (6, 9)

Explain the relationship between a discrete (e.g. data, simulated normal distributions) and continuous probability distribution (6)

Use the empirical rule to estimate the likelihood of scores under a normal distribution (6)

Use the distribution of data to predict likelihood of future observations falling within a specified range (6)

Use the normal distribution to estimate likelihood of future observations falling within a specified range (R) (6)

4.1. Model the distributions of estimates

4.1.1. Understand the concept of sampling distribution

Explain the concept of a sampling distribution (9)

Explain the problem a sampling distribution solves, i.e. that it helps to make sense of a particular estimate given sampling variation (9)

Interpret sampling distributions in specific contexts (9)

Recognize that sampling distributions can depict the random distribution of any statistic (not just the sample mean) from some DGP (9)

Recognize that sampling distributions are imaginary (9)

4.1.2. Understand features of sampling distributions

Describe the shape, center, and spread of a sampling distribution of estimates, including SDoM, b0, b1, F, and PRE (9, 10, 11)

Explain how the shape/center/spread of population and sample size are related to sampling distributions, and how the population's features will affect shape/center/spread of sampling distributions (9)

Define and calculate standard error (9)

Identify standard error visually (9)

4.1.3. Construct sampling distributions (R)

Construct sampling distributions by resampling (bootstrapping), simulation, randomization, and using mathematical probability distributions (R) (9, 10, 11)

Explain the relationship between a discrete (e.g. data, simulated normal distributions) and continuous probability distribution (6)

Be able to predict the center and shape of a sampling distribution depending on the method used to construct it (9, 10, 11)

4.1.4. Interpret sampling distributions

Calculate and evaluate the likelihood of selecting a random sample with a certain sample statistic, (e.g., a t distribution) (R) (9)

Identify what questions can and cannot be addressed using sampling distributions; i.e. questions about individual observations require a population distribution whereas questions about samples require a sampling distribution (9)

4.2. Use confidence intervals to compare models

4.2.1. Understand confidence intervals

Recognize that just as a fixed DGP could produce a range of estimates, a single estimate could be produced by a range of DGPs (9)

Recognize that a confidence interval represents a range of parameter values that could, with some degree of likelihood, have produced your estimate (10)

Define margin of error and be able to find it (10)

Recognize how standard error relates to confidence intervals (10)

Recognize that the upper and lower bounds of a confidence interval represent the highest and lowest parameter values beyond which our sample would have been unlikely (10)

4.2.2. Construct a confidence interval around an estimate (R)

Construct confidence intervals using simulation, bootstrapping, and the mathematical probability distribution (i.e., z, t) for various estimates (e.g., b0, b1) (10) (R)

Explain how level of confidence, sample size, and variance impact the size of a confidence interval (10)

4.2.3. Interpret a confidence interval

Interpret confidence intervals in regression and grouping models for various estimates (e.g., b0, b1) (10)

Explain why you might reject the empty model when a confidence interval for b1 contains 0 (10)

Interpret a confidence interval correctly (10)

4.3. Compare models using the F distribution

4.3.1. Understand F

Define F as a measure of the strength of a relationship per parameter used in a model; and as a ratio of variation explained to variation unexplained (11)

Use simulation to explore which statistics can be modeled with the F distribution. (R) (11)

Interpret an F distribution, and distinguish the F statistic from the F distribution (11)

Recognize the that the shape of the F distribution depends on the degrees of freedom for model and for error (11)

4.3.2. Conduct model comparison

Conduct F tests for ANOVA and regression models in some context for some purpose (7, 8)

Identify the region that corresponds to p-value in a sampling distribution of F or PRE (11)

Define p-value as the probability of obtaining an F or PRE statistic as extreme or more extreme than the one observed assuming that the empty model is true (11)

4.3.3. Interpret results of model comparison

Determine which statistics can be used to compare two group or three group models versus an empty model (11)

Recognize the limitations of the F test in a three group model (11)

Explain the relationship between an F test and a t test (11)

Recognize and explain the need for simple effects tests (11)

Use the results of ANOVA tables to make predictions about confidence intervals (11)

Explain the problem of simultaneous comparisons and how it is addressed by the bonferroni correction (11)

Explain the inherent risk of statisticians’ p hacking; calculate likelihood of p-hacking (11)

Explain and recognize Type I error and why its probability of occurring is never zero (11)

Explain the use of an alpha level and how it relates to type I and type II error (11)

Explain and recognize Type II error (11)

Use the F statistic to compare models; intuit about effect size based on size of F ratio (7)

Interpret a p-value and link to the assumed Data Generating Process (11)