In survey research, a sample of about $$1,000$$ individuals is typically considered adequate to make accurate estimates, regardless of whether the broader population consists of a hundred thousand or a hundred million people. This is because a $$95\%$$ confidence interval depends only on the size of the sample itself, not on the size of the population. Furthermore, while increasing the sample size continues to shrink the confidence interval—meaning the sample statistic is likely closer to the true population value—it does so at a progressively slower rate. Consequently, the minor gains in statistical confidence achieved by surveying significantly more than $$1,000$$ people are generally not considered worth the additional time and financial cost.

Sample Size and Population Size

To illustrate a confidence interval, suppose a sample of $$20$$ university students provides a mean calorie estimate for a cookie of $$200$$, with a $$95\%$$ confidence interval ranging from $$160$$ to $$240$$. This indicates that there is a high probability ($$95\%$$) that the true mean calorie estimate for the entire population of university students falls somewhere between $$160$$ and $$240$$. Furthermore, this interval allows for hypothesis testing; any hypothetical population mean outside of this $$160$$ to $$240$$ range (such as a hypothetical value of $$250$$) would be considered statistically significantly different from the sample mean at the $$.05$$ alpha level.

Example of a Confidence Interval

By definition, a confidence interval is a statistical range calculated around a sample statistic that is designed to contain which of the following a specified percentage of the time?

A statistical range calculated around a sample statistic designed to contain the true population parameter a specified percentage of the time, most commonly $$95\%$$. Some researchers advocate using confidence intervals as an alternative to traditional null hypothesis testing because they are often more intuitive to interpret. Furthermore, a confidence interval inherently provides the data needed to perform a null hypothesis test; if a hypothetical population mean falls completely outside the calculated interval, the sample mean is considered statistically significantly different from that hypothetical value.

Google

Inferential statistics are a set of mathematical techniques used by researchers to determine whether a statistical result or relationship observed in a study's sample reflects something that is genuinely true of the broader population.

Inferential Statistics

The optimal sample size for a survey generally depends on two primary factors: the researcher's desired level of statistical confidence and the practical constraints of the study's budget. While larger samples yield statistics that are closer to the true population value and provide greater confidence, they also require more time, effort, and money to collect. Researchers frequently use power analyses to balance these theoretical and practical considerations.

Determinants of Survey Sample Size

A sample statistic is a descriptive numerical summary—such as a mean or a correlation coefficient—that is calculated from the data of a specific sample. Researchers compute these statistics to summarize their observed data and to make inferences about the corresponding, unknown values in the broader population.

Sample Statistic

Null hypothesis testing is the most common approach to inferential statistics in psychological research. It provides researchers with a formal, structured process to evaluate whether an observed sample result likely occurred by random chance or whether it accurately reflects a genuine relationship or difference in the broader population.

Null Hypothesis Testing

https://kpu.pressbooks.pub/psychmethods4e/

KPU Research Methods in Psychology - 4th American Edition

Statistical conclusions in psychological research are fundamentally probabilistic, meaning that empirical findings are always subject to chance and the possibility of statistical errors, such as Type I or Type II errors. Because researchers cannot completely eliminate these potential errors, the results of any single empirical study can never provide absolute certainty or definitively prove that a scientific theory is true.

Probabilistic Nature of Statistics

Confidence Interval

A finding is considered statistically significant when it is highly unlikely to have occurred due to random chance alone. In the context of inferential statistics, achieving statistical significance suggests that the results represent a genuine, real effect within the broader population.

Statistically Significant

A Type I error is a false positive in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are statistically significant—thereby rejecting the null hypothesis—when, in reality, the null hypothesis is true and there is no actual relationship in the population. These errors happen because random sampling error can occasionally produce extreme results even when no true effect exists. The predetermined alpha level dictates the probability of committing this error when the null hypothesis is true.

Type I Error

A Type II error is a false negative in null hypothesis testing. It occurs when a researcher mistakenly concludes that their results are not statistically significant—thereby retaining the null hypothesis—when, in reality, the null hypothesis is false and a true relationship does exist in the population. In practical research, Type II errors typically happen because a study lacks adequate statistical power to detect the underlying relationship, often due to an insufficiently large sample size.

Type II Error

In a study of sex differences in talkativeness, Matthias Mehl and his colleagues observed that the women in their sample spoke a mean of $$16,215$$ words per day while men spoke a mean of $$15,669$$ words per day. Despite finding this difference within their specific sample, they ultimately concluded that there was no evidence of a genuine sex difference in talkativeness within the broader population.

Mehl's Study on Sex Differences in Talkativeness

In a study investigating the connection between daily hassles and stress symptoms, Allen Kanner and his colleagues discovered a positive correlation of $$+.60$$ within their sample. Based on this specific sample result, they concluded that a genuine relationship between daily hassles and symptoms exists in the overall population.

Kanner's Study on Daily Hassles and Symptoms

What is the primary purpose of using inferential statistics in psychological research?

When determining the optimal sample size for a survey, what are the two primary factors a researcher must generally balance?

A population parameter is a numerical value representing a specific characteristic of an entire population, such as the true mean or correlation. Because researchers typically cannot measure every individual within a population, they must rely on sample statistics to estimate these corresponding population-level values.

Population Parameter

There is a fundamental distinction between probabilities and statistics. Probabilities are theoretical quantities that describe the underlying data generating process, such as the inherent fairness of a coin. In contrast, statistics are empirical quantities computed directly from observed data, such as the observed frequency of a specific outcome in a sample.

Probability vs. Statistics

A statistical estimator is a specially designed function applied to an observed dataset to produce empirical estimates of underlying theoretical model parameters, such as probabilities. For example, a natural estimator for the true probability of an event is the ratio of its observed occurrences to the total number of sampling trials.

Statistical Estimator

A descriptive numerical summary, such as a mean or a correlation coefficient, that is calculated from the data of a specific subset of observed individuals is called a ____.

The $$t$$-test is the most frequently used null hypothesis test for evaluating the statistical relationship between two means. There are three primary variations of this procedure—the one-sample $$t$$-test, the dependent-samples $$t$$-test, and the independent-samples $$t$$-test—each designed to accommodate slightly different research designs.

t-Test

In null hypothesis testing, researchers make one of two formal decisions (to reject or retain the null hypothesis) concerning one of two possible states of reality (the null hypothesis is actually true or false). This intersection creates a matrix of four distinct possible outcomes. Two outcomes are correct decisions: rejecting a false null hypothesis and retaining a true one. The remaining two outcomes are statistical errors: rejecting a true null hypothesis (Type I error) and retaining a false null hypothesis (Type II error).

Outcomes of a Null Hypothesis Test

A major criticism of null hypothesis testing is that it is fundamentally uninformative. Because the null hypothesis posits that there is precisely zero relationship between variables in the population, critics argue that it is never literally true in the real world. Consequently, rejecting the null hypothesis merely confirms that some nonzero relationship exists, which provides researchers with very little precise information about the actual strength or nature of the phenomenon being studied.

Informativeness of Null Hypothesis Testing

Despite widespread criticisms, some statisticians and researchers strongly defend null hypothesis testing when it is understood and applied correctly. A primary argument in its defense is that it provides a standardized, principled method for researchers to demonstrate that their findings are not merely the result of random chance. This framework is considered especially useful and necessary when investigating new or unestablished scientific phenomena.

Defense of Null Hypothesis Testing

In 2015, the editors of the journal Basic and Applied Social Psychology took the unprecedented step of banning the use of null hypothesis testing and $$p$$-values in its publications. Instead of relying on traditional statistical significance thresholds to evaluate validity, the editors required authors to focus heavily on descriptive statistics and effect sizes. Although this ban was not widely adopted across the field, it significantly amplified the ongoing conversation in psychology about the limitations of the 'gold standard' of statistical validity.

The 2015 Ban on Null Hypothesis Testing

A quantitative measure, such as Cohen's $$d$$ or Pearson's $$r$$, used to estimate the strength of a statistical relationship within a population. In response to criticisms of null hypothesis testing, researchers are encouraged to report effect sizes alongside their tests. This practice is crucial because a $$p$$-value alone only indicates whether a relationship is statistically significant—which is heavily influenced by sample size—but fails to convey the actual magnitude or importance of that relationship.

Effect Size

In null hypothesis testing, researchers reject the null hypothesis in favor of the alternative hypothesis if the observed sample relationship would be extremely unlikely to occur under the assumption that the null hypothesis is true. This decision indicates there is sufficient statistical evidence to conclude that a relationship exists in the population.

Reject the Null Hypothesis

The $$t$$-test is the most commonly used null hypothesis test for evaluating statistical relationships that focus on the difference between two means.

$$t$$-Test

Null hypothesis testing is subject to several major criticisms, including frequent misunderstandings of results by researchers, the arbitrary nature of the rigid $$0.05$$ significance threshold, and the argument that rejecting the null hypothesis provides very little information about the actual relationship between variables in a population.

Criticisms of Null Hypothesis Testing

To illustrate the logic of null hypothesis testing, consider Mehl and colleagues' investigation into whether men and women differ in talkativeness. Assuming the null hypothesis (that there is no difference in the population) was true, they evaluated the likelihood of finding their sample's small difference of $$d = 0.06$$. Finding this result fairly likely under the null hypothesis, they decided to retain the null hypothesis, concluding there was no evidence of a sex difference in the population.

Mehl's Study on Talkativeness

Kanner and colleagues provided an example of rejecting the null hypothesis by studying the relationship between daily hassles and psychological symptoms. They asked how likely it would be to find their sample's strong correlation of $$+.60$$ if the null hypothesis (that there is no correlation in the population) were true. Because such a strong sample relationship would be fairly unlikely under the null hypothesis, they rejected the null hypothesis in favor of the alternative hypothesis, concluding a positive correlation exists in the population.

Kanner's Study on Hassles and Symptoms

Null hypothesis testing follows a structured logic: researchers first assume the null hypothesis is true, calculate how likely their sample results would be under this assumption, and then decide to either reject or retain the null hypothesis based on this calculated probability.

Learn Before

Related

Learn After