You run a crosstab. The numbers look different across columns. Men answered 42%, women answered 51%. Is that a real difference, or just noise in the data?
Significance testing answers that question. It is the statistical backbone of every survey report, and yet it remains one of the most misunderstood parts of the workflow. This guide explains what significance tests do, how to read them, and how to run them without writing CTABLES syntax.
What Is Significance Testing?
Significance testing tells you whether a difference you observe in your survey data is likely to exist in the broader population, or whether it could have occurred by chance due to sampling variability.
In survey research, the most common form is the column proportion test (also called a z-test of column proportions). This test compares each pair of columns in a crosstab and flags cells where the proportions are statistically different at a given confidence level—usually 95%.
When a proportion in one column is significantly higher than in another column, that cell is marked with a significance letter. This is the A/B/C notation you see in banner tables and topline reports.
How to Read A/B/C Significance Letters
Each column in your crosstab is assigned a letter: the first column is A, the second is B, the third is C, and so on. When a cell displays a significance letter, it means:
“The proportion in this cell is significantly higher than the proportion in the column identified by the letter.”
For example, suppose you have a crosstab of brand preference by age group:
| 18–34 (A) | 35–54 (B) | 55+ (C) | |
|---|---|---|---|
| Brand X | 42% C | 38% | 28% |
| Brand Y | 31% | 35% | 44% A |
| Brand Z | 27% | 27% | 28% |
Reading this table:
- Brand X, 18–34 column shows “C”: 42% of 18–34-year-olds prefer Brand X, which is significantly higher than the 28% among 55+ respondents (column C).
- Brand Y, 55+ column shows “A”: 44% of 55+ respondents prefer Brand Y, which is significantly higher than the 31% among 18–34-year-olds (column A).
- Brand Z has no letters: The differences across age groups are not statistically significant.
The Chi-Square Test: When and Why
Before looking at individual cell comparisons, researchers often run a chi-square test of independence on the entire crosstab. This test answers a broader question: is there any statistically significant relationship between the row variable and the column variable?
The chi-square test compares the observed cell frequencies to the frequencies you would expect if the two variables were completely independent. A large chi-square statistic (and a small p-value) tells you that the pattern in the table is unlikely to have occurred by chance.
In practical terms:
- p < 0.05: There is a statistically significant association between the two variables at the 95% confidence level. The differences in your table are probably real.
- p ≥ 0.05: You cannot conclude that there is a meaningful relationship. The differences you see could be due to sampling variability.
The chi-square test tells you whether something is going on. The column proportion tests (A/B/C letters) tell you where the specific differences are.
How It Works in SPSS (The Hard Way)
In IBM SPSS, running significance tests on a crosstab typically requires one of two approaches:
Option 1: CROSSTABS command—produces a chi-square test but does not generate column proportion letters. You get the overall test of independence, but not the cell-level A/B/C notation that clients expect in banner tables.
Option 2: CTABLES with SIGTEST—produces the A/B/C letters, but requires the Custom Tables add-on module (sold separately from base SPSS) and CTABLES syntax. Example:
/TABLE brand BY agegroup
/CATEGORIES VARIABLES=agegroup [A, B, C]
/SIGTEST TYPE=PROP ALPHA=0.05.
Even with the right syntax, the results land in the SPSS Output Viewer. Getting them into a client-ready format still requires manual export and formatting.
How It Works in SavQuick (The Easy Way)
SavQuick runs significance tests automatically on every crosstab you create. There is no syntax to write, no add-on to purchase, and no separate dialog to configure:
- Load your .SAV file—drag and drop into the browser. No upload to a server.
- Select row and column variables—pick any two categorical variables and the crosstab builds instantly.
- Significance letters appear automatically—A/B/C letters are calculated using chi-square-based column proportion tests at the 95% confidence level and displayed directly in the table cells.
- Export with letters included—when you export to PowerPoint, Excel, or CSV, the significance letters come with the data.
All significance testing features are completely free for all users. No Pro subscription required.
Common Pitfalls with Significance Testing
Significance letters are powerful, but they are easy to misinterpret. Here are the most common mistakes researchers make:
1. Confusing “significant” with “important”
A statistically significant difference is not necessarily a practically meaningful one. With large sample sizes (n > 2000), even tiny percentage differences can reach significance. A 2-percentage-point difference between two groups might be statistically significant but irrelevant for business decisions. Always consider effect size alongside significance.
2. Ignoring small base sizes
Significance tests assume a minimum sample size to produce reliable results. When a column has fewer than 30 respondents, the test becomes unreliable. Most reporting standards flag bases under 30 as “too small for significance testing” and bases under 100 with a caution note. Check your column bases before reporting significance results.
3. Multiple comparisons problem
When you test many cells at the 95% confidence level, roughly 5% of tests will produce a false positive (Type I error) by chance alone. In a large banner table with 10 columns and 20 rows, that can produce several spurious significance letters. Be cautious about highlighting isolated significant results when the rest of the table shows no pattern.
4. Forgetting to weight the data
Significance tests should be run on weighted data when a weight variable is available. Unweighted tests can produce misleading results if the sample is not representative of the target population. In SavQuick, apply your weight variable from the toolbar before running the crosstab—the significance tests will automatically use the weighted values.
5. Treating no letter as “the same”
The absence of a significance letter does not mean the proportions are identical. It means the test did not find sufficient evidence to declare them different at the chosen confidence level. The difference might be real but too small to detect with your sample size. This is especially common with smaller subgroups.
Confidence Levels: 90%, 95%, or 99%?
The confidence level determines how strict the test is. A higher confidence level means fewer letters in your table but more certainty that each flagged difference is real:
| Level | Alpha | Meaning | Common Use |
|---|---|---|---|
| 90% | 0.10 | 10% chance the difference is due to chance | Exploratory research, small samples |
| 95% | 0.05 | 5% chance the difference is due to chance | Industry standard for most survey research |
| 99% | 0.01 | 1% chance the difference is due to chance | Academic research, high-stakes decisions |
Most market research and social research uses the 95% confidence level. This is the default in SavQuick and in most SPSS significance test configurations.
Significance Testing Checklist for Survey Reports
Before including significance letters in a client deliverable, run through this checklist:
- Check base sizes: Are all column bases above 30? Flag any that are not.
- Apply weights: If the study uses a weight variable, make sure it is applied before running tests.
- State the confidence level: Include a footnote specifying the confidence level (usually 95%).
- Explain the notation: Add a legend explaining that letters indicate significantly higher proportions.
- Watch for patterns, not isolated letters: A single significant letter in a large table may be a false positive. Look for consistent patterns across related variables.
- Consider practical significance: A 2-point difference that reaches statistical significance may not warrant a business recommendation.
Run Significance Tests in Minutes, Not Hours
Significance testing should not require a separate SPSS module, custom syntax, or manual letter assignment. Load your .SAV file into SavQuick, run a crosstab, and the significance letters are already there—calculated automatically and included in every export.
All significance testing features are free. No account required to get started.