Test list
Klaviyo facts below apply to Klaviyo campaigns, which is your platform. Idea 3 changes two things (hero photo and button text), so I split it into T3a and T3b, one factor each. So your 6 ideas become 7 tests.
| Test | From | What changes (only this) | Factor | Winning metric |
|---|---|---|---|---|
| T1 | Idea 1 | Subject line: emoji vs no emoji | subject line | Open rate (with caution) |
| T2 | Idea 2 | Preview text: names the tea vs asks a question | preview text | Open rate (with caution) |
| T3a | Idea 3, part 1 | Hero photo: current vs new | content (image) | Click rate |
| T3b | Idea 3, part 2 | Button text: current vs "Brew yours" | content (button) | Click rate |
| T4 | Idea 4 | Sender name: "Brightloom Tea" vs "Mara at Brightloom" | sender | Open rate (with caution) |
| T5 | Idea 5 | Send time: morning vs evening | send time | Send-time test, no winner send |
| T6 | Idea 6 | Product grid: 3 teas vs 6 teas | content (layout) | Placed order rate |
Hypotheses to check, not predictions: - T1: an emoji in the subject line changes open rate. - T2: naming the tea in the preview text changes open rate. - T3a: the new hero photo changes click rate. - T3b: "Brew yours" changes click rate. - T4: "Mara at Brightloom" changes open rate. - T5: morning vs evening changes opens or clicks; Klaviyo reports it as a send-time test. - T6: a 3 tea grid changes placed order rate. Placed order rate is allowed because your Placed Order metric comes from Shopify.
Open rate caution:
| Measure | Yours | Klaviyo caution line |
|---|---|---|
| Apple Mail share of opens | 52% | about 45% |
Your share is above the line in Platform facts, so open rates may be inflated. For T1, T2 and T4, read open rate next to a custom report that uses the MPP property, or treat the open rate result as weak.
Split math
You asked for my default, so test size is 40%, with 2 variations (Klaviyo recommends 2). Rule: per variant = 18,400 x 40% / 2; remainder = 18,400 minus both variants; whole recipients, rounded down.
| Test | Test size | Recipients per variant | Remainder (gets the winner) | Over 50 per variant | Over 1,800 per variant |
|---|---|---|---|---|---|
| T1 | 40% | 3,680 | 11,040 | yes | yes |
| T2 | 40% | 3,680 | 11,040 | yes | yes |
| T3a | 40% | 3,680 | 11,040 | yes | yes |
| T3b | 40% | 3,680 | 11,040 | yes | yes |
| T4 | 40% | 3,680 | 11,040 | yes | yes |
| T5 | 100% | all 18,400 | 0 | n/a | n/a |
| T6 | 40% | 3,680 | 11,040 | yes | yes |
Each split test puts 3,680 recipients in each variant, above both delivery counts in Platform facts. That is about sends, not results; the label depends on win probability.
Calendar
One test per send date; Feb 11 falls in your blackout, so it has no test.
| Send date | Type | Test |
|---|---|---|
| Jan 7 | campaign | T1 Subject line: emoji vs no emoji |
| Jan 14 | campaign | T2 Preview text: names the tea vs asks a question |
| Jan 21 | campaign | T3a Hero photo: current vs new |
| Jan 28 | campaign | T3b Button text: current vs "Brew yours" |
| Feb 4 | campaign | T4 Sender name: "Brightloom Tea" vs "Mara at Brightloom" |
| Feb 11 | blackout (Feb 10 to Feb 12) | no test |
| Feb 18 | campaign | T5 Send time: morning vs evening |
| Feb 25 | campaign | T6 Product grid: 3 teas vs 6 teas |
| Mar 4 | campaign | no test (all tests placed) |
| Mar 11 | campaign | no test (all tests placed) |
How to read results
| Klaviyo label | What it means (Platform facts) | Next action |
|---|---|---|
| Statistically significant | each variation over 50 deliveries and win probability above 90% | keep the winner as your default |
| Promising | over 50 deliveries each, win probability above 75% up to 90% | retest 2 or 3 more times before you change the default |
| Not statistically significant | 75% or lower with over 1,800 deliveries per variation, or below 60% with under 4 points between the top two | drop it; no clear winner |
Platform facts also advise against testing near a holiday weekend. Your input names no holidays besides the Feb 10 to Feb 12 blackout, so I did not move any test for that; see question 4.
Counts
| Item | Count |
|---|---|
| Ideas you gave | 6 |
| Tests after splitting | 7 |
| Send dates | 10 |
| Blackout dates with a send | 1 |
| Tests scheduled | 7 |
| Send dates left with no test | 2 |
| Tests waiting for a date | 0 |
Questions for you
- Is 40% the test size you want, or do you want a smaller test with a larger winner send?
- How long should Klaviyo wait before it sends the winner? Your input does not say, so I left the test duration blank.
- For T5, what exact morning and evening times should I use?
- Does any send date sit near a holiday weekend for your customers? If yes, I will move that test to a free date (Mar 4 or Mar 11).
Needs your decision
- Approve the calendar before any test campaign is scheduled or sent: HELD.