top of page

How to analyze an A/B test with Claude and verify the p-value

2 minutes ago
9 min read
Cover with the title How to analyze an A/B test with Claude and verify the p-value over a teal city skyline, Data Studios original cover

Claude can analyze an A/B test end to end when code execution is switched on: it reads the uploaded CSV, runs the statistical test in Python inside its sandbox and reports the conversion rates, the difference, the p-value and a confidence interval. Most errors in this task come from choices made before any arithmetic, such as which test to run, whether the result is one-sided or two-sided, whether the traffic split is healthy and whether the conversion column was counted per visitor or per event.


The method below gives you a small illustrative experiment, the prompt to send, and the exact figures Claude should return, computed independently in Python with pandas and SciPy. With the illustrative data, variant B converts at 5.55% against 4.80% for variant A, a difference of 0.75 percentage points with a two-sided p-value of 0.0165 and a 95% confidence interval from 0.14 to 1.37 points. If Claude's answer differs from these numbers on the same data, one of the choices listed later has gone wrong.


The requirements come from Anthropic's help center articles on uploading files, on creating and editing files with Claude, and on model training, all checked on 21 November 2026.


··········


CLAUDE REQUIREMENTS FOR A/B TEST ANALYSIS ON UPLOADED DATA.


Statistical work on a dataset needs Claude's code execution environment, which Anthropic calls "Code execution and file creation". Without it, Claude reads the extracted text of an uploaded file and does the math in its reply, which is the setup most likely to produce a wrong p-value. With it, Claude writes and runs Python against the file, so the computation can be checked line by line.


Anthropic states that code execution and file creation is available on the Free, Pro, Max, Ultra, Team and Enterprise plans on the web, in Claude Desktop and in the mobile apps. It is on by default for individual plans and for Team organizations, and on by default for new Enterprise organizations; owners of Team and Enterprise plans can turn it off for the whole organization. Individual users find the toggle under Settings > Capabilities, where newer layouts label it "Create and edit files".


........


Item

What Anthropic documents

Plans with code execution

Free, Pro, Max, Ultra, Team, Enterprise

Default state

On for individual plans and Team; on for new Enterprise organizations

User setting

Settings > Capabilities

Admin setting

Organization settings > Capabilities

File types

CSV listed; XLSX requires code execution

Files per chat

Up to 20

File size, code environment

30MB per file for uploads and downloads

File size, chat uploads

500MB per file in the upload article

Network egress, Team and Enterprise

Set by owners; Enterprise off by default


........


The two file-size figures come from two different Anthropic articles and do not agree, so an experiment export that fits under 30MB avoids the question. A per-visitor A/B export with a user ID, a variant label and a 0/1 conversion flag stays small even at a few hundred thousand rows; event-level logs grow much faster and are better aggregated before upload. The same approach for oversized files is covered in the guide on analyzing a CSV over ChatGPT's 50MB limit, and the general workflow is described in how to use Claude for data analysis.


··········


THE ILLUSTRATIVE A/B TEST DATASET.


The experiment below is illustrative and does not describe a real company. A checkout page was split 50/50 between the current design (A) and a new design (B), with the plan of sending 10,000 visitors to each arm. Each visitor counts once, and a conversion means at least one completed order during the test window.


........


Variant

Visitors

Conversions

Conversion rate

A (control)

10,040

482

4.80%

B (new checkout)

9,960

553

5.55%

Total

20,000

1,035

5.175%


........


In a real upload the file would usually hold one row per visitor with columns such as user_id, variant and converted. The aggregated table is enough to reproduce every figure in the analysis, and it is also the first thing to ask Claude to produce from the raw rows, because a wrong visitor count at this stage invalidates everything after it.


··········


PROMPTING CLAUDE TO RUN THE A/B TEST.


A good prompt states the unit of analysis, the hypothesis direction, the significance level and the checks you expect before the test is run. Leaving these open lets the model pick defaults that may differ from your team's standard.


I uploaded ab_test.csv with one row per visitor: user_id, variant (A or B),
converted (1 if the visitor placed at least one order, else 0).
1. Check that user_id is unique and report visitors and conversions per variant.
2. Run a sample ratio mismatch check against a planned 50/50 split
   (chi-square goodness of fit, planned 10,000 per arm).
3. Run a two-sided two-proportion z-test of B vs A at alpha = 0.05,
   using the pooled standard error for the test statistic.
4. Report the absolute difference in percentage points, the relative lift,
   and a 95% confidence interval for the difference (unpooled standard error).
5. Show the Python code you ran and state every assumption you made.

Asking for the code is the step that makes the answer auditable. Claude's sandbox runs Python, so the reply should include the script, and you can rerun it locally against the same file. The same habit of asking for code and testing it is described in the guide on writing and debugging pandas code with Claude.


··········


THE CORRECT A/B TEST RESULT COMPUTED IN PYTHON.


The script below reproduces the analysis from the aggregated counts. It was executed with pandas and SciPy before publication, and its output is the reference to compare with Claude's answer.


import numpy as np
import pandas as pd
from scipy import stats

summary = pd.DataFrame({
    "variant": ["A", "B"],
    "visitors": [10040, 9960],
    "conversions": [482, 553],
})
summary["rate"] = summary["conversions"] / summary["visitors"]

a, b = summary.iloc[0], summary.iloc[1]
diff = b["rate"] - a["rate"]

# Two-proportion z-test with pooled standard error (two-sided)
pooled = summary["conversions"].sum() / summary["visitors"].sum()
se_pooled = np.sqrt(pooled * (1 - pooled) * (1 / a["visitors"] + 1 / b["visitors"]))
z = diff / se_pooled
p_value = 2 * stats.norm.sf(abs(z))

# 95% confidence interval for the difference (unpooled standard error)
se_diff = np.sqrt(a["rate"] * (1 - a["rate"]) / a["visitors"]
                  + b["rate"] * (1 - b["rate"]) / b["visitors"])
zcrit = stats.norm.ppf(0.975)
ci = (diff - zcrit * se_diff, diff + zcrit * se_diff)

# Sample ratio mismatch check against the planned 50/50 split
srm = stats.chisquare(summary["visitors"], [10000, 10000])

print(summary.round(4))
print(f"difference: {diff * 100:.2f} pp, relative lift: {(b['rate'] / a['rate'] - 1) * 100:.1f}%")
print(f"z = {z:.3f}, two-sided p = {p_value:.4f}")
print(f"95% CI: {ci[0] * 100:.2f} pp to {ci[1] * 100:.2f} pp")
print(f"SRM chi-square p = {srm.pvalue:.4f}")

........


Measure

Correct value

Conversion rate A

4.80%

Conversion rate B

5.55%

Absolute difference

0.75 percentage points

Relative lift

15.7%

z statistic

2.399

Two-sided p-value

0.0165

95% CI for the difference

0.14 to 1.37 points

SRM check p-value

0.5716


........


The sample ratio check passes: 10,040 against 9,960 is well within what a fair 50/50 split produces, so there is no sign that the assignment or the tracking dropped visitors from one arm. The test itself rejects the null hypothesis at the 5% level, and the confidence interval excludes zero. The interval is wide, though. A lift of 0.14 points and a lift of 1.37 points are both consistent with the data, and a business case built on the 15.7% relative lift is relying on the most likely value of a broad range.


··········


A/B TEST ERRORS TO CHECK IN CLAUDE'S ANSWER.


The same data can produce several different p-values depending on choices the prompt may not have fixed. Comparing Claude's output with the reference table usually points straight to the cause.


........


What you see

Likely cause

Correct handling

p = 0.0082

One-sided test

Use two-sided unless pre-registered

p = 0.018

Chi-square with Yates correction

Valid but more conservative; state it

p = 0.0165 from a t-test

Welch t-test on 0/1 rows

Matches at this size; z-test is standard

Rates near 4.8% and 5.5% but counts differ

Events counted, not visitors

Deduplicate user_id first

No SRM check reported

Step skipped

Rerun with the check


........


A one-sided p-value is half the two-sided one, so a model that "helpfully" tests only whether B beats A reports 0.0082 and makes the result look stronger than the team's usual standard allows. Yates' continuity correction for the 2x2 chi-square test gives 0.018; it is a legitimate method but should be named. On data of this size a Welch t-test on the 0/1 conversion column gives the same p-value to four decimals, while on small samples the methods diverge and the choice has to be explicit.


The more serious failures are upstream. If the export contains one row per session or per order, a visitor with three orders counts three times, the visitor totals inflate and the test treats correlated rows as independent observations. Peeking is the other trap: running the test every day and stopping on the first p-value below 0.05 inflates the false-positive rate well above 5%, and Claude has no way of knowing how often the test was checked unless you tell it. A related check on extreme values, which can distort revenue-per-visitor metrics, is covered in the guide on detecting outliers with IQR and z-score checks.


Sample size belongs in the review as well. For a baseline of 4.8% and a target of 5.5%, a two-sided test at alpha 0.05 with 80% power needs about 15,632 visitors per arm, computed with the standard power calculation for two proportions. The illustrative test ran with about 10,000 per arm, so it was underpowered for a lift of that size and reached significance because the observed difference was larger. Asking Claude to compute the required sample size before the test starts, and keeping the stopping rule fixed, removes most of the room for a misleading result.


··········


WHAT STAYS HUMAN-APPROVED IN AN A/B TEST DECISION.


Claude can run the arithmetic and explain the result, but the decisions that make the arithmetic valid belong to the team running the experiment. The hypothesis direction, the significance level, the primary metric and the stopping rule should be written down before launch and given to Claude in the prompt before any data is analyzed. The ship decision should also weigh guardrail metrics, such as refunds or support contacts, that the conversion test does not see.


A practical review routine has four steps: confirm the visitor and conversion counts per variant against the analytics tool, rerun Claude's script locally, compare the p-value and interval with an independent calculation like the one above, and record the assumptions Claude stated next to the result.


··········


CLAUDE DATA PRIVACY SETTINGS FOR EXPERIMENT DATA.


Experiment exports often contain user IDs or other identifiers, so the data-handling rules depend on the plan. Anthropic's privacy center says that for consumer products (Free, Pro and Max) chats and coding sessions are used to improve its models if the user allows it through the Model Improvement setting in the privacy settings, if a conversation is flagged for safety review, or if the user explicitly opts in to training. Incognito chats are not used for training even with Model Improvement enabled.


For commercial products such as Claude for Work and the Anthropic API, Anthropic states that by default it does not use inputs or outputs to train its models, unless the customer explicitly reports feedback or bugs or chooses to allow it. Conversations submitted through the thumbs up or down feedback button are stored for up to 5 years on both consumer and commercial products, and Team and Enterprise owners can disable feedback submission through the Rate chats setting under Organization settings > Data and Privacy.


Anthropic's article on file creation also describes the sandbox as an isolated container and warns that prompt injection could lead Claude to send information from its context outside the sandbox when network access is enabled. For experiment data with identifiers, a hashed or dropped user_id column and an organization with network egress restricted to package managers, or turned off, reduce that exposure.


··········


ANALYZING A/B TESTS WITH CLAUDE: WHAT DEPENDS ON YOUR SETUP.


Anthropic documents that code execution is available on every Claude plan, that CSV files are supported, that XLSX uploads need code execution switched on, and that up to 20 files can be attached to a chat. The per-file size limit is stated as 500MB in the upload article and 30MB for the code execution environment, so files under 30MB are the safe choice. Whether code execution and network access are available in a company account depends on the settings chosen by Team or Enterprise owners.


Anthropic does not document a dedicated A/B testing feature, and none is needed: the analysis is ordinary Python run in the sandbox, which is why it can be verified. On the illustrative data, the correct answer is a 0.75-point difference, a two-sided p-value of 0.0165, a 95% interval from 0.14 to 1.37 points and a passing sample ratio check. Any other figure on the same data points to a one-sided test, a different correction, double-counted visitors or a skipped check, and each of those can be found by reading the code Claude ran.


··········


FOLLOW US FOR MORE.


·····


DATA STUDIOS


·····


datastudios.org


bottom of page