Responsibilities

• Build, maintain, and prioritize a comprehensive testing roadmap across paid media, email/lifecycle, SEO, landing pages, onboarding, paywall, and creative.
• Develop a hypothesis backlog informed by quantitative funnel analysis, qualitative user insights, competitive research, and channel team input.
• Define and enforce a structured prioritization framework (ICE, PIE, or a custom variant) to stack-rank tests by potential impact, confidence, and ease of execution.
• Establish testing velocity benchmarks and ensure the team runs the optimal number of concurrent, non-interfering experiments at all times.
• Maintain an always-current view of what is being tested, what has been learned, and what is next, and communicate it weekly to leadership and channel owners.
• Design A/B, multivariate, holdout, and geo-based experiments with proper control groups, adequate sample sizes, and pre-defined success metrics.
• Partner with Data & Analytics and Engineering to ensure correct instrumentation, event tracking, and attribution before any test launches.
• Define primary, secondary, and guardrail metrics for every experiment to capture both intended lift and unintended side effects.
• Apply the right statistical method for the test type, traffic volume, and decision urgency, and know when to call a test early vs. wait for significance.
• Identify and mitigate threats to validity: novelty effects, seasonality, sample ratio mismatch, network effects, and interaction effects between simultaneous tests.

Requirements

• 3 to 5+ years of hands-on experimentation, CRO, growth marketing, or analytical marketing experience, preferably at a consumer subscription, marketplace, mobile-first, or dating/social platform.
• Ready to run on day one. You can audit an existing test program, identify what is misdesigned, and ship a better-instrumented experiment in week one.
• Deep understanding of experimental design: control/treatment setup, sample sizing, statistical significance, power calculations, novelty/seasonality/SRM threats, and how to mitigate them.
• Hands-on, in-the-tool expertise with at least one experimentation or A/B testing platform (Optimizely, VWO, Statsig, LaunchDarkly, Eppo, Convert, AB Tasty, or comparable).
• Strong analytical fluency. Comfortable in SQL and directly in the warehouse, plus a product analytics tool (Amplitude, Mixpanel, Looker, Heap, or similar) to pull your own analysis and pressure-test Data team outputs.
• Proven track record of running experiments across at least two of: paid media, email/lifecycle, landing pages, onboarding, paywall, or in-product marketing, with numbers you can speak to.
• Outstanding communication and storytelling. You can write a crisp experiment brief and present nuanced results to a non-technical audience with equal confidence.
• Experience with holdout group design and geo-based experiments, and a clear point of view on when each is the right tool.
• AI fluency. You already use AI tools to draft test plans, generate variant copy and creative, analyze results, write SQL, and automate repetitive tasks.
• Operator’s discipline.

Preferred

• Experience at a consumer subscription, marketplace, or dating/social platform where engagement and retention are primary growth levers.
• Familiarity with Bayesian experimentation methods and a clear view on when to apply them vs. frequentist approaches.
• Background in behavioral economics or consumer psychology.
• Experience building or contributing to a company-wide experimentation program from the ground up, including governance, tooling selection, and team training.
• Exposure to multi-armed bandit testing, contextual bandits, or adaptive experimentation.
• Experience designing experiments around AI-driven personalization or recommendation systems.
• Bachelor’s degree in Statistics, Computer Science, Marketing, or a related field, or an equivalent track record of rigorous experimentation