 | Prof. Shan HuangAssistant Professor The University of Hong Kong |
Shan Huang is an assistant professor at the Faculty of Business and Economics at the University of Hong Kong. From 2018 to 2020, she was an assistant professor at the Foster School of Business at the University of Washington, Seattle. She is a digital fellow at MIT Initiative on Digital Economy and Stanford Digital Economy Lab. Her research focuses on the digital economy, social networks, and business analytics (e.g., A/B testing). Shan’s current work aims to understand the business value and social implications of new social media. Specifically, her studies examine how social advertising and social referral affect product virality, how emotions shape online content diffusion, and how weak ties can or cannot break people out of the echo chamber, in massive social networks. She has a particular interest in understanding how certain phenomena vary across individuals, social ties, products, and markets, using population-scale datasets and large-scale field experiments, and uses various research methodologies (e.g., large-scale networked randomized field experiments, machine learning, network analysis, econometrics) to pursue her research agenda. Shan’s research has been published in prominent management journals, including Marketing Science and the Journal of Management Information Systems. She has been collaborating closely with the leading tech firms (e.g., Tencent) to understand the cutting-edge digital phenomena and the tools such as A/B testing.
Enhancing External Validity of Experiments with Ongoing Sampling
Abstract:
Participants in online experiments often enroll over time, which can compromise sample representativeness due to temporal shifts in covariates. This issue is particularly critical in A/B tests—online controlled experiments extensively used to evaluate product updates—since these tests are cost-sensitive and typically short in duration. We propose a novel framework that dynamically assesses sample representativeness by dividing the ongoing sampling process into three stages. We then develop stage-specific estimators for Population Average Treatment Effects (PATE), ensuring that experimental results remain generalizable across varying experiment durations. Leveraging survival analysis, we develop a heuristic function that identifies these stages without requiring prior knowledge of population or sample characteristics, thereby keeping implementation costs low. Our approach bridges the gap between experimental findings and real-world applicability, enabling product decisions to be based on evidence that accurately represents the broader target population. We validate the effectiveness of our framework on three levels: (1) through a real-world online experiment conducted on WeChat; (2) via a synthetic experiment; and (3) by applying it to 600 A/B tests on WeChat in a platform-wide application. Additionally, we provide practical guidelines for practitioners to implement our method in real-world settings.
Authors:
Shan Huang (with Chen Wang and Shichao Han)