Johanna Schmidle
Owns a ton of
beaded jewelry

ChicBeads

• Control Campaign: The control campaign focused on traditional display ads featuring static images of their latest clothing collection.
• Test Campaign: The test campaign experimented with video ads showcasing short fashion shows and styling tips from influencers.

• Part 1: Python Notebook
  1. Clean Data
  2. Visualize Data
  3. Statistical Tests + Hypothesis Testing (Mann-Whitney U Test/ Two-Sample T-Test, Pairwise Tests)
• Part 2: Interactive Dashboard
  

Summary of Findings

From Visualizations

• The control campaign reaches a broader audience, but that does not result in more people adding to cart. The test campaign seems to be more focused, resulting in a stronger correlation between Add to Cart and Purchase.
• The advertisement seems to match what customers are searching for (this is inferred from the strong correlation between views, website clicks, add to cart)
• The Control Campaign is more cost-effective than the Test Campaign
• The Test Campaign has the most sales
• Recommended for the Test Campaign to spend more, and the Control Campaign to spend less.

Mann-Whitney U Test/ Two-Sample T-Test

• Purchase: The Test and Control campaigns have no significant difference in Purchase rates.
• WebsiteClicks: There is no significant difference in WebsiteClicks rates between Test and Control.
• Reach: The difference in Reach rates between the Test and Control is statistically significant.
• AddtoCart: The difference in AddtoCart rates between Test and Control is statistically significant.

Pairwise Tests

• Purchase: There is no meaningful difference in the Purchase rates between the Control and Test campaigns.
• WebsiteClicks: The Test campaign does not lead to a statistically significant difference in WebsiteClicks compared to the Control campaign.
• Reach: The Test campaign has a significantly higher impact on Reach compared to the Control campaign, with a large effect size. The evidence strongly supports the presence of a meaningful difference.
• AddtoCart: The Test campaign has a significantly different impact on AddtoCart compared to the Control campaign, with a large effect size.

Additional Performance Indicators

• Cost Per Acquisition (CPA): $4.67
• Return on Investment (ROI): The Control Campaign performed better.
• Conversion Rate: The Control Campaign performed better.
• Click Through Rate (CTR): The Test Campaign performed better.
• The Test and Control campaigns have no significant difference in ROI rates.
• There is no significant difference in Conversion Rate rates between the Test and Control campaigns.
• There is a statistically significant difference in CTR rates between the Test and Control campaigns.

Overall

• Control Campaign: Although the Control Campaign effectively reaches a broader audience, this does not result in higher Add to Cart or Purchase rates. The campaign is more cost-effective, indicated by a higher ROI and Conversion Rate.
• Test Campaign: The Test Campaign is more effective in converting viewers to customers, as seen in the strong correlation between Add to Cart and Purchase rates. The Test Campaign also has a higher CTR, indicating more engaged viewers.
• Reach and Add to Cart: Both metrics show statistically significant differences favoring the Test Campaign.
• Purchase and Website Clicks: No significant differences between campaigns.
• The return on the advertisement spend is higher in the Control Campaign than the Test campaign.

Recommendations For Company

• Continue with the original advertising strategy. Continuously experiment and perform A/B testing to refine and identify the most effective strategies.
• Focus on more targeted advertising strategies, similar to those used in the Test Campaign, to improve conversion rates.
• Ensure that advertisements align well with what customers are searching for, as evidenced by the strong correlation between Searches and View Content.

Recommended Next Analysis Steps

• Regression Analysis: Construct a polynomial regression model to predict the number of Impressions, Website Clicks, and Purchases. This can be used to identify optimum levels of expenditure on the Control and Test campaign.
• Trend Analysis: Conduct a time series analysis to understand how campaign performance changes over time and identify any seasonal trends.
• Segmentation Analysis: Break down the results by demographic segments (e.g., age, gender, location) to understand which segments respond better to each campaign.

Database Information

Database: https://www.kaggle.com/datasets/amirmotefaker/ab-testing-dataset

• Campaign Name: The name of the campaign
• Date: Date of the record
• Spend: Amount spent on the campaign in dollars
• Number of Impressions: Number of impressions the ad crossed through the campaign
• Reach: The number of unique impressions received in the ad
• Number of Website Clicks: Number of website clicks received through the ads
• Number of Searches: Number of users who performed searches on the website
• Number of View Content: Number of users who viewed content and products on the website
• Number of Add to Cart: Number of users who added products to the cart
• Number of Purchase: Number of purchases