Impact of Sales Force Structure Change on Products Performance

Pilot Study

Business Intelligence Solutions
June, 2015

Objectives/Business Questions

  • Does 2-up promotion of Product A have a positive impact on its sales relative to 1-up promotion?
    • The hypothesis behind 2-up promotion: Engaging a 2nd representative in the promotion will accelerate product adoption and have a positive impact on product performance relative to 1-up promotion
    • Testing 1-up versus 2-up promotion will allow an assessment of the impact and relative value of a 2nd representative engaged in active promotion of Product A within a selected customer segment

  • Does the incremental revenue associated with the 2nd sale representative actively promoting Product A provide an acceptable return on investment?

  • Is promoting Product A the best short-term use of the SF1 sales force capacity?

Findings / Conclusions

  • There is no statistically significant  or practically
        important difference in Product A sales between
        Test 1 and Control 1 groups

  • Promotion cost for Control 1 group is two times higher than for Test 1 group

  • The 2-up structure does not produce desired/expected outcome for Product A

Structure of Test – Control Groups

  • Test 1: Product B and Product C
  • Test 2: Product B,  Product C,  and Product A
  • Control: Product B, Product C
  • Control groups are formed on the basis of the last 2014 quarter sales data
  • The Test and Control groups were selected to allow for a sufficient number of matched customers across the two groups to account for other variables that may impact Product A sales
    • By matching locations with respect to other variables (DTC, business size, geography, etc.) we can effectively isolate the number of representatives actively promoting  Product A as the differentiating factor between the groups



  • Form Test1- Control1 and Test2 - Control2 groups, using the data of the last quarter of 2014 and propensity score technique with:
    • nonparametric nonlinear logistic model
    • greedy one-to-one matching technique

  • Develop Stochastic Gradient Boosting regression models for the first quarter of 2015 for each pair of Test – Control groups, using the following dependent variables:
    • Product B sales
    • Product A sales
    • Product C Sales
  • controlling for all
    • “User demographics” variables (sales potential, milestone, state, business size, etc.)
    • promotion variables in last quarter of 2014
  • Estimate the difference in sales for different sales team

One-to-one Matching on Propensity Score
Propensity Score Basics

  • Propensity score
    • is the predicted probability of receiving the treatment (probability of belonging to a test group)
    • is a function of several differently scaled covariates

  • Propensity_Score  = f (Product_B_Sales_Pre,  Product_A_Sales_Pre,
                    Product B_Sales_Potential,          
                    State , Product A_Sales_Potential,
                    Product B_Potential_Decile,                      Promotion variables, etc.)                            where f is a non-parametric non-linear multivariate function, unique for each pair of Test – Control study
    • If State in ('MA', 'MI', 'MN', 'IL', 'FL', 'NJ') then DTC_Indicator = 1; else DTC_Indicator=0;
    • If State in ('NC', 'CA', 'NY', 'GA', 'VA') then Paper_Indicator = 1; else Paper_Indicator = 0;

  • A sample matched on propensity score will be similar across all covariates used to calculate propensity score

Control Groups

  • Control groups are formed on the base of propensity score methodology, using only the last 2014 quarter data

  • Control1 (for Test1 group with 547 Users):
    • Users are from Product A 1 – 8 deciles and from the following States: AL, FL, MI, MN, NC, NJ, WI 
    • Total Unmatched Number of Users: 4,244
    • Matched Number of Users: 543

  • Control2 (for Test2 group with 717 Users):
    • Users are from Product A 1 – 8 deciles and from the following States: AL, FL, MA, MN, NC, NJ, TN, WI 
    • Total Unmatched Number of Users: 6,784
    • Matched Number of Users: 717

Propensity Scores Calculation

  • Approaches/software on non-parametric logistic regression:
    • SAS SEMMA (Sample, Explore, Modify, Model, Assess) methodology within SAS Enterprise Miner
    • SPSS CRISP (Cross Industry Standard Process for Data Mining)
    • Salford Systems CART, MARS, TreeNet, and Random Forest
  • Approach selected: SAS SEMMA within SAS Enterprise Miner and Stochastic Gradient Boosting of Salford Systems
    • Test1 – Control1: (543 Product Users per group)
    •  Best model: Funnel architecture of Neural Net
    • Test2 – Control2: (717 Product Users per group)
    •  Best model: Cascade Correlation architecture of Neural Net

Propensity Score: Selecting the Best Modeling Paradigm

a second picture

Propensity Score for Test1 – Control1 Groups: Selecting the Best Modeling Method

a third picture

Propensity Score for Test2 – Control2 Groups: Selection the Best Modeling Method

a fourth picture

Matched-Pair Samples Comparison

  • Non-parametric tests:
    • For interval variables:
      • Kolmogorov-Smirnov Two-Sample Test
    • For nominal variables:
      • Chi-square test
  • Before matching there was a significant difference in predictor distribution across all variables for
    • Test1 – Control1
    • Test2 – Control2
  • After matching there was no significant difference in predictor distribution across all variables for 
    • Test1 – Control1
    • Test2 – Control2

Sales Analysis by Group
TreeNet/Stochastic Gradient Boosting Modeling

  • Total number of predictors: 42
  • Non-parametric model structure:

    Dep_var_Post = f(Dep_var_Pre,                         Promo_vars_Pre,                   …    User_demographics_vars)

Dependent Variable: Product B Sales Post

picture five

Dependent Variable: Product B Sales Post for  Test1 – Control1

sixth picture

Dependent Variable: Product C Sales Post for Test1 – Control1

seventh picture

Dependent Variable: Product A Sales Post for Test1 – Control1

pic eight