Study on BT Sales Employees
Around 600 sales people were assessed from August - November 2004 from 5 job groups within the Major Business team of the Commercial & Brands business unit of BT Retail. The main aim was to provide BT with information about the profile of good performers within the different groups in their sales force. This exercise was intended as a first step in the process of understanding the profile required for BT's future sales force to ensure that BT can remain competitive in what has become a fast-moving and rapidly changing industry,
This exercise was run as a validation study which enabled a broad range of assessments to be given so that they could later be refined to provide BT with bespoke assessments which only assess those qualities proven to impact on BT sales employee performance. 30 scales of CareerHarmony's major new personality inventory were administered, along with an Interpersonal Conflict Coping Inventory, and 3 ability tests. As expected there was a minor variation in which tests were found to be most predictive for each job group.
At least two of the three cognitive tests predicted performance for each job family, and the emphasis of the personality scales differed for the various job groups. An overall profile of competencies was identified that predicts good performance for the entire population assessed. This includes the three cognitive assessments, and 11 personality scales (Adaptable, Ambitious, Communicative, Dependable, Decisive, Emotionally Stable, Leadership, Problem Solving, Resourceful, Self Confident, and Self Developing).
When further assessments are administered it is recommended that a bespoke assessment session is created that contains just these tests. The validation exercise has shown that we can use a single BT norm across all job groups. This assessment can be created so that it provides a score for each job group, enabling managers to see the relative fit of an employee or candidate to each position. Next steps A higher level of validity was consistently achieved for the managers' ratings than for the DPR score.
This could highlight a weakness in the DPR process and should be investigated. The personality scales were not found to be as predictive for the Sales Farmer group. This could also be caused by a problem with the DPR process in this job group, or potentially additional assessments may be required for this group. This should be investigated. To provide BT with a better understanding of the desired profile of their future sales force we need to be sure we are looking at a representative sample of the existing BT sales force.
We would therefore recommend widening the validation exercise to establish whether the results found for the UK based Major Business team within the Commercial & Brands business unit are representative of other business units in the UK and abroad. Now we have an understanding of the profile of good performers in the Major Business team of the Commercial & Brands business unit we recommend starting the process of identifying how the employee profiles need to differ to ensure they can also be successful in the future.
Once we have a clear understanding of the existing and future good performers, then we can map the development and recruitment processes required to ensure BT successfully adapts the employee profile that it has now, to the one it needs in the future. We propose a meeting in January at which we can discuss these issues, and the synergies with the RightSkills project further. Background and Objectives Fit for Purpose is an internal BT project, aimed at preparing and planning for the anticipated changes in the business aims, strategies, operation and structure in future years.
Manpower has been engaged with the Fit for Purpose team in order to develop an assessment tool that can measure current employees' personality and abilities, and be used to further their ultimate potential performance. The first stage of the project saw the development of a tailored Internet based assessment tool, by CareerHarmony, a Manpower company. The content of the assessment battery consisted of a selection of tests which would provide an overview of an individual's competencies including cognitive and personality aspects.
It was decided to pilot and validate the assessment battery using data collected from employees of the BT Commercial and Brands business unit. The aims of the pilot were to tailor the assessment tools' norms to fit BT's employee population, as well as to optimize the tools' ability to predict successful job performance. An additional aim of the pilot was to study the differences between the different job families within Major Business. Specifically, the pilot aimed to check whether differential norms are required, and whether different combination of competencies assessment provide better predictors for the different job families.
This document presents the results of the pilot and validation study. 1. DPR Score: this score is a summary of an individual's achievements against their job description requirements and individual objectives. Each employee's DPR is discussed and agreed upon in a one to one meeting with their manager. The score is a simple ranking on a 9 point scale, in which a lower score denotes a higher performance. 2. Managers' Ratings: managers were asked to rate each employee on a list of scales, related to their job performance in comparison to other similar employees. These ratings were then integrated into one overall average score. Results
An initial analysis of the results included a process of choosing the most reliable and valid tests for the BT population. As a result, it was decided to include 19 of the total 30 scales of the JPPI, and the 3 ability tests. None of the Interpersonal Coping Style scales were found to be valid in this case, and were therefore not included in further analyses. The scales chosen were found to be more reliable and valid for the BT sales sample. These 19 scales were then included in the advanced stage of the data analysis. The technical procedures and analyses leading to the choice of the final JPPI scales are described in Appendix A.
The resulting norms and validity indices are described below. System Norms An initial aim of the study was to establish a norm-base for the BT population, while checking whether there is a need to establish differential norms for the different job families, or whether one norm group can be used across the BT sales community. To test the differences between the results of the different job families, one-way ANOVAs (analysis of variance) were conducted with the scales scores as dependent variables and the job category as a dependent variable.
Post hoc comparisons revealed slight differences between most scale means, whereby the Hunters were characterized by higher means than the Engineers. Other differences were less salient or consistent. Overall, the differences between the means were not meaningful, and therefore do not justify implementing individually set norms for these groups. The norms for the total sample are presented below. These have been implemented into the CareerHarmony system, and the individual reports that will be produced for each employee will be based on scores calculated by using these norms. Norms per job group can be found in Appendix B.
System Validity The aim of this stage was to design a composite score that would maximize the predictive power of the assessments. In this stage of the analyses, both the total sample and each job group were analyzed. An additional aim was to establish whether different combinations of assessment tools would provide better predictions for the different job families. Linear regression analyses were used in order to design predictor composites, which would include both the relevant of the 19 JPPI scales and the 3 cognitive ability tests: Graphs and Tables Comprehension, Planning and Organizing, and Verbal Reasoning.
(Please note that as mentioned above, based on detailed data analyses, 19 of the total 30 scales were found to be more reliable and valid for the BT sales sample). In order to choose the most relevant scales to be entered into the regression analyses, certain scales were identified for each job group by choosing the scales that showed the highest correlation with the criteria (for either Managers' Ratings or DPR scores). In the table below, the scales showing the highest correlation with the criteria are marked in green, and the scales showing a good level of correlation are marked in yellow.
All scales marked in green and yellow were entered into the regression analysis. The table below shows the validity indices of each of the 19 scales as well as those of the composite scores yielded by the linear regression (R). As shown in the table, a greater validity was achieved overall for the Managers' Ratings criterion (. 27) than for the DPR score (. 19), and a similar pattern was seen per job group, ranging from . 27-. 42 for the Managers' Ratings, and . 15-. 25 for the DPR score.
This exercise was run as a validation study which enabled a broad range of assessments to be given so that they could later be refined to provide BT with bespoke assessments which only assess those qualities proven to impact on BT sales employee performance. 30 scales of CareerHarmony's major new personality inventory were administered, along with an Interpersonal Conflict Coping Inventory, and 3 ability tests. As expected there was a minor variation in which tests were found to be most predictive for each job group.
At least two of the three cognitive tests predicted performance for each job family, and the emphasis of the personality scales differed for the various job groups. An overall profile of competencies was identified that predicts good performance for the entire population assessed. This includes the three cognitive assessments, and 11 personality scales (Adaptable, Ambitious, Communicative, Dependable, Decisive, Emotionally Stable, Leadership, Problem Solving, Resourceful, Self Confident, and Self Developing).
When further assessments are administered it is recommended that a bespoke assessment session is created that contains just these tests. The validation exercise has shown that we can use a single BT norm across all job groups. This assessment can be created so that it provides a score for each job group, enabling managers to see the relative fit of an employee or candidate to each position. Next steps A higher level of validity was consistently achieved for the managers' ratings than for the DPR score.
This could highlight a weakness in the DPR process and should be investigated. The personality scales were not found to be as predictive for the Sales Farmer group. This could also be caused by a problem with the DPR process in this job group, or potentially additional assessments may be required for this group. This should be investigated. To provide BT with a better understanding of the desired profile of their future sales force we need to be sure we are looking at a representative sample of the existing BT sales force.
We would therefore recommend widening the validation exercise to establish whether the results found for the UK based Major Business team within the Commercial & Brands business unit are representative of other business units in the UK and abroad. Now we have an understanding of the profile of good performers in the Major Business team of the Commercial & Brands business unit we recommend starting the process of identifying how the employee profiles need to differ to ensure they can also be successful in the future.
Once we have a clear understanding of the existing and future good performers, then we can map the development and recruitment processes required to ensure BT successfully adapts the employee profile that it has now, to the one it needs in the future. We propose a meeting in January at which we can discuss these issues, and the synergies with the RightSkills project further. Background and Objectives Fit for Purpose is an internal BT project, aimed at preparing and planning for the anticipated changes in the business aims, strategies, operation and structure in future years.
Manpower has been engaged with the Fit for Purpose team in order to develop an assessment tool that can measure current employees' personality and abilities, and be used to further their ultimate potential performance. The first stage of the project saw the development of a tailored Internet based assessment tool, by CareerHarmony, a Manpower company. The content of the assessment battery consisted of a selection of tests which would provide an overview of an individual's competencies including cognitive and personality aspects.
It was decided to pilot and validate the assessment battery using data collected from employees of the BT Commercial and Brands business unit. The aims of the pilot were to tailor the assessment tools' norms to fit BT's employee population, as well as to optimize the tools' ability to predict successful job performance. An additional aim of the pilot was to study the differences between the different job families within Major Business. Specifically, the pilot aimed to check whether differential norms are required, and whether different combination of competencies assessment provide better predictors for the different job families.
This document presents the results of the pilot and validation study. 1. DPR Score: this score is a summary of an individual's achievements against their job description requirements and individual objectives. Each employee's DPR is discussed and agreed upon in a one to one meeting with their manager. The score is a simple ranking on a 9 point scale, in which a lower score denotes a higher performance. 2. Managers' Ratings: managers were asked to rate each employee on a list of scales, related to their job performance in comparison to other similar employees. These ratings were then integrated into one overall average score. Results
An initial analysis of the results included a process of choosing the most reliable and valid tests for the BT population. As a result, it was decided to include 19 of the total 30 scales of the JPPI, and the 3 ability tests. None of the Interpersonal Coping Style scales were found to be valid in this case, and were therefore not included in further analyses. The scales chosen were found to be more reliable and valid for the BT sales sample. These 19 scales were then included in the advanced stage of the data analysis. The technical procedures and analyses leading to the choice of the final JPPI scales are described in Appendix A.
The resulting norms and validity indices are described below. System Norms An initial aim of the study was to establish a norm-base for the BT population, while checking whether there is a need to establish differential norms for the different job families, or whether one norm group can be used across the BT sales community. To test the differences between the results of the different job families, one-way ANOVAs (analysis of variance) were conducted with the scales scores as dependent variables and the job category as a dependent variable.
Post hoc comparisons revealed slight differences between most scale means, whereby the Hunters were characterized by higher means than the Engineers. Other differences were less salient or consistent. Overall, the differences between the means were not meaningful, and therefore do not justify implementing individually set norms for these groups. The norms for the total sample are presented below. These have been implemented into the CareerHarmony system, and the individual reports that will be produced for each employee will be based on scores calculated by using these norms. Norms per job group can be found in Appendix B.
System Validity The aim of this stage was to design a composite score that would maximize the predictive power of the assessments. In this stage of the analyses, both the total sample and each job group were analyzed. An additional aim was to establish whether different combinations of assessment tools would provide better predictions for the different job families. Linear regression analyses were used in order to design predictor composites, which would include both the relevant of the 19 JPPI scales and the 3 cognitive ability tests: Graphs and Tables Comprehension, Planning and Organizing, and Verbal Reasoning.
(Please note that as mentioned above, based on detailed data analyses, 19 of the total 30 scales were found to be more reliable and valid for the BT sales sample). In order to choose the most relevant scales to be entered into the regression analyses, certain scales were identified for each job group by choosing the scales that showed the highest correlation with the criteria (for either Managers' Ratings or DPR scores). In the table below, the scales showing the highest correlation with the criteria are marked in green, and the scales showing a good level of correlation are marked in yellow.
All scales marked in green and yellow were entered into the regression analysis. The table below shows the validity indices of each of the 19 scales as well as those of the composite scores yielded by the linear regression (R). As shown in the table, a greater validity was achieved overall for the Managers' Ratings criterion (. 27) than for the DPR score (. 19), and a similar pattern was seen per job group, ranging from . 27-. 42 for the Managers' Ratings, and . 15-. 25 for the DPR score.
Comments
Post a Comment