Joint-service battery

The story of the development of a joint-service selection and classification instrument, the Armed Services Vocational Aptitude Battery (ASVAB), is in some respects a cautionary one. Ultimately, the outcome was a battery that has persevered in various forms for roughly 50 years. Along the way, however, the developmental process encountered severe problems that proved disruptive and embarrassing for each of the military services.

The development of the very first iterations of the ASVAB proceeded without notable controversy. In 1966, the Assistant Secretary of Defense for Manpower and Reserve Affairs directed that a common aptitude battery be developed that all services could use for high school testing. Content areas of the existing Army, Navy, and Air Force classification tests that were determined to be interchangeable (e.g., word knowledge) across the services were identified, and items were drawn from each of the service tests in that content area to form a new subtest. New content was added as needed. Appropriate analyses were conducted to ensure the new battery met scientific and practical standards (Bayroff & Fuchs, 1970). The first version of the ASVAB was administered from 1968 to 1973 to high school students for vocational counseling and recruiting purposes. Two parallel forms of a second version were administered from 1973 to 1975 (Seeley, Fischl, & Hicks, 1978). One form was used for high school testing and the other for Air Force and Marine Corps operational screening (ASVAB Working Group, 1980).

Armed Services Vocational Aptitude Battery misnorming

The problems arose with the third iteration of the ASVAB. In 1974, the Assistant Secretary of Defense for Manpower and Reserve Affairs directed that the ASVAB be used as a common instrument for all the services—not just for high school students, as before, but now for operational enlisted selection and classification as well (ASVAB Working Group, 1980). Along with this decision came three immediate implications for the next iteration of the ASVAB: 1) as a determinant of selection and classification decisions for all the services, it would be more consequential than previous versions; 2) there was a perceived need to implement the new forms quickly, in part because the single form then used for operational screening was viewed as inadequate by the Marine Corps and in part by a desire by the Department of Defense to make the transition to joint testing expeditiously (ASVAB Working Group, 1980); and 3) for the first time, with the draft having ended in 1972 (Rostker, 2006), the test would be developed in an all-volunteer environment.

An ambitious schedule for fielding the new ASVAB collided with numerous delays. In the end, this meant little time was available for determining how the raw scores on the ASVAB could be interpreted in making selection and classification decisions (ASVAB Working Group, 1980). This involved an estimation of how individuals in general would score on this test, not just those individuals tested at this particular time and place. To make this estimation, scores on the ASVAB were linked to another test, a reference test, already calibrated against a representative population. However, later analyses suggested that the linkage process was flawed in multiple respects (Maier & Truss, 1983). The test used for linkage in the Army was the one then being used for operational selection and classification decisions. Thus, examinees were motivated to perform higher on the reference test and recruiters were more likely to coach examinees how to perform higher on this test than would have been the case for the earlier representative population. Also, the reference test used for the Navy and Air Force was found to be incorrectly scored (Maier & Truss, 1983).

The new ASVAB was fielded in all services in 1976. Almost immediately, there were suspicions that the standards being applied with respect to the new forms were incorrect, but pinpointing the problem proved to be difficult (ASVAB Working Group, 1980). Finally, research was conducted that generated standards that the services could agree on, and tests based on these were implemented in 1980 (Maier & Grafton, 1981). By then, it had been determined that the scores used for selection and classification had been seriously inflated, leading to the enlistment of many more with marginal scores on the ASVAB than had been intended. To Army General Maxwell Thurman, who was “arguably the single most important person in the history of the All-Volunteer Force” (Rostker, 2006, p 387), the result was a “calamity” (Rostker, 2006, p. 398). The entire episode, known as the ASVAB misnorming, was described as a “tragedy of errors. A travesty of sound psychometric practices and common sense” (Laurence & Ramsberger, 1991, p. 72, quoted in Rostker, 2006, p. 481).

However, the ultimate impact of the misnorming was that sound psychometric practices and common sense came back to the fore as never before. A Defense Advisory Committee (DAC) on Military Personnel Testing was established in 1981 composed of outside experts to oversee the process of updating military test practices. The DAC and technical and policy military groups met multiple times each year to provide oversight to the enlistment testing program (Maier, 1993).

Computerized Armed Services Vocational Aptitude Battery

As enlisted selection and classification was becoming a joint-service enterprise, a research program was initiated to generate a computerized version of the ASVAB (Wiskoff, 1997). Computerization offered some benefits in terms of efficiency, but more importantly it could permit tailoring test items to an applicant's ability (Sellman & Arabian, 1997). An item answered correctly could trigger the administration of a more difficult item; an item answered incorrectly could trigger the administration of an easier item. Each item would provide a more precise estimate of the applicant's true ability level. Time would not be wasted on the administration of items far above or below that level (Sands & Waters, 1997).

Interest in a computer adaptive test heightened as a result of the misnorming. The computer adaptive technology was viewed as a partial antidote to the coaching and scoring issues that emerged during this crisis (Wiskoff, 1997). It is difficult to coach a candidate on the correct answers to a test that has no set sequence of items and presents different items to different applicants. Also, computer scoring would reduce the likelihood of human error.

One lesson that apparently had not been learned from the ASVAB misnorming was that pressure to meet an ambitious schedule may have unforeseen consequences. After preliminary research had been completed in 1982, the Navy was identified as the lead service, and completion of the project was scheduled for 1985 (Wiskoff, 1997). However, it soon became apparent that this schedule could not be met. Available computer hardware and technical capabilities were not adequate for the requirements of full-scale enlisted testing in multiple sites across the nation (McBride, 1997). Funding and management issues (Martin & Hoshaw, 1997) also emerged. Finally, in 1993, implementation of CAT-ASVAB was approved (Martin & Hoshaw, 1997). While the development time was much longer than anticipated, the services, under the Navy's lead, persevered to overcome enormous technical and management problems.

Job Performance Measurement Project

As the misnorming revealed that the services were unintentionally not meeting the standards they had set for themselves, the question arose of what importance these standards actually had. What did it mean for someone to receive a low score on the ASVAB? This was a question Congress was very much interested in, and it laid the groundwork for what was to become known as the Job Performance Measurement Project (JPM). Each of the services was directed to conduct research to determine what the link between ASVAB enlistment standards and job performance was (Knapp, 2006).

The Department of Defense determined that the gold standard for performance assessment was hands-on performance. Each service was to include a measure of hands-on performance in its validation strategy. However, beyond this, the services diverged in terms of other criteria they employed (Knapp, 2006). The most ambitious strategy was the one used by the Army. It was driven by the philosophy that no single type of measure truly captured the complexity of performance. Hands-on performance reflected the maximum capability of an individual on certain well-defined tasks. Job knowledge was another measure of capability. However, ratings reflected performance over a longer period of time, and were more indicative of an individual's motivation to perform, even when not being tested. Similarly, administrative measures, such as awards or disciplinary history, provided a view of performance that could not be completely represented by a task-oriented proficiency measure (each of these types of measures was described in Knapp, Campbell, Borman, Pulakos, & Hanson, 2001).

The Army's research effort, known as Project A, was conducted over a 13-year period, from 1982 to 1995, and completely revamped conceptions of job performance and individual differences related to such performance. These individual differences were measured not just by ASVAB, a cognitive measure, but also by measures of personality, vocational interests, spatial ability, and psychomotor ability (Russell & Peterson, 2001). For the first time since the early 1970s, there was active service interest in using a broader perspective than cognitive ability alone to guide selection and classification decisions.

This multidimensional approach to both performance and the predictors of performance proved highly illuminating. It generated results that revealed that one could not simply ask whether a measure predicted performance, but rather one needed to ask the question in terms of what type of performance was being predicted. The combination of hands-on and job knowledge yielded one type of performance, sometimes referred to as can-do or task performance. Ratings that consisted of dimensions that reflected behavior that essentially transcended specific task performance, combined with administrative measures of behavior, yielded another category, which could be labeled will-do performance (Rumsey, Peterson, Oppler, & Campbell, 1996).

To those vested in the credibility of ASVAB, the results were a relief. ASVAB was a strong predictor of can-do performance. To those who believed that a broader combination of predictors was needed for selection, there was also support. ASVAB did not predict will-do performance as well as it predicted can-do performance, but personality and interests showed promise in being able to add to the ASVAB in predicting this type of behavior (McHenry, Hough, Toquam, Hanson, & Ashworth, 1990). In combination with the results from the other services participating in the JPM (Sellman, Russell, & Strickland, 2017), the prevailing conclusion was that ASVAB was doing the job intended for it, predicting technical performance, but that the selection system could likely be improved by the addition of noncognitive individual difference measures.

Armed Services Vocational Aptitude Battery today

Since 1976, the ASVAB has been a dual-use instrument. Its core is a combination of verbal and quantitative tests used for selection. These tests and more specialized technical tests are combined in various ways by the different services for job placement decisions. The ASVAB's utility for selection has been well demonstrated. Recently, its utility for classification has become somewhat controversial. Because it consists of cognitive tests, and such tests have been found to be highly correlated with one another, some have concluded (e.g., Murphy & Davidshofer, 1991) that the different ASVAB tests measure essentially the same thing: general cognitive ability. However, other scientists have conducted simulations showing that, even with high correlations between different ASVAB components, substantial classification utility is still possible (e.g., Zeidner & Johnson, 1994).

Personality measures

Project A stimulated interest in measurement of individual differences that could predict will-do performance, particularly personality variables. Despite this interest, progress toward including noncognitive variables in selection moved slowly at first. Early efforts to introduce a joint-service personality measure were unsuccessful due to concerns about faking (Rumsey, 2012). Faking was indeed a serious issue. If an individual responded to a personality item in terms of what he or she thought would impress the examiner, then the test was not measuring what it was intended to measure.

For a number of years, researchers labored somewhat inconspicuously to overcome this obstacle. Two researchers in particular, Leonard White and Mark Young, at the Army Research Institute for Behavioral and Social Sciences, explored building an instrument using a forced choice methodology. The idea was to disguise each item so it would not be clear what response would put the examinee in the most favorable light. The result of their endeavors was a measure known as the Assessment of Individual Motivation (AIM) (White & Young, 1998). Their work came to sudden prominence when the Army needed a new approach to the screening of those without high school diplomas, whose proclivity towards attrition made them poor risks for the Army. Research indicated that the AIM could be useful in identifying those non–high school diploma graduates who did not present the same level of risk (White, Rumsey, Mullins, Nye, & LaPort, 2014). The AIM was implemented, but early results were discouraging (White, Young, Hunter, & Rumsey, 2008). The researchers persevered, identified and replaced problematic items, and a revised version of the AIM proved to be a successful component of a new screening system directed specifically at nongraduates (White et al., 2014).

Following this success, further developments in personality testing proceeded with new momentum. Another group of researchers working in the Drasgow Consulting Group (DCG) developed a new personality test technology that could be applied to an entire applicant pool. A paper-and-pencil instrument with limited items could, with repeated and frequent use, be particularly vulnerable to compromise. The DCG technology streamlined the item development process to facilitate production of a massive number of items. Further, it employed a computer adaptive process (Stark et al., 2014). The ASVAB had already been converted to a computer adaptive format, but applying this approach to a personality measure was something new. The basic idea is the same, however: approach an individual's true position on a scale by iteratively selecting the items administered to that individual based on his or her responses to previous items until the individual's position is identified with sufficient accuracy.

The combination of an enormous item pool and the lack of a predetermined ordered list of items made the new adaptive tests highly fake resistant (Stark et al., 2014). Also, as noted above, with a computer adaptive approach, items could be limited to those that provided the most information about the individual on the attribute in question. Thus, a computer adaptive test could be administered in much less time than a traditional paper-and-pencil test. For these reasons, this approach was highly desirable for screening large numbers of applicants in a limited period of time.

Both the Navy and the Army have had success implementing computer adaptive personality measures. The Navy's version, the Navy Computer Adaptive Personality Scales (NCAPS), was introduced as a tool used in selection for Special Operations training (Stark et al., 2014). The Army version, the Tailored Adaptive Personality Assessment System (TAPAS), was provisionally implemented as a supplement to the ASVAB in enlisted selection. It replaced the AIM as a tool for identifying non–high school diploma graduates who were acceptable attrition risks (White et al., 2014) and was also used as a screen for high school diploma graduates whose ASVAB scores put them in jeopardy of missing the cut (Knapp, Heffner, & White, 2011).

TAPAS has been found to predict performance in training (White et al., 2014) and on the job (Kirkendall, Bynum, Kennedy, & Hughes, in press), as well as attrition (Hughes, O'Brien, Reeder, & Purl, in press; Kirkendall, Nye et al., in press). Both the Army (Wolters, Heffner, & Sams, 2015) and Air Force (Trent, Barron, Rose, & Carreta, in press) have found it useful for predicting counterproductive work behaviors. The Air Force has conducted research demonstrating its resistance to faking and its stability over time. Navy recruits have also participated in TAPAS research (Stark et al., 2014).

Vocational interest measures

The new noncognitive advances in military testing have also extended to vocational interest measurement. An early version of the ASVAB included interest items carried over from the Army Classification Battery, but the interest measures were subsequently dropped “due to low predictive validity” (Greenston, 2012). Each of the services has been conducting sporadic interest measurement research since 1949 (Ingerick & Rumsey, 2014) but with little to show for it in terms of operational success up to the turn of the century. Since then, however, such research has gained momentum. The Navy has, having identified key job characteristics as a basis for determining applicant interest in particular Navy jobs (Watson, in press), developed an interest measure that of this writing is scheduled for implementation for all Navy enlisted accessions in October 2018. The Air Force has developed a tool based on the Navy's work called the Air Force Work Interest Navigator (AF-WIN; Johnson, Romay, & Barron, in press). The Army has built upon past work to identify a new instrument covering 20 basic interest categories (Kirkendall, Nye et al., in press).

The controversy over the ASVAB's classification potential suggests the need for further examination of how much can be gained through optimal combinations of cognitive measures for job placement. Since, as shown in Project A (Peterson et al., 1992), cognitive measures and noncognitive measures are not highly correlated, it also suggests that incorporating noncognitive measures into the classification process may be particularly helpful. Preliminary research provides some support for the potential value of TAPAS as a classification tool (Nye et al., in press).