LANGUAGE VALIDATION REPORT

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP78-06215A000300040014-7
• Release Decision: RIPPUB
• Original Classification: S
• Document Page Count: 15
• Document Creation Date: November 17, 2016
• Document Release Date: June 15, 2000
• Sequence Number: 14
• Publication Date: October 3, 1969
• Content Type: MF

Approved For Release 20 1c T
1 CIA-RDP78-06215A000300~
MEMORANDUM FORE Chief, Language School
THROUGH Director of Medical Services
Director of Training
SIFdJM CT Lauiguage Validation Report
1. Attached PSS/UMS Research Report describes a recent
validation study of the Artificial Language Test, a general
test of aptitude for learning a foreign language. Agency
employees who were in language training for some period
during FY68-69 and on whom aptitude and objective achieve-
ment ratings were available constituted the group studied.
2. The finding, in brief, was that a significant degree
of relationship does exist between the Artificial Language
Test and training achievement. However, the size of the
relationship is not so large as to recommend the exclusive use
of the test in decisions for which the potential for language
acquisition is a major factor.
3. The Psychological Services staff appreciated the
opportunity to collaborate with the Language School in this
research approach to a problem of mutual concern, and we want
to thank the mea-nbers of the Language School Staff who contrib-
uted so substantially to the conduct of the study.
25X1AV
Acting Chief, Psychological Services Staff
Office of medical Services
Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
 Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
25X1A
Psychological Services Staff
Office of Medical Services
September, 1969
S-E-C-R-E-T
Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
 ? Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
Introduction
This research report describes a recent validation study
of the'Artificial Language Test (AL-AGO), a general test of
aptitude for learning a foreign language. The test is presently
included in the morning portion of PATB, the battery of tests
given by the Psychological Services Staff to applicants for
professional positions in the Agency. AL-AGO has been part of
PATB since December 1956.
Between February 1952, and October 1965, the AL-AGO had
been used in combination with other measures of language aptitude
to predict success in Agency language courses. In the inter-
vening years a number of in-house validity studies had been made
on all of these language aptitude tests. Typically, it was
found that AL-AGO provided the best single predictor of achieve-
ment in Agency language training. Reports of validity coeffi=
cients in the .40's and .50's were not at all unusual. Review
of this early research literature suggested that AL-AGO served
as a highly satisfactory aptitude measure. In light of the
very high correlations - in the .70's - between AL-AGO and
overall Foreign Service Institute (MLAT) ratings, it is not
surprising that AL-AGO should have had the degree of relation-
ship it did with achievement in language training. Recently a
State Department memorandum reported a correlation in the .60's
between MLAT ratings and language course evaluations for a
group of 691 trainees.
Approved' For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
S-E-C-R-E-T
 . Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
As of FY70 all candidates for Agency language training
have been required to take the full MLAT. Consequently, AL-AGO
is now treated as only an initial and approximate measure of
an applicant's general language learning aptitude. It is felt
by the Language School that the multi-score MLAT is the more
appropriate source of information for differential placement,
training emphases, student management, and selection in special
cases.
Objectives
Good psychometric practice recommends that aptitude tests
be periodically revalidated. This is especially true if changes
have occurred in the student population, the manner in which
training is conducted, course objectives, evaluations, etc.
With this in mind, PSS in cooperation with the Language School
undertook a study of the relationship between AL-AGO ratings
and rate of achievement in language training. A secondary
objective was to compare the distribution of AL-AGO ratings
of a representative group of language trainees with that of
a professional applicant sample. Results of this comparison
would indicate if these aptitude ratings had been systematically
used for screening candidates for language training.
Procedure
The. starting point of this study.was the preparation of
a listing of all individuals who studied a foreign language
for some period of time during FY68 or 69 and on whom at least
Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
S-E-C-R-E-T
 Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7 3
S-E-C-R-E-T
some language evaluations were recorded. FY68 and 69 were
selected because language evaluations from these years were
reasonably accessible and were administered in a relatively
standardized way. Prior to these years, there was some question
as to the format followed and the degree of standardization in
assigning language evaluations.
The listing of students was organized by fiscal year within
each language. For each individual the following pieces of
information were recorded:
Bases of evaluation:
Student proficiency levels both at the outset of the
training and at its termination were based on either the
instructor's estimate or on an "official" test. A coding
procedure was devised to differentiate between initial and
final rating bases. It was felt that relying on an
official test for determining proficiency level would
minimize the amount of subjectivity, and probably the
error variance, in the ratings. Hence, only those
final ratings based on an official test were subjected
to analysis. For two reasons this restriction was not
applied to entry level proficiency ratings. First, doing
so would have cut the study sample down to too small a
size. Secondly, it was felt that where a student had no
previous knowledge of the language he was to study (this
was very typically the case) his initial level rating would
not differ whether assigned by the instructor's estimate
or on the basis of a test.
Language elements rated:
The Language School training evaluation form typically
recorded student achievement in four elements of the
language studied: reading, speaking, pronunciation, and
understanding. For the purposes of this study, only the
reading and speaking elements were considered. The pro-
nunciation element was omitted on the assumption that
it had minimal rational relationship to the design and
intent of the AL test. The understanding element was
omitted on the assumption that the requirements for
displaying competency on a written test, like AL, and the
aural aspect of understanding the spoken words were quite
different. The reading element was included because of. the
written nature of AL, and speaking because of the similarity
Approved For Release 2000/08 :-Ctk1 DP78-06215A000300040014-7
 Approved For Release 2000/08/(4 - CYARRPj8-06215A000300040014-7 4
between actual speaking and the reconstruction of words
and forms required by the AL test.
Languages:
Twenty languages were represented in the overall listing
of students who had been in language training some time
during FY68-69. The distribution of students in these
different languages is shown in Table I. It can be seen
that the majority of the languages studied, 13 in fact,
had fewer than ten enrollees; only seven languages had
more than ten students who received final proficiency
levels based on an official test.
On the advice of the Language School, ratings of students
in Vietnamese and Thai were treated together. This was
true for German and Swedish ratings also. Students of
Russian--there were only 17 who had received their final
proficiency level by official test--were not included in
our analyses because many were in courses designed to
familiarize them with the Cyrillian alphabet for filing
purposes. Hence, ratings in reading and speaking were
.not altogether appropriate criteria for this group.
Since. knowledge of a student's final proficiency level
without knowledge of his entry level did not give any
indication of his progress in language training, the
basic datum used in this study consisted of the gain or
final minus initial level registered by each student.
During the period from which language evaluations were
drawn for this study, minor changes were made in the rating
of student achievement. Specifically, "pluses" indicating
particularly noteworthy achievement at a given level were
assigned to the five basic proficiency levels (No Proficiency,
Slight, etc.). Since 1 February 1969 pluses were given in
all course elements. Between 14 November 1968 and 1 February
1969 students could receive pluses in all elements except
reading and prior to 14 November 1968 pluses were given
only in the speaking element of the language course. To
maintain uniformity in assessing student achievement and
greatly simplify the data analysis, we have chosen to
disregard plus marks for this study.
Hours in Study:
For each student both nominal and actual hours of language
training were recorded. Nominal hours are the number of
hours a student was assigned to the Language School. Actual
hours are the number of hours spent in actual study. For
obvious reasons, actual hours in combination with gain
were used exclusively in describing a student's achievement.
Approved For Release 20004 : ilA-RDP78-06215A000300040014-7
 Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
DISTRIBUTION OF STUDENTS1 BY LANGUAGE STUDIED AND BY BASIS OF ASSIGNMENT
OF INITIAL AND FINAL PROFICIENCY LEVELS
z
BASIS OF RATING
1=Initial; 2=Final
1.
Instructor's
Estimate (I.E.)
2.
Official
Test
(O.T.)
0
1.
O.T.
2.
O.T.
0
2.
I.E.
0
1 .
O . T .
2.
I.E.
Cl] co z z u.
H w x U z x Z H w x C7 H r.~ I
W z U z w 0 - ul H E-+ z
x N W 0 < a a C a
w x z E Ei 50,
FC U U W C7 C7 H H 7 W W W 0.'i C/)
3 7 8 3 1 1 6 1 14 3
8 0 14 8 0 0 1 2 0 0 1 0 3 0
2 0 3 3 4 0 0 0 3 0 1 0 36 0
u u U 0 4 0
TOTALS 0 18 2 77 34 7 7 9 5 4 1 8 1 57 3
x
t0
z
a'
Cn
x
CA
0
w
(Q
H
<
E-~
x
Cn
H
x
P
38
2
9
5
22
213
14
1
1
1
1
55
4
0
1
0
1
58
3
0
0
0
0
10
59
3
11
6
34
336
1
Only students on whom AL scores and both initial and final speaking proficiency levels were available
are included in this breakdown.
S-E-C-R-E-T
Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
 5
Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7
AL ratings:
Adjectival ratings and actual raw scores on the AL test
were recorded where available. Raw scores on AL?for the
study sample ranged from 5 to 57---59 is a perfect
score. Some checks were made to determine if using raw
scores rather than the five adjectival ratings would lead
to greater predictability of rate of achievement. There
was no evidence that it did so, and hence the simpler
adjectival ratings were used in the analyses described
below.
Although 42'3 individuals were identified as having studied
a language during FY68-69, complete data were available on only
351 cases. About 250 of these individuals had received their final
proficiency levels by official test and hence were eligible for
inclusion in our subsequent analyses.
Results
1. DISTRIBUTION OF AL RATINGS
In the first part. of this result section, we compare Agency
language trainees and applicants on performance on the AL test.
If the AL test has in the past systematically played a part
in screening candidates for language training, we would expect
that proportionately more of the trainees than applicants had
received ratings of Good and Very Good and fewer Poors and Very
Poors. Marked restrictions in the distribution of AL ratings
in the validation sample arising from pre-selection on AL would
spuriously lower any relationship between aptitude and achieve-
ment ratings.
To check for restriction of range, the distribution of
AL ratings of 3,282 men and women applicants tested in FY67
for professional positions was compared with that of the 351
language trainees. We had no reason to believe that these
Approved For Release 2000108/04: CIA-RDP78-06215A000300040014-7
S-E-C-R-E-T
 Approved For Release 2000/08/04: CIA-RDP78-06215A000300040014-7,
S-E-C-R-E-T
two groups differed, on the average, in age and educational
level attained. The percent of each group receiving the
various aptitute ratings is shown below.
AL RATING
POOR, GOOD,
VERY POOR AVERAGE VERY GOOD
Professional Applicants
Language Trainees
31
25
37
38
32
37
x2=5.94, df=2
P>05