EDUCATIONAL TESTING SERVICE PRINCIPLES, POLICIES AND PROCEDURAL GUIDELINES REGARDING ETS PRODUCTS AND SERVICES

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP83-00156R000300050003-7
• Release Decision: RIPPUB
• Original Classification: K
• Document Page Count: 17
• Document Creation Date: December 19, 2016
• Document Release Date: December 20, 2006
• Sequence Number: 3
• Publication Date: February 1, 1979
• Content Type: REPORT

Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
Educational Testing Service
Principles, Policies and
Procedural Guidelines
Regarding ETS Products and Services
February 1, 1979
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
Accountability . . . . . . . . . . . . . . . . . 1
Confidentiality of Data . . . . . . . . . . . . . . 7
Product Accuracy and Timeliness . . . . . . . . . . 11
Research and Development . . . . . . . . . . . . . . 15
Tests and Measurement . . . . . . . . . . . . . . . . 21
Technical Quality of Tests . . . . . . . . . . . . . 21
1. Test Development . . . . . . . . . . . . . 23
2. Test Administration . . . . . . . . . . . . 28
3. Test Reliability . . . . . . . . . . . . . . 32
4. Scale Definition . . . . . . . . . . . . . . 34
5. Equating . . . . . . . . . . . . . . . . 38
6. Score Interpretation . . . . . . . . . . . 40
7. Test Validity . . . . . . . . . . . . . . . 44
Test Use . . . . . . . . . . . . . . . . . . . . . 47
Technical Assistance, Advice, and Instruction .
Glossary . . . . . . . . . . . . . . . . . . . . . .
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
Educational Testing Service (ETS) is a private, nonprofit,
educational organization with primary involvement in the
areas of measurement and research. In collaboration with
a wide variety of institutions and agencies, ETS provides
programs, research and services for the identification
and recognition of individual talents and seeks to
contribute to the expansion of opportunities available
to all individuals.
ETS recognizes its responsibility to the individuals and
institutions that it serves directly and to the larger
society that ultimately must judge.the value and efficacy
of its work. This responsibility is implicit in the
educational character under which ETS operates and is
reflected in the products and services offered by the
organization since its inception in 1947. Periodic and
intense self-examination has been one means by which ETS
has attempted to assure its continuing commitment to the
broad constituency which it serves. As a step in that
continuing process, ETS has developed these Principles,
Policies and Procedural Guidelines Regarding ETS Products
and Services (Guidelines), first published on August 1,
1977.
The Guidelines are designed to ensure that ETS products
conform to uniformly high standards with respect to seven
areas of basic importance: Accountability, Confidentiality
of Data, Product Accuracy and Timeliness, Research and
Development, Tests and Measurement, Test Use, and Technical
Assistance, Advice and Instruction. The first three
sections of the Guidelines deal with issues that relate to
all ETS activities: Accountability, the responsibilities
of ETS to those affected by its activities; Confidentiality
of Data, the rights to and limitations on access to data
collected by ETS; and Product Accuracy and Timeliness, the
control of quality and performance according to scheduled
commitments.
The remaining sections concern issues specifically relating
to ETS' main endeavors: Research and Development, Tests and
Measurement, Test Use, and Technical Assistance, Advice and
Instruction.
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
The Guidelines attempt to codify standards used in various
ETS programs and services that should be considered for more
general application and more formal articulation. ETS has
not developed these Guidelines because of any legal
requirements. They were not devised in response to any
standards published by professional organizations, although
ETS endorses the goals served by those efforts. These
Guidelines are the result of an extensive deliberative process,
which began at my direction in 1974. More than 150 members of
the staff have participated in the initial development of the
Guidelines and their subsequent review and revision. The
Guidelines are drawn from the particular circumstances and
needs at ETS and are designed to reflect its institutional
objectives. Because of their origin and purposes, the
Guidelines cannot generally or usefully be applied to
organizations whose practices, programs or services differ
from those of ETS.
The Guidelines include three types of statements, which
have varying degrees of generality and significance. The
principle that should underline ETS efforts in any given
area is set first; policies to govern decision-making, and
designed to foster more specific goals, are set forth next;
and procedures relative to the conduct of specific ETS
activities are last described. However, because the
guidelines must apply to a broad diversity of programs and
services with differing characteristics, each procedural
guideline represents only one method of achieving the
objectives stated in the policies.
Even the principles and policy statements might reasonably
be expected to change in focus or coverage, as scientific
developments or practical experience change the nature
and desirability of various goals. Thus, the Guidelines
are intended to encourage and not deter change and
improvement. They should foster the development, exploration
and use of alternative approaches that hold to high standards
and serve important ETS policies.
ETS believes that the Guidelines can contribute significantly
to the quality and utility of ETS products and services. The
Guidelines have been distributed to executive, managerial and
professional staff at ETS so that they may be applied in the
course of their work at ETS. ETS does not have complete
responsibility or authority, of course, to determine how the
Guidelines will be implemented in ETS programs for which
policy is substantially established by a sponsoring group
other than ETS. ETS has, however, taken steps to encourage
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
and assist those groups to implement the Guidelines as
their activities relate to ETS. New activities ETS enters
into will be held to this same expectation.
During the first year after their introduction, the Guidelines
existed in provisional form. They were used in evaluating
all ETS programs and services, and, where a substantial lack
of compliance was found, steps were taken to achieve compliance
within a reasonable time, or to revise particular provisions
to reflect more accurately the diversity and practical demands
of ETS operations. This process of review will continue as part
of our organization-wide quality assurance effort and a report
will be given to the Board of Trustees at regular intervals. We
are committed to the effort to apply a set of positive criteria
to our work and, taking the present document as a starting point,
we will go forward with the continuing process of improving
those criteria and consequently the programs and services ETS
provides.
February 1, 1979
William W. Turnbull
President
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
ACCOUNTABILITY
Principle
ETS acknowledges responsibility for the
effective stewardship of its resources
to the New York Board of Regents which
has issued its corporate charter; to the
governing boards that sponsor and set policy
for programs or services in which ETS
products or services are used; to the
individuals and committees that advise ETS
with respect to appropriate policy for its
programs; to the institutions and agencies
that use ETS products and services; to
persons who take ETS tests (and parents or
guardians of minor persons), submit data
for use by ETS or for distribution to
others, or participate in research and
development projects conducted by ETS; and
to the professional associations that
are concerned with educational and psycho-
logical measurement and research.
Policies
A. ETS will furnish appropriate information to those to whom
it is responsible so they may make informed, independent
judgments as to the effectiveness with which ETS exercises
its stewardship.
B. ETS will seek, consider and, as appropriate, act on the
views of those who sponsor, use or are affected by ETS
programs and services.
C. ETS will seek to obtain advice on its activities and
policies from qualified men and women who are not
employed or retained on a regular basis by ETS and
who are drawn from appropriate professional disciplines,
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
major philosophies and points of view, different geographic
regions, and the major ethnic groups within the relevant
population.
D. ETS will support the activities of professional associations
with respect to developing and implementing professional
standards or codes, making available the results of current
work, and fostering peer review of its activities.
Procedural Guidelines
1. Information should be provided to sponsoring organizations with
which ETS has contractual relationships in a form that permits
evaluation of ETS services in terms of:
a) quality;
b) timeliness;
c) costs; and
d) responsiveness to legitimate comments or
criticisms.
2. Procedures should be established to facilitate communication
with sponsors by:
a) meeting at least annually to provide information
and to receive comments on matters affecting
the operations with which they are concerned;
b) defining a mutually agreeable process to be
used to transmit comments from sponsors or
others and a time period within which the
evaluation of comments will be completed
and reports of actions to be taken by ETS
can be expected.
c) making available periodic opportunities for
sponsors to express opinions, judgments and
counsel concerning their activities or programs
directly to ETS officers not normally responsible
for such activities or programs.
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
3. Procedures should be established for making available technical
and other information about ETS products and services to users
so that they may evaluate the appropriate use of the product
or service and communicate comments or criticisms to ETS.
4. Procedures should be established to communicate with or provide
information to persons who use or take ETS tests, who submit
data for use by ETS or for distribution to others, or who
participate in research and development projects conducted by
ETS. This information should be communicated by ETS or the
sponsor in such a way that these persons may understand their
participation with respect to:
a) the identity and scope of the sponsor's
responsibility;
b) the nature of the product, service or
research by which they are affected;
c) the way in which the product, service or
research will likely be used by educational
institutions or others; and
d) the channels that have been established for
addressing comments or criticisms to ETS
or to the sponsor and response thereto.
5. Organizational and program financial information should be
recorded, processed and reported in accordance with generally
accepted accounting principles and under appropriate
safeguards to insure accuracy.
b. An annual report that provides information about organizational
activities and finances should be published by ETS on a regular
basis and made available to any person on request. Program
and project reports, including program financial information,
should be made available in a manner consistent with contractual
understandings.
 Approved For Rel
7. Requests for information that is not included in an existing
publication should be considered by the appropriate sponsor and
by ETS. If its disclosure is consistent with applicable law,
with ETS and sponsor policy, and with contractual obligations
governing confidential or proprietary information, the
information should be provided. If complying with a request
for information results in a cost to ETS or a sponsor or
affects the normal schedule of fulfilling ETS' responsibilities,
ETS may provide the requested information in a reasonable
period of time and at an appropriate price for the services
rendered. Procedures should be established, as appropriate,
to facilitate responses to these requests.
8. Changes in federal statutes, regulations and case law that
affect research and development, testing programs, or advisory
and instructional services should be monitored to insure that
ETS activities and operations are in compliance as relevant
federal laws or rules change. Changes in other statutes,
regulations and case law should be evaluated as appropriate for
the same purpose.
9. All proposed new ETS activities should be reviewed by counsel
for compliance with applicable federal law and state law as
appropriate. ETS officers and staff should direct the attention
of legal counsel to matters that might affect ETS compliance.
10. Advice should be sought, where appropriate, from men and
women drawn from diverse backgrounds, interests and experience
(e.g., appropriate professional disciplines, major philosophies
and points of view, various geographic regions, and major ethnic,
handicapped and other relevant subgroups of the population of
interest) who are qualified to make a contribution to the
direction and substance of ETS programs and who are not employed
or retained on a regular basis by ETS.
Approved For Release 2006/12/27: CIA-RDP83-00156 -
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
11. Individuals who become members of an ETS external advisory,
review or evaluation committee should be informed about the
results of the committee's work in a reasonable period of
time.
12. A reasonable accommodation should be made with respect to
the professional responsibilities of the staff in order to
permit staff members to attend professional meetings, to
contribute to the development of professional standards or
codes, to participate in and benefit from the dissemination of
information on subjects of professional interest and to stay
abreast of current concerns and accomplishments in related
fields.
13. ETS should have effective procedures for peer review whenever
that will contribute substantially to the quality of ETS
work.
14. ETS should have effective and equitable procedures for handling
questions of score authenticity arising in connection with the
administration of tests.
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
-7-
CONFIDENTIALITY OF DATA
Principle
ETS recognizes the right of individuals and
institutions to privacy with regard to
information supplied by and about them that
may be stored in data or research files held
by ETS and the concomitant responsibility to
safeguard information in its files from
unauthorized disclosure.
Policies
A. ETS will ask individuals to provide information about them-
selves only if it is potentially useful to those individuals,
necessary to facilitate processing of data or serves the
public interest in improving understanding of human performance.
Insofar as possible, individuals should be informed of the
purpose for which the information is requested.
B. The right of individuals to privacy regarding information
about them that may be stored in the data or research files
held by ETS extends both to processed information, such
as scores based on test-item responses, and the raw data on
which the processed information is based.
C. ETS will protect the confidentiality of data supplied by
institutions or agencies about themselves, and so identified,
to the extent that such confidentiality does not conflict
with ETS obligations to individuals.
D. ETS will not collect or maintain in its data or research
files any critical information that in its judgment cannot
be protected adequately from improper disclosure.
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
ETS will encourage the organizations with which it works to
adopt policies and procedures that adequately protect the
confidentiality of the data transferred by ETS to those
organizations.
Procedural Guidelines
1. Information about an individual, which has been identified as
such, may not be released by ETS to organizations other than
those for which the information was collected without the
consent of that individual. A written exception may be made in
the case of research studies during which the Committee on
Prior Review of Research has determined that release of the
data serves a public need, that there is no satisfactory and
reasonable alternative way of obtaining the information, that
the recipient researcher will use the data in appropriate ways
and that there are adequate assurances of confidentiality.
2. Information about an institution, which has been identified as
such, may be released from ETS only in a manner consistent with
a prior agreement or with the consent of the institution or
with the approval of the cognizant ETS officer and representative
of the appropriate sponsor (if any).
3. An individual should be able, on payment of a reasonable
fee, to authorize the disclosure of information about himself
or herself from program data files held by ETS to any appropriate
recipient, provided that such authorization is in writing and
that disclosure is not inconsistent with other ETS or sponsor
policies and does not violate the privacy of other individuals.
Identification of the requester, through signature and data
file number, or other appropriate method, should be required
before any such information is released.
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
4. In an emergency and when it is to the benefit of the individual,
an authorization by telegram or telephone for the release of
personal data should be acceptable, provided that such authorization
includes adequate identifying information and that such release
is not inconsistent with other ETS or sponsor policies. By
prior agreement with the individual, authorization by a designated
agency or institution should also be acceptable. In such
instances, the individual should be informed that the disclosure
has taken place.
5. If an individual is not competent because of illness or other
considerations, information about that individual may be
released from data files only with the consent of the individual's
parent or legally appointed guardian.
6. Unless the access to confidential data can be safeguarded, ETS
should not participate in any time-sharing network, data bank,
or other electronic data processing or storage system involving
units outside ETS.
7. On submission of appropriate identifying information and
payment of a reasonable fee, an individual should be able to
obtain information about himself or herself in ETS-held data
files for the following purposes: to ascertain the accuracy of
personal or biographical data and to request verification,
within a reasonable period of time, of test scores or other
processed information from tests, questionnaires, or school
records, provided such release is consistent with sponsor
policies.
8. Procedures should be developed for systematically eliminating
from data files information that is judged to be out of date
and, hence, of minimal value.
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
9. Information from ETS-held program data files provided by
individuals for a designated purpose should not be used or
released for another purpose (such as a validity study or
research project) without the individual's consent except
when used or released in a form that cannot be identified
with the individual.
10. ETS should refuse to provide personally identifiable information
except in accordance with these guidelines unless served with
a subpoena or other court order. In that event, ETS should make
appropriate efforts to quash or narrow the subpoena or order or
to obtain a protective order to minimize the exposure of
personally identifiable information.
11. At the time information is collected and to whatever extent
practical, programs should inform individuals of the conditions
surrounding the release and confidentiality of the information
about them.
12. Individuals should be identified in ETS research files only by
code numbers. Information linking the code numbers to names
should be kept in a secure location only as long as necessary
for purposes such as follow-up studies or collating new data,
after which the names should be destroyed.
13. Every organization with which ETS works should be informed
of the confidential nature of any data transferred by ETS to
that organization or collected by the organization on behalf of
ETS so that appropriate procedures can be employed by the
recipient organization to protect the confidentiality of
such data.
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
PRODUCT ACCURACY AND TIMELINESS
Principle
The accuracy of ETS' principal products and
the timeliness with which they are made
available are important parts of the
responsibility ETS has undertaken with
respect to its sponsors and the diverse
public it serves.
Policies
A. ETS will establish standards of accuracy and timeliness with
respect to each principal product.
B. ETS will use quality controls that are adequate to assure that
its standards of accuracy and timeliness are met.
C. ETS will make realistic delivery commitments and reasonable
efforts to meet those commitments.
D. ETS will sacrifice the timeliness of the delivery of information
if the desired accuracy of that information is substantially
in question.
E. ETS will seek to inform those adversely affected if, subsequent
to its release, information has been found not to meet ETS
standards of accuracy.
F. ETS will seek to inform those adversely affected if there
is a probability that there will be substantial departure from
ETS standards of timeliness with respect to a principal product.
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
Procedural Guidelines
1. Principal products should be identified and a standard of
accuracy using units of measurement appropriate to the type of
product should be established for each.
2. When appropriate, quality control should include an adequate
and independent recomputation and a visual reexamination of
ETS-processed information based on an appropriate sample of
cases sufficient to identify errors within the limits of the
applicable standards of accuracy.
3. When the computational nature of the information is such
that it is impossible or impractical to determine the accuracy
of the information by independent recomputation, staff members
who are technically competent to do so should assess its
"reasonableness" as a part of quality control.
4. There should be a quality inspection of intermediate products
a) the accuracy of variable information (e.g.
parameter data, algorithms), verified by
independent recomputation or assessment,
influences the nature of an ETS process or
computation and is critical to the process that
generates an ETS principal product; or
b) detection and correction of errors would
facilitate meeting the delivery commitment
on the principal product; or
c) the nature of the principal product is such
that it is impossible or impractical to
determine the accuracy of the information
by independent computation using the source
data.
Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
 Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
Approved For Release 2006/12/27: CIA-RDP83-00156R000300050003-7
STAT