AGENDA FOR MEETING, 29 NOVEMBER

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP78-06215A000100010005-2
• Release Decision: RIPPUB
• Original Classification: U
• Document Page Count: 62
• Document Creation Date: December 19, 2016
• Document Release Date: October 27, 2006
• Sequence Number: 5
• Publication Date: November 26, 1973
• Content Type: MF

STAT Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
A1I41 hNi-' USE O,N
MEMORANDUM FOR: Members of the Curriculum Committee
SUBJECT : Meting, 21 November
1. The Curriculum Committee will meet on Thursday, 29 November,
in the DTR Conference Room at 0900 hours.
2. The Agenda:
a. Progress report on suggestions made in the
7-8 November meeting:
-- Collection of course objectives for the
Course Data Folder Project -
-- Search for guidelines in course critique
-- Information on coverage of items of
interest in courses conducted by FTD~
b. Comments and review of the draft of an OTR Notice
on end-of-course reports (draft attached).
i. ONLY
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
r INISTRATIVE - g,TE:-;;L USE 044owt
,"NNW,
MEMORANDUM FOR: Members of the Curriculum Committee
SUBJECT : Minutes of Last Meeting - , 7 3
1. The Curriculum Committee met on 29 November 1973 in the DTR
Conference Room at 0900 hours with all members present.
2.1 _J announced (1) the USSR Country Survey has not
been canceled; it will run on a request basis; (2) the evaluation
of the Systems Dynamics Course, which is scheduled to begin in
early January, will come before the Curriculum Committee after the
first r tation; the Committee will discuss its efficacy; (3)
is interested in running a course entitled, "Tomorrow's
Secretary," for the Office of Logistics. The cost is $11,25 per
participant for a class of 20; the Office of Logistics is willing
to pay. The Committee will evaluate this course after the first
running.
3. The minutes of the last meptincr wp viewed and discussed.
STATINTL s a subcommittee to
meez witn to Solicit his views on training for CTs in connection
STATINTL
with any changes in operations training in the coming year.
1 ::1
said that the DO population for the BO or the
STATINTL coming year is aid that he s' 1 could not handle STATINTL
that many people. will discuss this with STATINTL
With the thought o a sing the DDO that there has been some misuse
STATINTL of the BOC in the past, asked for some statistical information
on graduates of recent runnings of the BOC. Mr. White said that he
could get these easily.
4. Concerning coverage of items of current interest in OTR courses,
Mr. White said that we might be too responsive to the whims of special
groups but agreed that the compilation of this inventory of what is
being covered in our courses might provide a good starting point for
a more in-depth discussion of the subject. There was discussion of
using the Committe I Irdinating body on all such requests for
STATINTL special coverage. said that he felt that this sort of thing
would fall in. line as part of our black book project.
5. The Committee then reviewed the end-of-course report draft
notice making specific changes and suggestions. The word "critique"
will no longer be used in the Office of Training; it will be replaced
by the words "course evaluation". Another draft of the notice will
be prepared.
ADMINISTRATIVE - USE ONLY
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
AINISTRATIVE USE ONI
6. I Iwill visit 0 in connection with in-house
STATINTL production of TV tapes and films. discussion of the
relative merits of film and tape. I Isaid that, we need
some evidence and statistics from ALT about t eir instructional
methods and their current needs.
7. 0 reviewed his progress on the black book project.
He felt that the Committee must decide how it wants to go about this,
and he distributed reading material to the Committee. He proposed
that we take two or three courses and use them as experimental units;
for those courses we will try to write measurable objectives. Mr.
I lasked those present to designate staff members to work with him
in translating current objectives for courses into measurable
I If the project is successful, we will move to other courses.
volunteered the IPC course, and indicated that STATINTL
STATINTL will be his designee. STATINTL
STATINTL
STATINTL
8. told the Committee that he had heard from
concerning meeting with the Committee last August a out
traininiRz for outgoing officers whose cover has been changed because of
the STATINTL
an outline of the subjects to be handled in a training course and
estimate of the people involved -- an average of 15 officers a month
based on a 11 month teaching year with the peak period being in the
spring. 0 will study the request in conjunction with the
reworking of the Orientation for Overseas. In addition, STATINTL
has received an informal inquiry from WH about economics training
for its officers. will talk t and will sound STATINTL
out OER as to input. The results will be reported to the Committee.
Both requests may present staffing problems.
9. The next meeting of the Committee will be held on 13 December
with the major agenda item being the evaluation of the Language
Learning Center curriculum.
Approved For Relaa_s_e b'WRi6i _CtAR'DP7$ 10100010005-2
 Approved For Fs'G6~ 04 000100010005-2
NOTICE
NO.
SUBJECT: End-of-Course Reports
1. In recent months, considerable variance has been noted in the
content of end-of-course reports prepared by instructors in the
Office of Training.
2. To establish some uniform guidelines about the preparation
of end-of-course reports, the following information should be
contained in each report:
a. Statistics on class composition; number or percentage
by Directorate or component; grade and/or age distri-
bution; length of Agency service; equal employment
opportunity statistics should not be included;
b. Changes or innovations in the course running; how the
course differed from previous runnings; or how the
course was changed to fill some new objective;
c. Problems encountered and proposed solutions for the
next running;
d. A response to significant student critiques including
a statement to the extent to which the students'
objectives were met;
e. Plans for post-training feedback;
ADMINISTRATIVE - INTERNAL T~~E.
Approved For Release 2006/11/04: CIA-RDP78-06215,3'0100010005-2
 Approved For Release 3~-1 C I P i~'ITA-O9 500100010005-2
.,yV y.711 13 V- ?
f. A summary of the conduct of the course; and a
statement with reference to the extent to which
objectives were met.
2. End-of-course reports should not exceed two pages; they
should be submitted in an original and two copies.
Alfonso Rodriguez
Director of Training
.1AMINI3T:- ,,
Approved For Release 2006111!04 : CIA
-RDi '-OK15?0100010005-2
 Approved For 414~se2 06/ /~M j=Ch4 ?+PVMVZ A0001000100 -2
NOTICE
NO.
SUBJECT: End-of-Course Reports
1. In recent months, considerable variance has been noted in
content of encl-of-course reports prepared by instructors in the
Office of Training.
2. To establish some uniform guidelines about the preparation
of end-of-course reports, the following information should be
contained in each report:
a. Stat.4_stics on class composition; number or percentage
by Directorate or component; grade and/or age distri-
c
bution; length of Agency services ('equal employment
opportunity statistics should not )e include
y}
b.' %ha6j;es or innovations in the cour:;e runnin the --t e
? r~
rc~~ --_.-?-b i `i~rnvi-Iii i c s--.. __.or_ how-_th
e
e
next running ;
d.. A r'w~-4speasi-,t-e significant student iftel-Uding
tattient to the extent to which the students'
obj ect ves- ve-re--me ;
Approved For Release 2006111/04: CIA-R?P78-06~Z1SA 00100010005-2
 n1ZINg", .,.., w Y.. i"!-,", ? ,,
Approved For Release 6b U4 : C1A=R''C P~$ ' At00100010005-2
sununary of the conduct of the course; and a
statement with reference to the extent to which
objectives were met.
2. End-of-course reports should --t~; -PA-t- they
ILLEGIB
should be submitted in an nricini1 wn,a
two copies;
/r sr
Alfonso Rodriguez
Director of Training
Af>
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
ADMINISTRATIVE - INTERNAL USE ONLY
OTR NOTICE
No.
SUBJECT: End-of-Course Reports
1. End-of-course reports will be submitted to the Director of
Training within two weeks after the completion of the course; if this
schedule cannot be met, the Office of the Director of Training should
be notified.
2. In preparing end-of-course reports, the following information
should be included in each:
a. A summary of the conduct of the course, and a statement
with reference to the extent to which objectives were
met; a statement of course objectives should be in the
end-of-course report if it is not an integral part of
the course schedule;
b. Statistics on class composition (equal employment opportunity
figures are to be omitted) including number or percentage by
Directorate or component, grade and/or age distribution, and
length of Agency service;
c. Major changes or innovations in the course running;
d. Problems encountered, and plans to resolve the problems
in the next running;
ADMINISTRATIVE - INTERNAL USE ONLY
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
AMINISTRATIVE - INTERNAL USE ONLY
e. A summary of, and comment on, student reaction to the course.
2. End-of-course reports should be brief; they should be sub-
mitted in an original and two copies with the following attachments:
course schedule as amended, the roster, and course evaluations. Routing
shall be from the instructor to the Unit Chief; SA/OT (if applicable);
C/PDS/DTG; C/PDS; and the Director of Training.
Alfonso Rodriguez
Director of Training
2
ADMINISTRATIVE - INTERNAL USE ONLY
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
OTR Notice
No.
SUBJECT: End-of-Course Reports
D
R
A
F
T
1. End-of-course reports will be submitted to the Director of
Training ten working days after the completion of the course; if this
schedule cannot be met, the Office of the Director-of Training should be
notified.
2. In preparing end-of-course reports, the following information
should be included in each:
a. A summery of the conduct of the course, and a statement
with reference to the extent to which objectives were
met; a statement of course objectives should be in the
end-of-course report if it is not an integral part of
the course schedule;
b. Statistics on class composition (equal employment
opportunity figures are to be omitted) including number
or percentage by Directorate or component, grade and/or
age distribution, and length of Agency service;
c. Major changes or innovations in the course running;
d. Problems encountered, and plans to resolve the problems
in the next running;
e. A summlaxy of, and comment on, student reaction to the
course.
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
USE OPj-,Y
3. End-of-course reports should be brief; they should be submitted
in an original and two copies with the following attachments: course
schedule as amended, the roster, and course evaluations. Routing shall be
from the instructor to the Unit Chief; SA/OT (if applicable); C/PDS/DTG;
- -P DT ;
C/PDS;hand the Director of Training.
Alfonso Rodriguez
Director of Training
I 0 art,,,
2
Usi, ONLY
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO81600~f~C4~ 5-2
Curriculum Com.,:;tee low
29 29 November 1973
DTR Conference Room
The Curriculum Committee met in the DTR Conference Room on
29 November 1973 with all members present.
0
egan with several announcements for the committee:
1. The USSR Country Survey, which was canceled last year, was run on a
requested basis. It ran for 0 ; submect was raised whenSchedule of
Courses showed it with an asterisk "as scheduled." It is not actually
a cancelled course as it was thought to be.
2. Info science courses.
The systems dymanics course is scheduled 5 January. After its first
eva1 a%#em- presentation, there will be an evaluation of it. Ith cost
is high. After its running, the Curriculum Committee will discuss its
efficacy. Cost is high. O*D is paying 50-50. is being conducted by
(foremost contractor in this field in the U.S.) 4-5
aliferent sects using this system which are being expert ented with
in the Agency. Important for OTR to seek leadership in fields like this.
We do not know the extent of itsapplicatin, but it is important that we
go on down the road and do it before some one else does it for us. Let's
try it. It's for analysits -- intelligence and budgetary...
something of a dilemma in preparing the course. Committed to another
Workshop for DIS in April. If we continue to have community
responsibility, put more successful workshops on, we will have more
demand. On the other hand, the individual office workshops may be the
route to go. We'll see how the workshops for the Office of Finance
feels about this. The workshops for components will probably be the
answer.
0
We'll lokk forward to hearing results on this.
Orientation for subprofessionals. There are some serious gaps in some of
the training for subprofessionals. In addition, there is some gap in
clerical training.
STATI NTL
Talkin about two different things here--
II program
and an orientation for sub professionals.
I suggest we let OL spend the $250 and then we'll see about it.
Let's watch the budget. We need some money in our budget for
curriculum development. Whenever this kind of thing comes up, we
react. If we are to be experimental and be reaching ahead we need
some flexibility in doing things more than what our current budget
allows. It's an agonizing process to scrape up this money from unidenti-
fied sources. STATINTL
knows of this-- n a course for sale which is called
Tomorrow's Secretary. wants to bring it into OL. While in
OTR she learned some and is now applying some of the things she
learned. If its valid for OL it's valid for the whole Agency. If
this is right, OTR should be putting on a course of common concern. It's
$11.25 per participant fora class of 20.
Look at curriculum and take a look at employees who are haina out
Apprd d r RT tasvee26$~/'110 4n F~ M- ~'1tf 5f,OC 1-t~3t71
 Minutes of last mtRut meeting
We have had a couple of meetings to comeup with options. We're
looking at our commitments, and our earliest target date for a
change is about the first of the fiscal year.
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
memo to training officers re course for sub-professionals and
wee where we stand, and then 3iscuss.
Suggest that before much water goes under the gate that we ask John
STATINT~ to be here and tell us with respect to his ideas about the
reordering of the training for CTs. Could significantly affect this
and we can't do a revision of the current training profess in a
vacuum.
Options for use of OFC for CTs:
1. Put them all through as a screening mechanism...
2. Take borderline cases with interest in DDO and who seed questinable
and siphon these out and put them through as a prerequisite to the
Appointed
!We're examining our options now. Putting our optinns on planning board to
!see if it is possible to do it. But the next stage is to find out how many
instructors we'll need to do this.
STATI NTL
as a subcommittee to meet with C/CTP and have him
express himself on what his views are .
Current estimate for DDO population for theBOC for the coming year will be
120. Try to get them to break this figure down as to type of
,people they are talking about and get them to agree to revise it downward
by about a third or a fourth.
We have to be practical about it. We simply cannot handle that many in the
1 BOC.
It would mean a different kind of BOC.
Then it's not a BOC. You would not get the same kind of training. This is
Jwhy we want to take a look at possibility of offering an OFC so that people
in the DDO's 120 figure who are not really going to be case officers can be
put into the OFC. We must offer these people reasonable trainng.
Dale, Alan, John to meet and will advise further. Sure that Dale is planning
app=pirate memo and statement for the record of the misuse of BOC by the
DDO. Matter will become acute as we go into this high requirement stage.
Selection mechanism within DDO for BOC shuld be in some way hooked up to
Oso that we can keep them honest. k
Have an appt with
1 -1
!I'm sure that C/CTP will admit that he also has a couple of people in the
BOC who shouldn't be in there. In the future, we'll have to get a pretty
firm policy decision on this...
1. What is the role and responsibility of OTR with respect to the evaluation
of students;
,2. Should we be able to drop students from a course for non performance;
13. Whitt kinds of red flags during the assessment and interview stage before
BOC should be raised as grounds for extra assessment or perhaps
;put them through another screeningprocess via the OFC.
STATINTL
Why don't you II commit to paper what you think it should be. S
TATINTL
Also convey message to 0 that OTR is stirring.
Would like to be anointed a little more formally by the DTR before doing this.
I think it's a question that has to be taken up with DTR and see how far he's
prepared to go.
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 { Approved For Release 2p0 /11/04: CIA-R P78- 6215A00010001 5-
~We don't have t1.lata tha we need perhaps to ma, some j~agments. We
should take a look at last couple of BOCs and see what the people who
finished it did... overseas, etc.
Dale doesn't have them, but said he could get and see what'SSTATINTL
Jhappned to the last couple of classes.
Keep in mind that we have x amounts of space, x amouutts Of instructors, and
Ix amount of money, and since there is not going to be any increase, it's
going to be hard.
The message is to be given to our principal customer and then we'll hear
further.
Since IWA is also affected in the training of CTS, should someone from there atti
attend these meetings?
Par 7 of minutes
Re items of current interest in courses, it might be appropriate for the CC
to deliberate on the subject of the extend to which it is desirable or
necessary to include in a variety of our courses mere allusions to these
little interests of the moment. Are we doing too much of this?
Are we being too responsive to the whims of special interest groups.
This inventory of what is being covered might provide a good starting off
(point for such a discussion.
+"+ We're having so many fragmented dealings which might lead to a curriculum
imbalance.
Is it not then clear amount us that the Unit Chiefs say to their people to
not respond 0 too easily to special requests.
We have a responsibility to coordinate this, and there was an attempt to do
this in the past.
When these requests some in we should ask that it be made a formal request.
Ie really should hav ea procedure forhandling these things and the
decision should not reside in the hand of any unit chief. It ought to come
to a central point/body. It shold come to this body. If it results in a
rejection, then it ought to go to the DTR.
If you're talking about a suggestion for a particular guest speaker
presentation, I would certainly hate to come running to the CC for each one.
If you're talking about a course offering, then I certainly would, but not
for just one particular topic or lecture.
If the Committee wants to establish guidelines concerning what seems to be
pproprirate content with respect to AOD, IWA, that's fine, but as far as wantinj
o put a new activity in the Mideareer or the AIS, this would be an enormous
task.
I don't think this Committee should be wasting its time on this.
at we need is a centralized coordinated useful data base so that some analysis
6'an be made in not only to tell us what is going on in /at a cettain
point so that we can see collectively what is going on. We need a total
ompilation of what we are doing. We are just starting this in our black
book project.
on't the schedules give us this content?
Sometimes the schedules don't give you this information. We have seen it
any times. Wkem-t,kese-egfieere-frequently-made
The Committee here can be aware of anything new today. One thing that is
Coming is the T.C. ...this will be quits an item. I think we should be alert
to the fact that someone in the I.C. staff talked to the DTR and said I want
I Approved For Release 2006/11/04: CIA-R?P78-06215AO00100010005-2
 !to rikvgdf(trfRele i{D r!i16W4C:.ClAaR &66015AOO@91 0 O6a&rses. DTR
!said, "I told him would do everything I could to help him."
This is an example of how we can keep each other alert to the new things
coming. ttx:tk As they do, let's talk together and be aware. STATINTL
I would guess that 75% of the guest speakers appear in courses.
If he has no problems, theg why are we worrying?
The new oranization solves many of these kinds of problems.
End-of course report notice...
suggested we strike the first paragraph onthe notice and STATINTL
Ladd: end-of course reports will be submitted to the DTR through the
unit chief within two weeks of the termiaal date of the course:
exceptions being any unspecified instance where the schedule cannot be met
the office of the DTR should be notified.
Objectives should be dealt with in the first paragrph.
We should find a new name for critiques...
I looked for guidance in construction of critiques. Geneeally, people
do not use critiques in the sense that we have them. They go back to a
Ivery specific learning objective. Discussed CSC paper dated 26 October 1973,
!discussed CSC forms...a controlled response and free response section.
II also showed the members another guide: Training Opinion
Questionnaire; also showed them our new 10491.
+Committee decided we would use the word "course evaluation" instead of
in mins critique. No longer will the word Critique be used in OTR because it
has negative overtones.
in mins Objectives should be written into the course schedule.
Statement of course objectives should be in the end-of=course report --
in the body of the report or on the attached schedule unless it is an
integral part of the tours schhdule.
will be next week. This refers to the priority STATINTL
jfor the xa in house production of TV tapes and films. At our earlier meeting,
pointed out to us that overseas stations have only certain technINTL
piecess of equipment...
)sicussion turned to professional producer of tapes and films: STATINTL
e have been relying on our internal assets and relying somewhat on external
Maybe we have to hav a picture of the equipment on hand in any station in
he world. OL can give us this.
to OTR serious about going the route of heavy involvement of production of
tapes and film and we we are will we continue to operate as catch and catch
pan. If so, should we go out and hire our own consultant?
~s this set of questions appropriate for this group?
STATINTL
!That good has it done us to have gone this route once before with 0 etc.
ire their films useful today: Have they been useful?
have a paper on tapes and it gives statistics, but don't have paper on
statistics for film.
One big advantage with tape...once you're finished with it, you can erase it
and use it again. Costs of color film is expensive - $400...black and white
.s - $200, tapes - $40.
jroblem is it ALT use over a.
Approvedor e ease 2006111164 : CIA-R?P78-06215AO00100010005-2
 5
STATI NTL
STATI NTL
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
They can put together some statistics to be accurate indicators of frequency
of us. Think to films may be in almost constant use. Some go back
10
to the 50s. They may be txrlmte dated in technique and costume, but they are
valid for content.
Only guy we have is
120 days a year.
STATI NTL
and he's on contract and he can work for us only
In addition to l we have a number of people who are interesteeaalNTL
eager to spring into the vacuum and they are a lot of people who want to step it
its easy to get yourself i. into a k point ofhaving no product or a oneway
product. ) My real beliefe is that OTR does not k ow why it is doing somethin?
We must take decision in the firm knowledge of whyxixa we are taking it.
Don't let things happen without decisions.
Decisions must be made within the content of obuectives.
is coming up with a proposal for full time academic traininw?STATINTL
to Study this sort of thing. C C should decide on its needs before the
Career service board makes its decision on sponsorship.
Does compl tion of study guarantee ability? Will he be fully qualified
or in an apprentice stage?
Feel we have to get someoutside help. I'm not so sure that we should pay much
attention to film. Perhaps we should think more about the video tape thing.
Let's cincentrate in the future on video tape.
We're not technically equipped yet to do a cimparable job on tape. We have all
the equipment we need for motion picture film. We do not have color cameras
for TV, we have no editing capability and a few other things are missing to
get our technical capability up to the same level for which we have in
producing movie film.
Will production costs offset this?
They should.
There seems to be no disagreement that OTR goes the route of profiessional
assistance. Also, TV operation looks to be the picture in the long run.
Question: what route do we go to acquire professional assistance.
Feel that a rather full scale staff study of the OTR production problem
the alternatives open, thus -- film versus video, and the alternative
(which failure to take action along one of these routes would leave us...
la staff zxtg study such as the one which was done on the reorganization
!of OTR.
Or maybe a presentation to this group.
I think we could use the assistance of an expert in the field who could examine
our effort and make some suggestions on what are doing in this field.
We must really think about the problems of ALT overseas and our responsibilities
to them.
At the very minimum, what is needed is some evidence from ALT about their
dependende on that instructional method and whether what they have needs
1u ating. We need statistics and information from ALT and Ken is going to
11 1 Related to this is the fact that what what our CSB says when it
receives the training request forl may depend on the CC;s STATINTL
ti
iti
th
e ques
on.
pos
on on
;STATUS OF BLACK BOOK
reviewed the requirement we have from J the DTR for the project.
pp, d For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 STATINTL' has collecta.r`the objectives. Don't think w'can do anything
,constructive about this today. We must decide how we want to go about this.
.writing, rewriting, or putting a stamp of approval on objectives....
distributed an article from Materials and Methods in Adult Education;
entitled, "Planning for Instruction with Meaningful Objectives,
He also distributed course objectives fmxx submitted by the Intel Institute
and Functional Training Divison.
He also recommended for C C perusal, "Taxonomy of Educational Objectives."
The above should lead us to thinking about our ems[ corporate problem.
I leave you with a proposition:
We volunteer a couple or three or four of our courses as experimental
unite. And the experiment will be in the nature of trying to write measurable
objectives. I am not suggesting a change in courses, but I am suggesting
a series of thinking sessions with people that you designate on courses that
you designate. Try to translate our current objectives into measurable
objectives. If this is successful, we will move to others. We're
concerned with a requirement for evaluating and there is no way to do it
unless our objectives are set out so that they are measurable.
Rig IPC
Volunteer the because it has a very weak current statement of
objectives and because we are going to make some major changes in this
course. And now is a good time to do it. We can kill two birds with
one stone.
.Rick can help with anything in my group.
!Agenda for next meeting- 13 December:
1. Review of minutes of this meeting
2. Evaluation of LLC curriculum
Next meeting following will be 10 January.
requirement. May present some problems of staffing. Looks like the program
,will take one week; also would like for wives to participate.
and also somewhat imprecise estimate of the numbers or people. He e us -a
program. is still vague as to what the estimates of people are.
An average of 15 officers a month based on 11 month teaching year. Peak
period would be the spring months. Said I would study and look it as a
reworking of the Orientation for Overseas. This seems to be a bona fide
an informal inquiry out of WH for economic r their officers.
' ey yes or no at this point, but will go talk toSTATINTL
Ito talk to them and findout what is on their mind and sound out
OER as to their input. Will come back here and discuss it. This one will
also present a staffing problem.
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 opTlc 1A . roRb1 NO. 10
1, 4`AT'cF i?? oved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
UNITED STATES GO`' o iNMENT N"r
TO
FROM
SUBJECT:
Memorandum
Director, OTR
01k. 8646
DATE: 20 July 1973
Program Performance Measurement.
The Emerging Role of OTR in establishing
Requirements for Training.
I am submitting this memorandum intended as a discussion of
(1) how to provide the data base necessary to tell OTR how well
the programs are meeting their objectives on a current basis
(2) an approach to determine to what extent the knowledge and/or
skills imparted are applied on the job and
(3) how OTR can establish requirements as contrasted with
historically reacting to requirements.
A concept of a data base is presented which is intended to aggregate
all of this and also attached is a basic form that can be used to
respond to all of these goals.
In order to institute the best type of (program) performance
measurement a number of aspects must be considered which not only influence
the behavior and iaorale of those being measured but also affect the ability
to satisfy the student -- our customer.
This paper is divided into four sections
Section 1 - A Discussion of Program Performance Measurement
Section 2 - OTR's Rule in Establishing Requirements
Section 3 - Some Observations
Section 4 - Appendix A - C
Chairman, Functions Course
Buy U.S. Savings Bonds Rel r ~ a the Payroll Savings Plan
Approved For Release 2006/11f 1A-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
A Discussion of Program Performance Measurement
There are four steps involved in Program Performance Measurement
that must be followed in a continually repetitive sequence to achieve
the full benefits of such efforts.
The first step requires the careful specification and analysis of
basic program objectives in each major area of activity. To accomplish
this one must back away from the particular program being carried on,
look at their objectives, and ask what are we really trying to
accomplish. This definition of goals to be accomplished should start
ideally at the top of the "organization" so that each level can be
certain that their definition of goals falls within the scope of what
has been defined by the next higher level. OTR certainly must
provide the training necessary for agency personnel to accomplish or
improve the accouplishment of the various agency missions but such
definition is too broad and must be narrowed. At the other extreme
for OTR to say we shall train "x" number of people or increase our
student through-put by some quantity is too narrow and must be
broadened. Specification of OTR objectives must fall between these
two extremes. The more we learn about how to reach an objective,
the more clearly we understand the objective resulting in a constant
interaction between the decision process and our knowledge of our
true objectives.
The second step involves the analysis of the output of a given
program in terms of the objectives initially specified In the first
step. For the OTR programs, our output addresses not how many or
how many more students did we turn out but rather what improvements,
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
1.90.1 Vae
knowledge and/or new skills have we provided to the student. Will
our educational efforts influence his behavior on the job and improve
his job performance. This is often referred to as feedback a term
taken from engineering servomechanism theory.
The third step calls for measurement of the total costs of the
program -- not just for one year but over at least several years
ahead. This would require OTR to identify the resources expended
in each program, to accumulate resource costs by program and to
extrapolate into the future periods from a historical cost data
base.
The fourth step involves the analysis of alternatives if and
only if programs are competing for limited resources. As an
expository application let us examine how thesr~ steps or this cycle
of events applies to the Information Science Program. and the
Information Science for Intelligence Functions Course within that
program.
Course objec?:.ives serve two purposes:
1. They express the desired results of our customers in
terms of accomplishments or goals to be achieved by
the course.
2. They provide the basis or elements fundamental to course
evaluation.
The most difficult aspect of any professional performance
measurement scheme is not how to measure but what should be measured.
What should be measured is dependent upon how one is organized and
whether accountability and responsibility for activities which
3
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
satisfy customers needs are clearly defined and understood by those
charged with such responsibility and accountability. Clearly
organizational analysis is beyond the scope of this paper nevertheless
the interrelationship between the design of an organization and an
evaluation of an organization's performance must be clearly established.
These ideas have been used in our Information Science Program.
For example I have assumed from our previous planning that the single
overall objective required by step one of our Information Science
Training Program at the Chief/Information Science Training Staff level
is to educate Intelligence Professionals in the Information Science
Disciplines. For the Current FY 73 Community Program segment, the
numbers of students planned were as follows:
2 offerings
--?
Survey
--
3 weeks
--
30 stdts = 60
2 offerings
-?-
Functions
--
4 weeks
--
25 stdts = 50
2 offerings
--'
Inf. Sci.
--
1 week
--
25 stdts = 50
Total Students
160
Total Course Weeks
16
Assuming for step two that the proper output measure of the
Information Science Program objective is to quote numbers of students
exposed to Information Science then I could certify for the Functions
Course that 27 + 30 or 57 students completed the Functions Course
during FY 73. One can further document the specific inputs (and
their costs required in step three) required to generate this output
of 57 students i.e., lecture time of resident and guest faculty,
course preparation time, computer time and charges (by problem and
lecture if necessary) etc. From a Systems Analysis point of view,
- 4 -
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
the output (57 students) is tied to the related and required inputs
(the resources and their costs) but is this a sufficient and proper
measure? This measure of output as a single measure reflects the
number of students put through the course but does not reflect the
full range of values or benefits provided to our students and their
respective organizations. In order to measure the value provided
by the Functions Course and its contribution to the Information
Science Program overall, it is necessary to go beyond the single
measure of student output and resource input to determine whether
we accomplished the course objectives and accordingly satisfied
the students our customers. What is missing is the careful specification
and analysis of the program objectives which produces a more meaningful
output measure.
To elaborate further -- for the Functions Course alone -- four
priority ranked specific student-oriented objectives were established
(step one):
1. To familiarize you with the terminology and basic
techniques of Information Science.
2. To develop your capability of identifying and defining
problems in your professional intelligence field which
are amenable to solution by information science techniques
and to solve such problems at the elementary level.
3. To improve your cotmunications capabilities in conferring
with information science professionals.
4. To encourage you to pursue further the development of
your own, and your organization's information science
resources and capabilities.
How well we accomplish these objectives in the Functions Course
with a fixed set of resource inputs is reflected in the course
- 5 -
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/1.1/04: CIA-RDP78-06215AO00100010005-2
evaluations completed by the students. A summary of the student
evaluation responses from the last Functions Course demonstrates
to what degree these objectives were fulfilled for that particular
class (step two):
STUDENT EVALUATION-FINDINGS
OBJECTIVE OUTSTANDING/EXCELLENT GOOD FAIR
27 Students 3
90% of the class 10%
15 13 2
50% 43% 7%
21 7 2
70% 23% 7%
23 6 1
77% 20% 3%
From these findings one can conclude. that all objectives were
accomplished. S-A.nce objectives 2 and 3 rank the lowest, efforts
should be directed toward improving those areaa for the next
offering of the Functions Course. A follow-up questionnaire, four
to six months after the course, would further confirm the degree of
job application accomplished by the students or each course.
These objectives and the associated evaluation scheme used for
the Functions Course may not be applicable for all courses conducted
by OTR. Each course must have specified:its own set of objectives
based on the unique requirements of the customer needs to be satisfied
by each course. How each course is conducted depends upon (1) the
subject material to be presented, (2) the skill and knowledge of the
assigned manpower presenting the course and (3) any procedural
directives issued by top and/or middle management that apply
-- 6 -
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
specifically to a particular program or course. These are the
factors that influence not only the specification of objectives
but also the degree to which the course objectives can be
accomplished.
OTR's Role in Establishing Requirements
Although this process, its steps and their interrelationship
have been explained using the Functions Course as a "real live
intelligence application" the same concepts apply to any level within
the organization. For example, the objectives that OTR must accomplish
have in the past depended upon the requirements that have been
levied by "Top" management. How OTR is organized has depended upon
(1) the entire list of such requirements and (2) the aggregation of
similiar activities and/or courses (requirements) into` manageable
(school) segments. Such aggregations must provide for clearly defined
areas of responsibility and authority which are the pre-requisites to
accountability. To say the least I have been disappointed in the limited
agency career development program that exists only for specific segments
or very narrow specialties. The time has come for OTR to become more
aggressive not only in career development training but also in much
of the currently named "component" training. The training that OTR
conducts for the Directorate of Operations and the Directorate of
Intelligence I consider to be basically component training. In order
to effect batter utilization of training assets by OTR, OTR must
effectively plan, organize and control. the total training effort of
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
the agency. Planning sets the stage, organizing sets the resources
and controlling sets the degree to which the organization will
continue to survive. Planning determines a schedule of what is to
be accomplished, Organizing determines the resources to be invested
and how to accomplish the planned schedule and controlling determines
how well the plan was accomplished. Since control requires a
measured comparison of what was accomplished to what was planned to
be accomplished, there can be no control without a succinct plan.
The degree or amount of control required is dependent upon the
professional capabilities of the personnel. In this regard OTR must
decide what role it is to play in Agency Training.. This really means
a restatement of the objectives to be accomplished, time-phased in a
reasonable balance between assets, resources and time for accomplishment.
I am reminded of the little boy who was asked by his father (a world
reknowned Production Control expert) what he wanted for his birthday
next month. The boy replied "a baby brother." His dad responded
with "that's impossible son." To which the boy quickly replied
"you have taught one all of the principles and practices of effective
production control, just put more men on the job".
Once the objectives are carefully specified and analyzed, then
OTR can determine if the present organization is properly structured
to accomplish this priority-ranked list of objectives. The only
definitive principles of organization structure that apply are that
(1) each organization falls somewhere between the extremes of being
functionally organized and being product or service organized and
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
(2) like-activities should be grouped into manageable segments so
that the specified objectives of such "grouped" activities can be
responsibly accomplished.
If OTR is to exert more influence in career development, then
OTR must write career development plans (programs) in conjunction
with each responsible directorate. These career development programs
then become the focal point of the QTR program planning and the
subsequent course planning. The existence of career development
plans does not require OTR to be organized along the same career
structure. In fact, an OTR element such as the Information Science
Program can cut across a number of career development plans by
providing segments of instruction or courses that fit into various
career progression patterns. Other such examplas include the IWA
introductory orientation, the mid-career and management courses and
the senior seminar.
If OTR recognizes a need or void in training then a course or
program should be developed and presented. Demand for such a program
can be generated through effective advertising campaigns. We should
look to the methods used in successful marketing research and advertising
campaigns and apply such tactics to researching the need and promoting
newly developed programs.
To determine how well each course is meeting its objectives, I
offer as an example (appendix A) the form used in the Functions Course.
As a follow-up questionnaire, I offer the form (appendix B) designed for
the Functions Course. To determine the allocation of manpower/time
assets, I offer the Faculty/Staff Activity Report (appendix C) as the
g .-
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
basic time allocation data collection device. One can thus relate
resource expenditures to the benefits provided to our customers --
the students and to those responsible for review of career development.
Some Observations
It is often charged that such measurement sets up biases in
decision-making by concentrating on costs and ignoring intangibles
and human factors which cannot be quantified. Or conversely by
naively attempting to put numbers on such imponderable elements
thereby misleading the decision-maker. Such thinking often forces
personnel to play games. For example (in the evaluation of personnel
who (1) present lectures or (2) participate as guest lecturers in
other courses.) If I knew that my performance was to be evaluated
on (1) the number of students who were lectured (2) the number of
my lecture hours and (3) the ratio of students per lecture hours
then I would make certain that my lecture hours per course were at
a maximum and that I lectured only to large sti:dent-groups or classes.
I could so maximize my performance measures that I would always be
number one on this performance roster.
However sincere these critics may be -- they reflect a complete
misunderstanding of the relevent issue. And sometimes they simply
reflect the chagrin that particular pet projects may not show up
well under such measurement schemes.
Program Performance measurement does require a systematic analysis
(the means) of program proposals and decisions, concentrating on those
particular decisions (the ends) which have inherent budgetary
- 10 -
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
i
consequences. Please note that systematic analysis does not have to
be quantitative and is not co-extensive with quantitative analysis.
The word "analyze" does not have the same meaning as the words
"quantify" or "measure" although analysis often includes some form
of measurement. Management by Objectives, the PPB process or whatever
the current "in" title of a good management philosopy is, all of these
concepts seek to subject to a systematic analysis both the tangible
and intangible elements of a program decision. We live in a world
that must make decisions often using limited or meagre information.
This is more akin to the European Style of Management which opts
for a decision as contrasted with the American Style which opts for
"adequate" information before making the decision. In any event
let us not become so Management by Objectives (MBO) oriented that
we become managers who can't make decisions unless the plan calls
for a decision.
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Releases qt~,1iiIp4,4L I fTP78-06215A000100010005-2
IilFOXIATICN SCI INCE FOR INTELLIGENCE FUNCTIONS
CLASS:
I['TRC,"IUCTION. The Staff is concerned with the quality of its educational
pr?og:r n. Your constructive cofiir,,. nts are solicit-2J and will be used as part
of the basis for improving the ability of future presentations of the
Functions Course to meet the information science training requircnents of
user organizations.
1. What is your overall impression of this course?
Outstanding Excel lcnt Good Fair_-Poor,
~. Did we fulfill each of our course objectives for you? Indicate below:
(a) To familiarize you with the terminology and basic techniques
of Information Scicn.e.
Outstanding____ Excellent Good Fair Poor
(b) To develop your ca;:ahil it; of identifying and defining problems
in y !ir p~o'c ~ss'ional ;ntell.c;;c..Ce ffe'id which' ara amenable to
solution by i.i ra: n ~.i on sci e:r?ce techniques and to solve such
probl gists at the el c ;er,tary level .
Outstanding Excellent Good Fair Poor
(c) To fi., rove your c.- a;?+icotions capabilities in conferring with
information science prof ssionals.
Outstanding Excel l antGood Fair Poor
(d) To encourage you to pursue further the development of your own,
and your orjanization's information science resources and
capabilities.
Outstanding Excellent Good Fair Poor
(e) Do you believe the course content is compatible with the course
objectives? Y(:, l\o
Discuss this cox-.patibility and C inscribe any changes that you
think would increase this cormpatibility.
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 4. Would you recce,: ,end this course to a co-worker?
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
...r Ye:; No
5. What is your joy title and what are your major tasks?
6. What are your personal reason(s) for attending this course?
7. Reflecting on your learning from the course what percent of the course
do you feel will contribute or be of use to you? Please identify by
entering one (x) chock for each line.
a. Immediate Use
ZMY 75;6G `3,
b. Long-range Use
8. Please rate the course in light of your answers to Questions #4, #5,
#6 and R.
Outstanding____ Excellent Good Fair Poor
9. As a follow-up to your end-of-course evaluation, any subsequent
comments based on work experience at your home station would be most
welcome and helpful in updating course relevance. Would you be
interested in accc';npl ishing a post-graduate questionnaire four months
after completion of this Course?
Yes No
Approved For Release 2006111/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
Reflect on your learning from the course this week. For each subject element listed please indicate
your personal reaction to the appropriate space.
ST B.ECT ELEMENT
I can see no
I am aware of
I am aware .
I'd like to incor-
I will -certainly
application for
a few appli-
of many
porate the infor-
incorporate
this element in
cations for this
applications
mation & tech-
the infor-
my work.
element.
for this
n_iques from this
mation and
element in
element in zny
techniques
my work.
work.
from this
element in my
work.
.~lem.entary System
Concepts
Basic Programming
Statistics
Library Programs .~
..
DELPHI
Decision Trees
4
Netw6rk Analysis
Intelligence Problems -
making use of
these new techniques
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
Reflect on your learning from the course this week. For each subject element listed please indicate
your personal reaction to the appropriate space.
SUBJECT ELEMENT
I can see no
I am aware of
I am aware
I'd like to incor-
I will certainly
application for
a few appli-
of many
porate the infor-
incorporate
this element in
cations for this
applications
mation & tech-
the infor-
my work,
element,
for this
niques from this
mation and
clement in
element in my
techniques
my work.
work,
from this
element in
m work, ??'
Probability
Linear Programming
Basic Programming
Single Correlation &
Regression
Library Programs
DE PHI
Probability Assessment
Intelligence Problems -
making use of these
new techniques
t
~
f
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
3. Reflect-on your learning from the course this week. For each subject element listed please indicate
your personal reaction- to the appropriate space.
SUBJ. CT ELEMENT
I can see no
I am aware of
I am aware
I'd like to incor-
I will certain'
application for
a few appli=
of many
porate the infor-
incorporate
this element in
cations for this
applications
oration & tech-
- the infor-
my work,
element.
for this
niques from this
mation and
element in
element in my
techniques
my work,
work.
from this
clement in m'
Vy'ork,
Information Storage And
Retrieval
Queueing
Basic Programming
Decision Theory
Library Programs
DE LP II
ISS
COINS
Intelligence Problems -
,
making use of these
new techniques
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
3. Reflect on your learning from the course this week. For each subject element listed please indicate
your personal reaction to the appropriate space.
SUBJECT ELEMENT
I can see no
I am aware of
I am aware
I'd like to incor-
I will certainly
F
application for
a few appli-
of many
porate the infor-
incorporate
this element in
cations for this
applications
mation & tech-
the infor-
my work.
element.
for this
niques from this
mation and
element in
element in my
techniques
my work.
work.
from this
element in
my work.
Retrieval Operations
Search Strategy
Bares ian Analysis
Semantic Distortion
File Construction
MIS /PP B
Human Factors
Mo ,+ell-inb &
Simulation
PERy
Intelligence 'Problems -
making g use of these
nev techniques
I
~
E
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
r N"WO
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 NANE :
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
WEEK OF: TO
This form is simple to maintain if done each day. Indicate the
amount of time spent performing any activity listed. (Indicate
others if necessary.) An activity is considered significant if
it requires more than 15 minutes of your time to do it. In the
case of short duration jobs, i.e., filing, merely record the
approximate total time spent in that activity during the day.
turn in to branch chief by Friday noon each week.
ADMINISTRATION:
Management:
Mon.
Tue.
Wed.
Thur.
Fri.
O.T.
letter/report-research
letter/report-preparation
letter/report-coordinating
Briefings-preparing
Briefings-presenting
Briefings-attending
Meetings-preparation
Meetings-attending
Telephone
Clerical:
typing
filing
reproduction
-`?
meetings
telephone
training
--------------------------------------------
CURRICULUM :
Instruction:
Research/study
------
--------
--------
-------
-------- -
------
Lesson preparation
Lesson presentation
Lesson attendance
Coordinating
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
NW
CURRICULUM ACTIVITY CONTD.
Scheduling,:
preparation
coordinating
Student Advising/Evaluating
Faculty Advisor Time
Seminar Evaluation
Paper/report evaluation
Briefing evaluation
Counseling
Record administration
Exercises:
Development
Preparation for
Participation
SECURITY:
Security indoctrination/training
PROFESSIONALISM:
Professional development
Proficiency maintenance
-------------------------- -----------------
PRAVEL/LEAVE:
Travel/TDY
Leave
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Rel~6se OO6%''%04 ?-C1A-RDP78- 3
1~5' f300100010005-2.
Ideo Tape & Motion Picture
N
IF,
ms
W
Video Tape
eauestor Date Finish Subject
. Sem. Apr 73 Apr 74 Unspec
I A (MC) Apr 73 Nov 73
Apr 73 June 73 Guest Speaker
Apr 73 Sept 73 CO/Agent Mtg.
:Ir,'A (IWA/IPC)Apr 73 Open DCI/DDCI, et al
? Sept 73 Uns?ec
TS Mar 73 Apr 73 "News Excerpts" Completed
ang Sch Apr 73 FY 74 Unspec
ps 5 h Apr 73 May 73 Guest Speaker
It
it
STATI NTL
STAT
No follow-
up
No follow- "30 min. tape for stud. brief-
up .ing"
Completed
Idea Stage
"PRC Pol. & IAITIN-js.
lecture: exceprts
Re-do, 1 hr. tape, for IW'A &
CA sem
Idea Stage Excerpts from sev. films - for
IWA
Open "Damage Report" No follow:v- Excerpts
up
? Open "The DO Story" Idea Stage
Sept 73 Open CA Case History Disc. Stage - Multi-media STATIN
Oct 73 Oct 73 "Black Sept." Complete3 1" & 1/2" copy from motion pic.
Completed 1"
Completed AGS Prog. 3/4" cassette
itel Inst Sept 73 Oct 73 Brzezinski
-is
STATSPEC
Apr 73 Aug 73 "The Teletype Mes."2 Completed 4 more to
Oct 73 Open Guest Speaker Ping, Stage Smoker Cessation, 3/4" Cassette:
(color), borrow cameras
Status Remarks
No follow- 6-8 tapes, 10-1.5 min. each
up
8-10 Guest Spkrs 1 done-Sept 2 in #36, 2-3 in #37 (Sept)
R-R 1/2" 4-5 in #38 (Nov)
eP ers Oct 73 Oct 73
;iS May 73 June 73
"Position Audit" Ping. Stage 1/2"
Guest Speaker Completed Smoker Cessation, 1" B/W
not used
Approved For Release 2006/11/04: Q P78-06215A000100010005-2
 Approved For Release 20Q6/1.1/04: CIA-RDP78-06215A00010001.0005-2
1400, Motion Picture
auysfor Date Finish
S IAA, Apr 73 Open
Apr 73 NA
Apr 73 NA
" Apr 73 Open
L-ng Sch Apr 73 FY 74
AT Apr 73 Nov 73
Apr 73 Nov 73
Apr 73 Nov 73
Apr 73 Nov 73
ApK 73 Jan 74
Apr 73 July 74
Apr 73 Sept 74
St bj_cct
Guest Spkr Prog
Agency 'Spkrs
Drug Abuse Sem
Unspec
`'Pers. Mtg."
Setp 72 Dec 73
Sch June 73 Open
z:ng Sch
June 73 Open
;' JDM&S Aug 73 Sept 73
/DCI Sept 73 Sept 73
:)/Sec Oct 73 Nov 73
STATINTL
Point in Time" In process 1+ hrs. color, at presei
(80% done) Feb 74 more re,:",
st c
ROC Program
Lang Proficiency
Testing
Anniversary &
Awards Ceremony
Status Remarks
Awaiting Up-date, cut length
dec. re new
edition
3 Comp feted Periodic color shifted to
videocassette
.No follow- Color
up
ITB holding 1.5 min.
ITB holding 15 min
ITB holding 15 min.
ITB holding 30-40 min.
No follow- Studio production
up
No follow- Color, 2 films, 10-12 min. ea
up
ITB holding 3.5 min script submitted May 7
ITB holding 40-45 min.
STATI NTL
ITB holding 60 min.
In process
0
Ping. Stage 20 min..
Completed 45 min. (color)
Completed Color
"Defensive Driv." In process Color, title inserts & film pi
Ping. Stage Color
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
*AW
1 November 1973
MEMORANDUM FOR: Chief, FTD/ISTS/OTR
SUBJECT Some Comments Relating to OTR Policy
Concerning Course Evaluations
BACKGROUND
1. Course evaluations designed by skilled professionals
and properly administered can provide a means of determining
customer acceptance of training provided by OTR. A student's
reaction to a course, or to parts thereof, may be influenced
by a wide variety of factors, such as his own work
experience, or his boss's attitudes toward new methods.
Course evaluations, however, should be designed to zero
in on relevant factors. For OTR, relevancy must stem from
and be determined by course objectives. Course objectives,
in turn, must grow out of the Agency's perceived training
needs. Assuming effective teaching of relevant subjects,
then to the extent that a students expectations and a
course's objectives are similar, that student's reaction
to training will be "favorable."
2. Keeping in mind, then, that course objectives should
govern relevancy, those student comments directed toward the
degree of achievement of course objectives are the ones of
utility to OTR. Other commentary, of course, may well be
interesting, even of peripheral value, and so should be
welcomed by instructors.
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
DISCUSSION
Job-Related Courses
3. If courses are skills training, or job-related,
they are intended to produce an early payoff and there
are two specific areas of the course evaluation. The
first relates to how much the student learns (Information
Transfer) in the course, and this is an OTR responsibility.
The second relates to job environment and whether that
environment allows him to apply the new information and
techniques on the job; this is not OTR's responsibility.
(This second area assumes that the course is relevant to
his present position.)
Information Transfer, An OTR Responsibility
a. The amount of information transfer is readily
measured by having the students answer a list of
questions (structured from tho course content) on
the first day and the same questions on the last
day of the course. The difference in correct
answers between the final questionnaire and the
beginning questionnaire is a meaningful, positive
measure of student learning or information transfer.
In this manner OTR measures the effectiveness of
the course or learning experience.
Work Environment
b. Whether or not the work environment allows the
student to incorporate new information and techniques
Approved For Release 2006/11/04 ;,CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
*We 11000~
in his work (not an OTR responsibility), OTR can
help make the student aware of applications of the
information and technique, and may persuade or
influence the environment through education to
accept and use new methodologies. OTR can,
through positively structured (level of intensity)
scales related to course content, determine to
what degree each student has been made aware of
applications, and also whether his work environment
will allow him to make application of the new
information and techniques.
Career Related Courses
4. Career-related courses are intended to produce a
long run (career) payoff and there is only one specific
area of course. evaluation i.e., what did the student learn.
Information transfer (as in job-related courses) is readily
measured by comparing questionnaires answered on the first
and last day of the course. If a course is unusually long
(more than 4 weeks) then a questionnaire can be divided
into two or more questionnaires each covering a bloc or
segment of course content. Such questionnaires are positive
in their motivation because there is reward in the comparison.
Students will always learn something and this measure gives
credit for learning.
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
Evaluation of the Course (Student) Evaluations
5. It is to be expected that student responses will
exhibit a range of answers; OTR can satisfy all the people
some of the time and some of the people all the time, but
we cannot satisfy all. of the people all of the time. When
presenting new information, new techniques, and new develop-
ments, we can expect skeptics to be present. With well-
structured course goals or objectives, if the majority of
the class agrees that the objectives have been accomplished,
we can be reasonably sure we have succeeded. It is true
that a lone dissenter may be singularly correct, but this
is the responsibility of the course director to evaluate
and decide for his course.
Approved For Release 2006/11/04: CIA-RDP78-06215A000100010005-2
 e fascinating
pictures. check
cal library to
back issues of
(PTE (Society
id Television
teresting bios-
iechanical ge-
)r appeared in
inte 72) issue
fond Fielding
SMMPTE arti-
~k entitled A
If Motion Pic-
blislted by the
t Press.
producers of-
i without pre-
I've got it all
is the cont-
d I spend the
able to put it
simple. Skip-
ng phase will
in spending
ng overlooked
Itching and re-
inc the entire
seen spent had
adequately at
most convc-
a of planning-
of good plan-
idea on a sep-
using as few
ten to arrange
t the most ef
and sequence
by everyone
hcs can show
each card, as
es. The plan-
y then serve
script for the
pher, whether
s to be clone
"amateurs."
also called a
best method
)king subject
pproval from
shooting any
siting hoard
and trainees
our ideas are
llicit enough,
Approved For Release 2006/11/04: CIA-RDP78-06215AO00100010005-2
.,I 'S,.J YJ / ,J^-/ul Ci
J f'??s ~~].' -] t't Aa.s 7.c?. 4! fz+,"~nf if."~ ry 7f r
by Martin M. Broadwell
Evaluation by trainees-the happiness rating-is, Broad-
well suggests, worse than useless. Instructor perfor-
mance checklists aren't much better. What's needed?
Something new . . . and truly helpful to the , instructor.
Talking to a group of training spe-
cialists some time ago, I chal-
lenged them to conic up with an
acceptable reason for bothering to
evaluate the training they were
doing. It was an interesting thing
for all of us to see the direction
the brainstorming took. When
honesty rose to the forefront-a
sight to behold among trainers-
most agreed that the main reason
was pressure from the organiza-
tion. Each had as idea that, deep
down inside, lie or she could really
tell how good tl c instructing was
without going through any formal
evaluation activity. "But," came
the conclusion. "management
doesn't seem to be satisfied with
that kind of ev