REPORT OF THE EVALUATION OF UNSOLICITED AEGIS SEARCHES

• Document Type: CREST
• Collection: NGA Records (Formerly NIMA)
• Document Number (FOIA) /ESDN (CREST): CIA-RDP78B05703A000300010015-6
• Release Decision: RIPPUB
• Original Classification: S
• Document Page Count: 14
• Document Creation Date: December 28, 2016
• Document Release Date: December 31, 2003
• Sequence Number: 15
• Publication Date: June 1, 1970
• Content Type: REPORT

Approved For Release 2004/02/11 :
A81305703A000300010015-6
Report of the Evaluation of
Unsolicited AEGIS Searches
June 1970
25X1
Declass Review by NGA
U
G5u?1
E%cIudES irr~ aaiamallc
dor,n~c.r~ia~ 2rd
Approved For Release 2004/02/11 : CIA-RDP7.8B05703A000300010015-6
 Approved For Release 2004/02/11 : CIA-RDFL7 03A000300010015-6
INTRODUCTION
The evaluation of AEGIS, reported in late 1969, indicated that a
significant number of production analysts, engaged in the conduct of
research and the preparation of finished intelligence documents, make
no use of AEGIS and, in some cases at least, are unaware of the ex-
istence of the system. Where AEGIS is known but unused,it is assumed
that the analyst either (a) is completely satisfied with his own files
or (b) is unaware of the capabilities of AEGIS and how to exploit the
system most effectively. The evaluation also revealed that virtually
all production analysts who had authored recent finished intelligence
documents, without AEGIS support, would have found an AEGIS search to
be of value in their research and in some cases of major value.
`These findings strongly suggested that CRS should adopt a more
aggressive role in publicizing AEGIS and informing intelligence analysts
on the capabilities of the system. As a result, CRS has prepared a brief
introduction to AEGIS intended for distribution to intelligence analysts.-
In addition, a number of unsolicited searches, designed to introduce
production analysts to AEGIS, have been conducted. The methods used in
the conduct of these searches, and the results, are described below.
METHODOLOGY
To identify potential research topics for which unsolicited AEGIS
searches might be conducted we obtained advance data sheets for the
Approved For Release 2004/02/11 :CIA-RDI 00 793A000300010015-6.
rn
 Approved for Release 2004/02/11 : CIA-RDP78B05703A000300010015-6
March. 1.970 issue of the DDI Research Projects in Process. This
listing of ongoing research projects was distributed to each of the
five CRS area divisions. Each division selected those projects
appearing to fall within their own area of responsibility and marked
each project according to the following set of codes:
A. AEGIS is known to have been used for this
project or the production anlayst is known
to be an AEGIS user.
B. AEGIS not used, analyst not recognized to
be an AEGIS user, project not really suitable
for AEGIS processing.
C. AEGIS not used, analyst not recognized to be
an AEGIS user, project suitable for AEGIS
processing.
It was from group C that we intended to draw candidate research topics
for the conduct of unsolicited AEGIS searches. A subset of C was marked
by the area divisions as being particularly suitable for AEGIS processing.
The results of this analysis of Research Projects in Process are
presented in Table 1. In all, 419 projects were considered and of these
160 (38%) were known to have made use of AEGIS (i.e., AEGIS is known to
have been used on the project or the responsible analyst is known to have
made use of AEGIS in the past), 114 (27%) were considered unsuitable for
AEGIS and 145 (35%) were considered by CRS staff to be suitable for AEGIS
,processing although no use had been made of the system and the requester
was not recognized to be a user of AEGIS; Of this last group, 45 projects
were considered to be particularly good candidates for AEGIS assistance.
Approved For Release 2004/02/11 : CIA- R05703A000300010015-6
 FLL;1
?0L
Approved For Release 2004/02/11 : CIA-RDP7bET05703A000300010015-6
These figures are highly revealing. Taking a comprehensive
list of DDI research projects, planned or actually in progress as of
March 1970, 27% were found to be of the type that could not be helped
by AEGIS searches (although they might and usually could be assisted
by other CRS resources). Of`the remainder, those current research pro-
jects for which it-is believed that AEGIS could provide help, only a
little over 50% (160) are known to have had the benefit of an AEGIS
3 05
search. Put differently, we could say that AEGIS is only being used
on about one half the occasions it might be used profitably by DDT pro-
duction analysts or that about 35% of DDIresearch roects may
j be
completed without exhausting all available in-house intelligence sources.
Obviously, then, AEGIS could be used much more than it is being used
at present. Therefore, any steps taken by CRS to publicize the system,
and to encourage further use, appear fully justified. The present pro-
ject was intended as one step in this direction.
For each of the 145 research topics identified as being suitable
for AEGIS support, the appropriate CRS area division was asked to con-
tact: the production analyst responsible, describe AEGIS to him (where
necessary) and offer to conduct an AEGIS search on the area of his
research.. If the analyst agreed, an AEGIS search was conducted and the
results delivered to the analyst. If the analyst declined the assis-
tance, the reason for the refusal was recorded. For all searches
actually conducted the user was asked to evaluate the results by
Approved For Release 2004/02/11 : CIA-RU703A000300010015-6
 Approved For Release 2004/02/11 : CIA-8B15703A000300010015-6
completing the evaluation form illustrated in Figure 1. Usually
this was completed by a CRS analyst after discussing the results
of the search with the user.
RESULTS 4
An attempt was made to contact all 145 analysts in the period
March-April 1970. From these contacts :2 searches were actually
conducted. Table 2 summarizes the reasons why no search was carried
out-in the other 103 instances. In 11 cases non-CIA personnel were
responsible for the final research and it was decided to exclude this
group. In two cases the :responsible analyst-could. not be contacted (e.g.,
he was on a tour of duty abroad) and in two further cases it was
discovered that AEGIS support had been provided earlier through another
analyst. After discussion between CRS analyst and-production analyst
it was decided, in six cases, that AEGIS would not after all be very
useful (e.g., the writer was working largely with open source material).
In 26 cases the project in question was either completed or too far
along the line to benefit from an AEGIS search. On the other hand there
were 19 instances in which the project was at a stage too early to
benefit - i.e.., a situation in which the scope of the project had not
been fully formulated. In these instances the analyst contacted in-
dicated that he might request on AEGIS search at a. later date. Six of
the 145 projects had either been cancelled or postponed,indefinitely by
the'time the responsible analyst was contacted. In 17 cases
Approved For Release 2004/02/11 : CIA-RDP78B05703A000300010015-6
S 4ET
 Approved For Release 2004/02/11~':Y f l ltP78B05703A000300010015-6
5 -
the analyst declined ana.AEGIS search on the grounds that his own files
were adequate and in another 14 cases an AEGIS search was refused for
some reason not explicitly specified except that the analyst did not
feel that such a search would be of value to him. Rather complete
records of the results of each contact were made by each area div-
ision. For purposes of illustration, two of these reports are in-
cluded in Appendix 1.
From the 42 searches actually conducted, 31 evaluation forms
had been completed, more or-less fully, by May 22,19.70, at which time
a cutoff was established to allow preparation of this report. The
results of these evaluations are shown in Appendix 2, division by div-
ision, for each of the searches. These data indicate production div-
ision, number of citations retrieved, number or proportion judged relevant,
number or proportion new to the analyst, whether the analyst was pre-
viously aware of AEGIS, his assessment of the value of the search and
an indication of whether or not he will use AEGIS in the future.
These data are summarized in Table 3. From this table it can be
seen that 18 of the search recipients claimed to be previously aware
of AEGIS and 13 (427) did not. Twelve of the searches (40%)* were
judged to be-of major value by the analysts receiving them and 15 of the
searches-(50%) were judged to be of minor value. Thus, of the 31 un-
solicited searches evaluated, 90% were judged to be of some value by the
*We are excluding one search for which no value judgment was obtained.
Approved For Release 2004/02/11 : C RDP78BO5703A000300010015-6
 - 6 -
Approved For Release 2004/02/11 : CIA-RDP78BO5703A000300010015-6
analysts for whom they were conducted. Only three searches (107)
were judged of no value.
.It was recognized at the outset: of the project that the value
of some of these unsolicited searches might be reduced considerably
by the fact that they were conducted at late phases of various re-
search projects - when the responsible analyst had already completed
most of his data gathering. Therefore, the analyst was also asked to
judge how valuable the search would have been if conducted at the very
outset of his project. Viewed in this light, 17 of the searches (577)
were rated of major value, 12 (40%) of minor value and only one of no
value. The "no value" search was unusual in that the analyst judged
the search output against her original research topic although the scope
of the search had been broadened considerably beyond this at her own re-
quest.
Perhaps the most encouraging figure of Table 3 is the one in-
dicating that, of the 31 production analysts surveyed in this way, 29
(94%) indicated that they would be likely to make use of AEGIS in
support of future-research projects.
SUMMARY AND CONCLUSIONS
I feel that this experiment has been of considerable success. A
number of production analysts previously unaware of AEGIS have been in-
troduced to the system in the best possible way - by receiving an AEGIS
Approved For Release 2004/02/11 : CIA-R? 03A000300010015-6
 Approved For Release 2004/02/11 : CIA-PDP78BO5703A000300010015-6
product designed to assist their current research activities. Some
90% of these analysts found the results to be of some value and 40%
found them of major value. If AEGIS searches had been conducted at
the very beginning of each research project, 97% would have been of
value and 57% of them would_possibiy have been of major value. Almost
100% of the production analysts studied indicated that,on the basis of
this introduction to AEGIS services, they would be likely to use the
system in support of future research activities.
The findings of this study confirm those made in the earlier
AEGIS evaluation. The following facts now appear quite evident:
1. A fairly large number of production analysts are not aware
of the existence of AEGIS.
2. A larger number have never made use of the system.
3. When introduced to the system, by means of an unsolicited
search, almost all analysts find the AEGIS product to be of some value
and in some cases of major value.
4. These unsolicited AEGIS searches could be of even greater
value if conducted at earlier stages in the various research projects.
5. There is still a hard core of analysts who have never used
AEGIS and refuse to use the system even when its use is made as easy as
possible for them.
SEGfET
Approved For Release 2004/02/11 : CIA-RDP78BO5703A000300010015-6
 e gcpya F{
irC.
Approved For Release 2004/02/11 : CIA-RDP78B05703A000300010015-6
8
It is. worthwhile considering some of these findings in greater
detail. Table 2 presents a summary of reasons why AEGIS searches were
not conducted in 1.03 research areas for which the system was judged, by
CRS personnel, likely to be able to provide assistance. Some of these
reasons are "legitimate" and others less legitimate. It is obvious that
there is an optimum time at which an AEGIS search should be conducted.
Obviously such a search has little or no value after a research project
is completed or at a point where the analyst is in the final stages of
preparing his report. In 26 of the 103 cases, the project under review
was too far along the line to justify AEGIS support - a legitimate
reason for not conducting a search. Conversely, in 19 cases the research
project was in too early a stage - the analyst had not yet formulated his
requirements well enough to express them in a formal request for assis-
tance. This again is a perfectly valid reason for not conducting a search,
and the analysts involved indicated that they might make an AEGIS request
at a later time.
However, in 17 cases the production analyst refused AEGIS assis-
tance on the grounds that his own files were adequate, while in 14 cases
the analyst refused help without specifying the exact reason. We,tust,
assume that these two cases are identical. That is, in both instances
the analyst contacted felt that he had sufficient sources of infor-
mation available (usually personal files, presumably) without using AEGIS.
~E CI E T
Approved For Release 2004/02/11 : CIA-RDP78B05703A000300010015-6
 Approved For Release 2004/02/11 : CIA-RDP78B05703A000300010015-6
9 -
How legitimate is this? Obviously, in some cases it is perfectly
legitimate - for example, when the requester is working almost
exclusively with data of the type that AEGIS does not include or in-
cludes only very selectively (e.g., open sources). However, I feel
that in some cases the analyst is excessively confident in the com-
pletness of his own files. In the earlier AEGIS study, where 22
searches were conducted after the completion of research projects,
a number of analysts found valuable references in the AEGIS search
that they were not previously aware of.
is quite true that the CRS dissemination system is efficient
enough to reduce the need for retrospective searching quite drastically.
However, even a good dissemination system cannot deliver everything of
interest. The dissemination system must "profile" the interests of an
analyst, or group of analysts, in fairly broad terms. It relies on
foresight; that is, it predicts which documents are likely to be rele-
vant to future research interests of various analysts. Such pre-
dictions can never be 100% correct. Moreover, research interests of
analysts do change somewhat as assignments are altered. It is always
difficult and costly to maintain dynamic interest profiles, constantly
updated with every slight shift of interest.
The dissemination system is effective and many production analysts
'have,'over many months or years, developed very valuable personal files
that will and should be the first source consulted when they begin a
new research project. It is for this reason that AEGIS may be regarded
Approved for Release 2004/02/11: CIA
 Approved-For Release 2004/02/11: CIA-`RD I> B05703A000300010015-6
10 -
as of greatest value to the relatively new production analyst, who
has not yet had time to develop his own personal files, or to the
analyst whose area of responsibility has changed rather suddenly.
However, it appears unwise to assume (as many analysts do) that one's
personal files are complete. Such an assumption could be dangerous -
even one highly pertinent document, uncovered by AEGIS but previously
unknown to the analyst, could significantly alter the research results
and justify the AEGIS search completely. The use of AEGIS does not
require an excessive amount of effort on the part of analysts. Is it
not reasonable that these analysts be required to exhaust all infor-
mation sources before committing their findings to print? Although I
have little doubt that some analysts have very excellent files, I
question the wisdom of 1007 reliance on these files, especially when a
backup central system is readily available. Doubts on the completeness
of personal files are cast by examining the "novelty" factors summarized
in Appendix 2. For example, in the FE/PAC searches, one requester judged
eight of the retrieved items to be of major value, of which two were pre-
viously unknown to him. Another judged 15 of major value, of which two
were previously unknown. Even two major value documents, previously un-
known to the production analyst, could easily justify the conduct of a
machine search. But frequently the value is much greater. In some of
the searches. analyzed, 70-80 major value "new" items were uncovered by
AEGIS. Moreover, it is perfectly legitimate and indeed sensible to con-
a corroborative search - to check on the adequacy of Ine's own files.
duct
Perhaps AEGIS does not turn up anything of
Approved For Release 2004/02/11 : CIA-RDP78BO5703A000300010015-6
u
SITi-f
 I q0" U,
N
Approved For Release 2004/02/11 : CIARQQ7BB05703A000300010015-6
importance that we weren't previously aware of - but at least this
gives us increased confidence that our previous resources were
comprehensivr and that we have not missed anything we should know of.
In the intelligence field such a "fail-safe" mechanism appears well
justified.
In summary, many production analysts do not presently make use of
AEGIS in situations in which the system might well provide considerable
,assistance. These non-users appear to fall into three major groups:
1. Those who do not know that the system exists.
2. Those who vaguely know it exists but have little idea
of its true- capabilities and limitations.
3. Those who rely entirely on their own files and make no
use of the central system even though they are aware of
it.
That AEGIS can be of considerable value to those who previously
made no use of it has been shown conclusively both in this study and in the
earlier evaluation. The problem is partly one of-communication and public
relations. The system must be brought to the attention of all who could
potentially benefit from it. The AEGIS user guide will be one vehicle
for advertising the service. The technique of conducting unsolicited
searches, as reported in this study, is another method. Even though the
actual. group of analysts served in these experiment was small - forty
only - it is likely that many more will be introduced to the system
through contact with those immediately served. We already have several
documented instances of one analyst, served in the present study,
Approved For Release 2004/02/11 : Cl 4105703A000300010015-6
 Approved,For Reuse 2004/02/11 :-C A-RDP78B05703A000300010015-6
referring colleagues to AEGIS for assistance. In other words, any
unsolicited service of this kind will have a certain snowball effect,
with people being influenced second - and third-hand.
.Where should CRS go from here? It could continue to provide a
dynamic service by the conduct of unsolicited searches. In other words,
each issue of Research Projects Min Process could be used to identify
ongoing research and thus to allow the responsible analysts to be con-
tacted. This would increase usage of AEGIS by making users out of
certain analysts who at present do not employ the system. It is a good
public relations approach and has indirect benefits (word-of-mouth
advertising). Presumably, since we are concentrating on analysts who
have not previously used AEGIS,the:number of analysts to be contacted
would be reduced with each succeeding issue of'RPIP until a plateau is
reached -'when virtually everyone engaged in Agency research is aware
of AEGIS except for the new analysts who have recently joined the staff.
In contacting analysts, through the use of RPIP, the entire range of CRS
resources should be advertised and subsequently exploited,AEGIS would be'
exploited in the context of other available resources.
This approach, although valuable, has certain basic limitations.
First, for some research projects, the offer of AEGIS help will be made
too late to be of material assistance. When they first appear in RPIP
some projects are already well. along the road, but it is important that
the AEGIS search be conducted much earlier - as soon as the scope of
SECUET
Approved For Release 2004/02/11 : CIA-RDP78BO5703A000300010015-6
 NOW,
Approved- For Release 2004/02/11 : CIA-RDP78BO5703A000300010015-6
the research has been exactly delineated. Second, this approach has
no effect on the group of analysts that consistently refuse to use
the system either through inertia or supreme confidence in their own
local resources.
AEGIS is a relatively efficient central system providing access,
by computer search, to a fairly comprehensive collection of recent in-
telligence documents. When used by production analysts it is frequently,
although not invariably, found to be of considerable value. Yet not all
analysts are aware of the system or, if aware, make use of it. It would
seem not unreasonable to require every production analyst to exhaust all
available internal sources before completion of a research report or memo-
randum. My own inclination is to recommend that the Agency establish a
mechanism whereby, at the beginning of each new research project,* the
responsible analyst must discuss his information requirements with the
appropriate area analyst on CRS staff. If CRS resources are likely to
be of assistance - whether these resources be AEGIS, biographic files,
installation files, or other specialized file or CIA Library - they
should be exploited at this point. Routinely, the production analyst
should be required to document the fact that relevant CRS resources have
been exploited before his research project is considered completed.
Quite apart from the fact that it is both logical and efficient to
conduct a literature search at the beginning of a research project,
use of AEGIS at this point will allow ample time for,. iterative
searching. That is, the production analyst., on viewing the resu Its
of a preliminary AEGIS search, may be motivated to revise the state-
ment of his need and to make a new request - more specific or more
general - that better describes the true area of his interests.
Approved For Release 2004/02/11 : Cl 05703A000300010015-6
I