ENCLOSURE 2 RECOMMENDED GUIDANCE FOR EVALUATION OF THE COMMUNITY ON-LINE INTELLIGENCE SYSTEM (COINS) EXPERIMENT

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP80B01139A000100060008-9
• Release Decision: RIFPUB
• Original Classification: S
• Document Page Count: 7
• Document Creation Date: December 19, 2016
• Document Release Date: November 10, 2005
• Sequence Number: 8
• Content Type: REGULATION

Approved F%
2elease 2006/02/06~._aCIA-RDP$0801` 4000100060008-9
2
Recommended C~_ .dance for Evaluation
Of u o
Community Cn-lines i e.~ce System (CCU :~
ExperI1"C'ont
C\70101.) " c im;:~1.c c
-'
i i cn 1, interaacnc an- ~ ccr:; n .:r hascc
--- .,.cation system
at the Tl.Vc) S7 t y `C-v--
How i t e chi ey- ment of this obj - c_
a.
(1) The y'?__-5t ?7a~'~ of thy' 0b-fcCt_'/`
p rtia ly achieved; _`-.0., the system has
hardware ins `l? ed a .u co _nected by means
a
c' ed co.rmlunica- on c-_rcui :.s , and some ?int"
riles are ava'- c in some ag'enc-es ._o
i`,ote iT?rro a C: Sc_ at all pa tic;
., .e . all elements of ti=c- arc function-L:n~~ :'
is elanned and users can fficie_:~_~> access any CC=:h S
in _'easonable time. during _easonable period o` each
_.._ng day, the first _ a:,-- t Will have been act ievee.
(2) A Test and analysis Plan has been
_, woved to co" lee' and routinely publish statistics
._ n
/_7stem utilization and performance, and user iens
ac attitudes to%,7ards CC"~YS.
(3) The _e-H_.i.h_-__-. r of the system. is
-pendent upon its abil_- to operate securely ar_:
e. ..s Irate tr c t - CLve . _ C violations of "-ee d c.
c _. be limited to an acc ta:blc level.
b. What- cor.st .:.':~ uccess or failure
i ver:ent o_ this oa j ad:-,:iye
(1) Success of this objective will
c..._ the technical c.apa?Dof she system to G
erely reli ably and cif _c_n tly .
(2) Failure vii. 11 result from ineffic
_-ation- or lack of re'-a.abi~ity of the n -tworti, e :: cs
-time cr excessive __~terrc~ation response time.
Approved For Release 20.06/02/06 : CIA-RDP80BOl139A000100060008-9
 Approved For Release 2006/02/06 : CIA-RDP80B01139A000100060008-9
B:ESI.:.CDPY
Available
Approved For Release 2006/02/06 : CIA-RDP80B01139A000100060008-9
 Approved I ReI :;'_2Q0&/02/06 CIA-RDP80B JA000100060008-9
(3) An excess =ve -occurrence of ace idc : -,.
> ila c and violations of nee? to know will const
prima facie evidence of a ailure to achieve the of -
2 ObiC.:C Li the COINS caeabill_ty
~-e _ rnO t?--~ t_`-- -:-; -_~ ---` -
1 i:aCil accessible to /
a . , in one ^,C z C v
n ' - -
es, tin the essential G e ( -C=-
`- L , - agenci nd no c
ien make heiil s ta} for inclusion in a COME-
ci f ._i cs C?ah
ike netw%,or'k.
a.- mow is the ac ieyem n t Df this objective to be
identified and m asured?
1L:l..1Zt V .._
(1) Initially, l.he COINS Experimental network" -
is co figured to facilitate relatively. quick reference o
darn bases maintained in the computer systems of participating
agencies by intelligence analysts it all other a genciss . it
is in ten red that interrogations be short and direct and chat
r_
responses be limited in output volume. A capability _or
transfer of complete files for batch processing operations
will not be available during the experimental phase.
(2) The COINS switch log will maintain data
on file usage, and, in addition, each analyst who queries
a COINS file will provide information on file usage satisfaction
The L t
via the User Log Form. The in-depth interviews of CC_NS
users which each COINS participating agency will conduct
should be particularly fruitful for obtaining data an user
reactions to the COINS concept of accessing files and user
file preferences, in addition, the 1 HC has requested ;.hat
etch of its members provide data on the =types of files
recommended for inclusion in an operational COINS.
(3) After determining by experience and interviews
that a given file is suitable for an on-line interagency
system, the question is: Can at characteristics
be identified and assigned meaningful values against which
to measure the suitability of proposed new files wittout
resorting to actual on-line retrieval experience with
t_' a new files? if possible, an attempt should be made
.l:idway during the Test and Analysis Pnasc to identify
those files that appear to be suitable for interagency
use and to evaluate the characteristics which determine
that suitability. Some of the factors to be examined
are: subject and area coverage, sources of information
data elements included, size of each record, date span,
update frequency, number of users by agency (potential
2
Approved . For Release 2006/02/06: CIA-RDP80BO1139A000100060008-9
 Approved F, elease2006/02/06:: blA.-RDP86B01 000100060008-9
?1.: ,.~~,._:i for a proposed _-_-..::`. ''G:"' :onsc time require d i_'
users, alternative moiicds Of data exchange, etc.
J
success or rciJ 1t1Y.i-,
ol-
(1) i'hC CJ 1J C;CL 1VC fill be successful the
ac,-IS-73 to inform r.t tio- through
\,C is re,"1SOT71ably ei-~ L a r,: G'eSLllt:, in gG:1C'! ci_
CO _1
savorablo User reaction whoa con ::ra sled with the :"??O ('e
convcnt.lo::ial TZnethods of data retrieval and exchanZ,ie.
(2) This cbjeC~1va 'lii_l be suCCe:=sfully me
ai set of f Iles can be identified w :icn would have _"3ig h
utility in an 1 i eragency On-ill':e Sys' .C i. This Coul d
Ci.t"tc ' judgcments that can efficient COINS-like sy stem
containing more suitable data, bases would have hig r er
utility.
-
(3) Experience with other on-line information
rotrieval Systems iias sOwn that with an increase
efficiency and experience gained by users ~_ncrea :'c c021_~
denco is realized and .:!ore specific requirements come in-_t0
focus. If the machani sm provided by COINS management for
data base modification (CONS Op erational Procedure No.
1-1968, 22 November 1968) is utilized as a result of user
interest files of greater utility may result.
3. Objective To provide practical training, and. :c'::' - _
c_ ^~e= i ence for _ntelli_genc` analysts in the use of to-ate ~ c
o'-line ln' ell1~ ?nc 1 et'_ ieyc sys tem.
a . How is the achievement of this ob jj ect_ve - tc
be identified and measured?
(1) Each COINS participating agency has
conducted, is conduct.nc; , and will conduct training
s e ... s; ; -~ G I C IBS u'
.. O_ r s f 0l pOte n t - s ors . Each has
produced written training materials describing. how to use
various portions of the COINS network. The COINS Er_viron-
rneat Report, which the Test and Analysis Panel has asp_ed
each COINS participant to prepare, will detail the r,easures
taken by, each agency toanue.res t analysts in the use of
COINS and the level and detail of training provided.
3
Approved For Release 2006/02/06 : CIA-RDP80801139A000100060008-9
 Approved Felease .0066/02/06'3'tIA-RDP8t B0' > 4000100060008-9
(2) PCriodi " 1. C.)orts from the COINS P r:c1 wi11. pr.Ovi'Cc ?nfor~.,.:. :10n on training aCtlVltie ,
status and progress.
(3) In-C?-p ~ ? __i. e views of users which
p c"":iino by the Test and Panel will -r-r'ovide i.:ic 1 g.'it
' ?t:O thdegree of r ln.i_ provided, expert nee gained
:zci additional training tired.
b What CO1?: ' l_i. '. ;iiccn:Ss or failure in the
Ci7 ey`1;`en i Ole ~i s cb E`C iyc!
The object.ive_ ~w?ill 'be succes`:fully achieved -
i_ a rcpresentat1Ve _oup of analvsts in each COINS
pc ~rticiput i ng agency J is able :.v query COINS f i! es w --.--h
a minimum of difficult\'. in addi't'.On, those who are
so trained must exercise, the system sufficient-ly to
gain a level of e:i7Cri.erce which would permit their
evaluation of system utility.
4. Ob-j ective To d :tc:: r.e those elements of the total' syst
a ? gn which encourag e faclli-'Cate an, 9]--%7s-C- u Coo a.--
those elements wnic ~Ctl_SC`L'1"c.CC or hinder use,,-c.
a. How is tr_c achievement of this ob,ec -ive so
be identified and
The following factors must be e* amil_cc ~c
determine the threshold Between encouraging and discouraging
analyst use. For example, analysts night be encouraged
to use a system that was h ignly reliable but would be
discouraged from using a system that was not reliable.
(1) Suitability of data bases.
(2) Number, ty3oe, location and ease of
operation of remote terminals.
(3) Query languages: one vs. many, ab'brevia
tions/mnemonics versus anguage; query procedures
(i.e., the utilization of preprogrammed queries vs. use
of an on-line query language).
(4) Response time
(5) System reliability
(6) System availability. (days and hours)
(7) Usability and readability of outputs
Approved For Release 2006/02/06 : CIA-RDP80BOl 139A000100066008-9
 Approved F elease-2006Fb2/06a UA=RD080B01WA000100060008-9
(g) Typo end amount of user training
(9) Cur. rcncy of data bases
( 1 0 ) lntcr c'`~ivc vs. non- ir_teractive procE
(11) Type and c;uali t.v' of analyst aids
(andbooks, on-line {i]_? and cuery descriptions, etc.;
b. What cons :1te succe--s or. failure in thc-
ac. hievement of this O>~` ^tiye
This objective ill be successfully met iy
coon c cncy is consclcotious in its Collection of Gates: c n
user reactions to COINS. Low user partici,pation in CG71Z
could result in inadeE uate data. collection.
5. Obi ectve To CO~.lec t.l.e necessary data n i1S :'Gc' , 1
s ., c:ra c f _f is i_cncv rzr_d 1 -aye ii.v and user reaction to
i:'.7_`ia;t i.: an evaluation o"' the feasibility and i-,4 utility Of
O'; J:aticnaJ. iripicm~ontacion of as follow-on Cva-~C-~_
sys tCe1.
a. How is the achievement. of this objective to
be identified and meas._ed? T
The Test and Analysis Panel- has deve' ooed a
Plan, procedures, and forms for Coll ectinq the desired data
and this "plan has been p';eroved by all COINS 'part, i cipan is .
However, it should be constantly reviewed (monthly a t
least) during the T&A -oer i od to determine if additions., .
deletions, or modifications are necessary.
the
-b. What constitutes success or failure in
achievement of this ob j ec'4iv e?
(l) If the data collection plan is fol-lowed as
recommended, and if t.here is sufficient usage of the t-_
work to _ubtain_ the desired user reaction_ then this objective
will be partially or fully met. Partial failure could
result from gaps in the data collection plan which were
not anticipated' during the planning stage-.
(2) This objective will be met if the data
collected during the o-perati onai test shows conclusive l \,
that an operational CONS network is or is not needed a
this time. -
6. Objective Determine the elemen is and
w` C
would he critical to the success of a full sy
CO~rS1~ri~ ar~l~n ~ d~ R/Ey lfl N1 iv Ci
s .eclf lcati ons for an o ~erationa! system.
 Approved Release ~Q06J62106_.=IA~RDP80801 00Q100060008-9
a. How is the ?:i_c_,;~racnL of this objc:c.
-S arc. 'tot, the
If all oche ? o"
be avc: it alb: e to C: 'Velop systom plan fo an
COINS nett ork . In this objective i s v,n i c` c.
in that it is completely "pendent on the satlsfa:.CtlcT'1 e
ala. the' other olbjective s.
b. What co:,.,'Li cress or alilllre in C
V.11;C;
(l} It will suCCCc:CA if suffJ.cient date is
71Ji_ X11 n to deg' ine t ho principal parameters of an o ra
co., munity syste'1m11.
(2) This o?bjcctive could fail ii inadequate
or inconclusive data: is collected in an area critical to
the dcsi. Tn. of an operational system.
Approved For Release 2006/02/06 : CIA-ROP80B0l139A000100060008-9