Chapter 3: Technology
To make available to analysts commercially developed or customized collaborative integration and analytic tools that will support the best quality and most timely analysis possible, with due consideration to cost and security; to provide analysts and customers seamless access to data, information, and expertise in a total knowledge management environment.
Desired Outcomes:FY 2005
- Fully interoperable collaborative tools deployed across the IC and to key outside experts.
- Collaboration environment shared with collectors to facilitate tasking and feedback.
- Customers have full access to collaborative and knowledge management environments if they desire.
- Digital production allows dynamic updating of a living knowledge repository.
- Object-oriented user interface for all major data stores, fully linked to visualization tools.
- Analytic and cognitive tools for all analysts to organize information and visualize connections.
- Pilot capability for rapid multi-discipline data fusion/integration and dissemination.
- Synchronous tools allow secure collaboration with experts any time from anywhere.
- Fully interoperable data stores make sharing information seamless within the IC.
- A dynamic knowledge base is fully accessible from anywhere at any time by authorized users.
- Knowledge base linkage to collectors with information needs/gaps automatically identified.
- Single search using natural language prioritized responses, with visualization tools.
- Smart push and pull, automated summarization and database population reduces filtering task.
- Cognitive tools will assist analysts in conceptualizing, testing, and substantiating analysis.
- Robust capability for geographic, temporal, and phenomenological near-real-time data fusion, integrated analysis, and dissemination.
Beyond expertise, technology is the most important, but also most expensive, enabler for an analyst. Analytic organizations have invested heavily in IT to provide analysts the tools needed to do their job. The IC has also made a huge investment in secure communications on which both the collection and the analytic and production communities rely.
The Changing Face of Technology
Technological change has accelerated dramatically over the last decade. Although improvements in information technology have helped analysts keep pace with the increasing workload at a time when resources and manpower have declined, they have been a mixed blessing. Many of our administrative and processing functions have devolved to line analysts. One recent study indicated that in one organization nearly 40 percent of an analyst’s workday is spent in activities that do not contribute to finished production. Moreover, the pace of technological change overwhelms many analysts.
Technology is widely viewed as critical for the production of intelligence. Even if past personnel reductions are partially reversed in the future, improved communications and collaborative tools will still be needed to keep pace with the demands of our customers. The Community is already spending significant resources and effort to improve current capabilities and advance towards a true collaborative and technologically enabled workplace. But, much more needs to be done if we are to reach our goal of getting control of our chief resource—knowledge—with the help of emerging knowledge management technologies. To achieve our goal, we need to focus on three distinct, but overlapping, areas:
Collaboration. How can we collaborate to ensure that our systems are interoperable and capable of sharing information in a timely fashion with those with the required expertise? (Underpinning our ability to collaborate is connectivity across elements of the IC and between the IC and its customers.)
Databases. How can we improve the corporate management of resources so that we can adapt more swiftly to changing threat environments by sharing information through interoperable databases that are easy to access and use?
Analytic tools. How can we, as a Community, help our analysts deal with the problem of information overload—a problem that is likely to increase with more collection, collaboration, and access to data? What is the most effective way to cooperate in developing integration and analytic tools to help our analysts better organize and exploit information and produce the best assessments possible?
Collaboration: Pooling Knowledge To Get the Best Answer
From an analytic standpoint, the goal of collaboration is to assemble the right expertise in a timely fashion to provide the best analysis to the consumer—regardless of where in the analytic community the expertise resides. The expectation flowing from the DCI’s Strategic Intent is to deploy tools that will establish a shared, electronic working environment for all communities of interest—including outside experts, industry, and academia—in a “virtual workspace.”
Ongoing Activities to Link the Community Through Technology
Several Community organizations or initiatives, organized at the DCI’s direction over the last two years, are helping NIPB organizations to address the current technology challenges while positioning the IC for the future:
As a Community, therefore, we need to build toward:
A virtual work environment enabled by collaborative tools, data integration tools, policies, and a security framework that allow analysts to share knowledge and expertise.
An environment that connects native desktops across the IC with appropriate security to convey/share knowledge, while linking analysts to collectors, consumers, and outside experts.
Each of the agencies is already aggressively developing collaborative processes and tools to help them work more effectively within their organizations and with larger communities. Most of the technology is based on commercially available platforms and software, allowing them to migrate to an IC environment that is fully interoperable. In addition, the program offices are talking to each other regularly, ting to derive lessons from each others’ experiences. Interoperability testing under the auspices of IC CIO is already underway.
The Community has made a solid start in deploying collaborative tools, but it has a long way to go. It is still at the pilot stage of deployment even in the more advanced communities of interest, and most analysts have limited access to collaborative tools. Fewer than half of the IC analysts have access to any collaborative environment other than a classified version of the Internet. Far fewer regularly use any collaborative environment. Moreover, agencies are still coming to grips with nagging cultural issues and struggling to bring about enterprise-wide deployment. Key security issues and lack of agreement about rules of engagement are still major impediments to collaboration across the IC, let alone with experts outside the IC.
Database Interoperability: Empowering Collaboration
To provide the substantive information necessary to make collaboration work and improve the ability of analysts to access, share, manipulate, and integrate data, the analytic and production community has focused on changing the way it holds and stores data, information, and knowledge. One expectation flowing from the DCI’s Strategic Intent is to develop a data storage system that links the Community’s primary technical collection databases to the Community’s global communication system. Another objective is to provide intelligence to customers faster with new digital products that permit data mining, customer ‘push’ and ‘pull,’ immediate customer feedback and the ability to influence intelligence tasking requirements.
To move us along in this direction, we need to develop, within the context of a knowledge management strategy:
- Secure, reliable access to all intelligence data at any time, from any location for all those with a need-to-know.
- Databases that must be easy to fill, maintain, update, and validate, while avoiding needless duplication and providing a common knowledge base for the entire Community.
Database issues are finally getting the attention they require. Several strategic plans, starting with the DCI’s Strategic Intent, are focusing attention on database issues. The Strategic Intent envisions the Community working collaboratively not only through better connectivity and tools, but also through increased data-sharing. A key element in this effort is to achieve consistent tagging of information. The Intelligence Community Chief Information Officer (IC CIO) recently blessed a metadata standard for intelligence information. /2 More work, however, has to be done to standardize tagging of intelligence content.
This means the Community will need to fundamentally rethink the way it stores and manages its information.
- In the future, the IC will rely on an infosphere, and its secure and classified sub-set, the intelsphere, which is the virtual knowledge repository of authoritative intelligence information, relevant reference material, and resources used to store, maintain, access, and protect this information.
Much of the impetus for these efforts has come from DIA and the Defense Intelligence community. They have been in the forefront of many of these efforts, which contribute to the goal of the Chairman, Joint Chiefs of Staff in his Joint Vision 2010: “attaining a decisive information advantage and achieving a common operating picture.” Defense Intelligence has been extremely active in developing the concepts underpinning knowledge management in order to provide full battlespace visualization to warfighters and military planners.
Database interoperability and knowledge management are still in their infancy and face formidable challenges. The simplified, single interface for data and information that analysts want is still far off in the future. Analysts still face many impediments to access to the information they need. They should not have to master numerous specialized software and tools, for which they lack the time or the inclination. Proven search strategies or analytic techniques should be widely shared and easy to access through some “best practices” mechanism. Analysts also need access to reliable expertise finders and robust directory services. Communities of Interest (COIs) should also not unduly restrict them in searching out needed information and expertise in areas parallel to their own. Visualization and data-mining tools should not require extensive analyst input to function effectively.
Analytic Tools: Coping with Information Overload
The development of analytic and data integration tools will be one of the most important and expensive areas for the analytic and production community. Without such tools, the shrinking analytic work force will have no hope of managing the flood of new intelligence information or shifting smoothly from one crisis or issue area to another. To achieve progress, we need to develop:
- An automated analytic workflow process relying on advanced analytic tools, such as visualization, search and processing, knowledge management and dynamic updating.
- New tools that reveal connections, facilitate analytic insights and deductions and stream- line search by prioritizing information, automatically populating databases, and integrating data.
One of the foremost challenges for analysts today is coping with the daily flood of information. Several analytic tools are currently being deployed to facilitate collaboration and assist in managing this ocean of information within workgroups or communities of interest. These tools allow junior or newer analysts to tap the expertise of knowledgeable senior analysts and colleagues through the sharing of data sources, advanced search parameters and thresholds, and knowledge maps against which searches are run. They also allow the information to be dis- played graphically, facilitating insights or drawing parallels that might otherwise escape notice.
Analytic tools are just beginning to become broadly accessible. Many are being deployed within individual programs, even if they have broader utility. In some cases, security problems are limiting their effectiveness or availability. Like collaborative tools and database efforts, these tools must be user-friendly and not require extensive training. They also must avoid requiring extensive front-end analyst input to function effectively. Finally, they need to be highly adaptable to analyst needs. In addition, further efforts must be undertaken to develop databases, tools, and techniques for rapid multi-INT data fusion/integration, analysis, and exploitation. Large scale databases, advanced algorithms and high-bandwidth communications are key technology enablers for this goal.
Technology has frequently been developed with the analyst in mind without adequate consultation and awareness of line business processes or requirements. As a result, many tools have languished unused. The IC has neither the funds nor the time to develop capabilities for which there is no need or market. Organizations developing or customizing commercial tools need to work intimately with users to assure the best fit of technology to needs, while allowing for revolutionary, not just evolutionary, change.
Investment Strategy: Recommendations and Initiatives
In order to meet the requirements of this priority area, the IC must invest in the following areas:
1. Collaboration. Collaboration will require a consistent and costly effort to deploy collaborative tools and focus on overcoming cultural and business process obstacles. We need to link analysts to collectors, customers, and forward-deployed analysts in a collaborative environment by 2005.
- Establish an IC collaboration center, under an executive agent, with contractor and Intelligence Community staffing, to lay out a road map to move from pilots to enterprise and IC-wide deployment. The center will focus on integrating programs, technology, improved processes, and human resources across the enterprise to meet the challenges of federated, knowledge management in a collaborative environment; mapping, testing, and recommending improvements to community analysis and production processes in key business areas; identifying metrics and codifying best practices; and, facilitating the integration of advanced analytic tools and methods into production processes.
- Fund and study additional collaboration pilots through FY 2003, with a view to migrating toward common IC standards that will allow interoperability.
- Pursue more extensive interoperability testing of current tools and identify a strategy for providing IC-wide collaboration by FY 2003.
- Advance security issues to enable collaborative analysis during FY 2001.
- Deploy collaborative tools within NIPB programs to create a critical mass of experienced users. By FY 2003, tools should be available on the desktops of all analysts in the large national analytic agencies, and should be available to all NIPB analysts by FY 2005.
2. Databases. The Intelligence Community needs to start immediately reducing cultural and technology barriers to data sharing. NIPB and Community organizations are already taking several initiatives to advance database interoperability.
- Commit NIPB organizations to interoperable database development and line up their participation in an IC database forum. Establish a baseline inventory of databases critical for analysts to access and share.
- Identify some pilot efforts to improve sharing of data.
- Form a working group to frame an NIPB-wide approach to addressing the resource issue of coping with new data sources, both classified and open source.
- Identify the National Intelligence Council (NIC) as a community testbed for digital production processes on Community products beginning in FY 2001. Candidates include the full range of NIC products.
- Support IC efforts to make finished intelligence and database information more accessible.
- Endorse the Defense Intelligence community’s efforts to improve fill rates and make military databases easier to use and access across the IC. Look into how to integrate similar efforts at NSA and CIA to make sure the most comprehensive and up-to-date information is accessible across the NIPB.
3. Analytic Tools. Deploying analytic tools to help analysts deal with the flood of new and existing sources of information by 2005 will require an expensive and focused effort. To manage this process the NIPB needs a plan that includes the following elements.
- Identify executive agents for key technology efforts to conserve resources and reduce stovepipe approaches over the FYDP. This step needs to be taken immediately to better coordinate efforts in an area where many informal exchange mechanisms now exist.
- Designate the new IC collaboration center as a central clearinghouse for efforts in tool development and deployment and for lessons learned. Include an approach to accelerate analytic tool deployment in a road map that the cell will develop for IC-wide efforts in FY 2001.
- Focus the IC’s research and development (R&D) strategy on supporting analytic tool requirements, providing a study in late 2000 on how the commercial sector deals with analogous problems, and suggesting some lessons learned that would apply to the IC.
- Conduct a study in FY 2001 of the level of effort required for analytic tools over the next ten years, to be incorporated into the collaboration road map in time for inclusion in FY 2003 budgets.
- Develop a seal of approval program to encourage the use of commercial off-the-shelf (COTS) rather than government off-the-shelf (GOTS) whenever possible.
- Provide adequate funds through FY 2010 to support full-scale deployment of highly capable analytic tools, many not yet developed for the commercial sector, that will be needed to search the huge volume of existing data stores and new sources that will be arriving at the analysts’ workstations.
- Support and expand technological innovation associated with data fusion algorithms and processes across all collection and analytic disciplines.
- Closely track the new tools and concepts that will emerge from the R&D sector and work closely with this sector, helping to steer its current efforts.
2. Metadata tagging is critical because it will provide a consistent referencing system for intelligence information. Metadata is data about data. It includes information such as date of distribution, classification and other security restrictions, source of information, date of information. It can also provide information about content of the information, although considerable work needs to be done to achieve consistency in this area.