Intelligence Analysis and Professional Journal Publishing1 by Anthony C.

Cain A Paper Presented at Brunel University Conference on Understanding and Improving Intelligence Analysis 12-13 July 2012, RAF Club, London, UK Good morning and thank you to Stephen Marrin and Brunel University for hosting this conference on a topic that is one of the foundations of success in foreign policy, military strategy, and current operations. Let me qualify my remarks by establishing that I am not, nor have I ever been, a member of the Intelligence Community (IC). Years ago as a B-52 aircrew member I was a consumer of intelligence at the tactical level, I have attended many intelligence briefings, and I have even been friends with officers who will admit to being part of the often maligned IC, but I have never been charged with the responsibility of producing intelligence products or performing intelligence analysis. So, when I received Stephens call for proposals my first impulse was that I have little to say on the topic. Anticipating a theme of a panel later today, I experienced some of the daunting tasks that our IC confronts when I worked as a historian on the overwhelming amount of data in the French Air Force archives in Paris. As I read the theme of the conference, however, I saw that the organizers were looking for perspectives from outside the normal IC disciplines. This, of course, conferred upon me license to leverage my experiences as a scholar to propose what may be done to help our sometimes beleaguered colleagues in the IC provide today's policy makers, strategists, and operators a

The views expressed in this paper are those of the author and do not reflect the official policy or position of the US government, the Department of Defense, or Air University.

Page 1 of 12

better, more relevant product from the increasingly vast amounts of data that the community collects and processes. My perspective derives from my experiences as Editor-in-Chief of two of the US Air Forces quarterly professional journals. The first was Air and Space Power Journal (ASPJ) which I led from 2002-2004. The second was Strategic Studies Quarterly (SSQ) where I was appointed the founding editor in 2007 and served until 2010. To frame my views, let me first say that leading a service-sponsored journal involves different tasks and perspectives from those found in for-profit publications. The relative freedom provided from worrying about subscription rates, advertising, and a host of other business practices allows one to focus solely on the journal's content quality. And it is there that I would like to describe how Intelligence Analysis may benefit from the discipline of journal publishing. Let me begin by citing an Air Force doctrinal definition of the topic. AF Doctrine Document 2-0, Global Integrated Intelligence, Surveillance, and Reconnaissance Operations, defines intelligence analysis and production as "The conversion of processed information into intelligence through the integration, evaluation, analysis, and interpretation of all-source data and the preparation of intelligence products in support of known or anticipated user requirements."2 To avoid the problem associated with using a word or concept to define itself, I'll eliminate the "analysis" component of the Air Force's definition and deal only with the "integration, evaluation, and interpretation" activities associated with intelligence analysis. In short, these aspects imply a process of breaking information down into component parts, describing the relationship among the various parts, and then placing the information into a larger context within which something useful can be done with the information.

United States Air Force. AF Doctrine Document 2-0, Global Integrated Intelligence, Surveillance, and Reconnaissance Operations 6 Jan 2012, 50.

Page 2 of 12

This process of "sense-making" describes how preparing content for publication mirrors that of intelligence analysis. Many of the articles journal editors receive come to us in raw formoften devoid of context other than an author's enthusiasm for seeing the material appear in a publication. From time-to-time, authors will have clearly identifiable agendas and are eager to use the journal to advance that agenda. Finally, the vast majority of the content that editors recieve falls into the category of "noise;" articles that fail for one reason or another to meet publication standards or the editorial focus of the journal, but that nevertheless must be sorted, analyzed, evaluated, and dealt with. No single individual can be expected to have the knowledge and insight required to deal with the volume and variety of articles from a position of expertise just as no single IC individual or organization can deal with the much larger volume of data that confronts them as they attempt to produce valid and actionable intelligence products. One mechanism that has evolved to help the IC cope with this proliferation of data is the identification of specialized disciplines, the "ints" (i.e., ELINT, COMINT, MASINT, etc.) devoted to collecting, sorting, and analyzing clearly distinct forms of data. This is, of course, helpful and necessary, but it may also create stovepipes that serve as barriers to effective analysis. The same kind of stovepiping can occur in the journal business. No editor can possess the expertise required to evaluate all articles in a comprehensive fashion. The best one can hope for is to develop a fair degree of generalized insight over time that allows one to judge the merits of article submissions in broad terms. I have used several layers of analytical mechanisms to overcome this problem that also plagues the IC. First, I have relied on the stovepipes to provide an initial assessment of an article's value. In other words, I chose subject-matter expert referees to read the articles and to provide feedback to me and to the author. Second, I went back to the

Page 3 of 12

generalist approach by convening a group of scholars from a wide variety of disciplines to evaluate and discuss articles that had passed through the referee process. Finally, to manage the strategic direction of the journal, I relied on a group of senior mentors who were less concerned with the details of individual articles than they were with the effectiveness of the journal in accomplishing its stated objectives. The most important characteristic for each one of these analytical support mechanisms is to ensure thorough feedback between the editor, authors, and the other participants in the process. Over the years I have experienced a wide variance in the quality of feedback provided by subject-matter expert referees. As a novice journal editor I was taken aback by the lack of useful feedback that experts provided. Some failed to provide much more in their comments other than "reject," or "don't publish." Others attempted to provide an evaluation of the article's merits, but failed to take the next step which would entail telling the author how to move the article from its present state toward a more publishable piece. The best referees engaged with the authors, provided detailed and specific feedback, and volunteered to participate in improving the articles to ensure they were published. I soon realized that, aside from being the one on the hot seat who had to communicate directly with the authors regarding why I was not publishing their works, most of the problem with referees stemmed from my failure to provide them with specific instructions on what I wanted them to do. It seemed to me that journal articles, at least for the journals I was involved with, should be evaluated according to four specific criteriaquality, support, currency, and relevance. Feedback along these axes should be both general and specific, so I devised a form that would guide referees to provide feedback according to these criteria. There was a section devoted to each one of the four specific areas. Within each section was a scale that ranged from "poor" to

Page 4 of 12

"excellent" that the referee could mark to indicate how effective the article was in terms of that specific area. Below the scale, I placed three columns corresponding to the "pooraverage excellent" markers on the scale. Inside each column I developed a series of bullet statments intended to guide the referee in describing how he or she evaluated the article. For example, in the area of Quality, under the "Excellent" column, the bullets read: Extremely well written & thesis clearly stated early in article. No spelling or grammar errors. Well organized & arguedlittle editing required. Paragraphs have clearly written topic sentences. Paragraphs end with effective and smooth transitionspulls the reader along.

The intent was to provide a framework for the referee to begin evaluating the article while also providing a baseline for more substantial feedback. Finally, each section contains space for more specific comments designed to communicate to the author strengths, weaknesses, and how to improve the article. In practice, I have seen referees take advantage of the descriptive phrases by circling them in each column and using those phrases as a springboard for detailed feedback to the author. This relatively generic and simple tool improved the referee process, gave authors concrete suggestions on how to improve their work, and helped me, the editor, analyze the effectiveness of both referees and articles in a rather efficient manner. A side benefit that emerged after we started using the form was that it gave us baseline for a permanent electronic record of the article's life cycle from inception through the eventual disposition, either publication or rejection, should there be questions or concerns regarding the journal's actions or policies. My policy was to provide feedback to the authorwithout revealing the referee's identityto give the author the opportunity to incorporate the referee's insights and feedback before submitting the article to the contributing editors for further evaluation.

Page 5 of 12

This process could contribute to improving intelligence analysis by providing a way to consolidate evaluations of initial analysis into a single report before proceeding with further analysis. Collectors and analysts could provide both objective and subjective assessments of the reliability and implications of the intelligence data in an efficient, portable, and permanent way. Additionally, the "referees" could be charged with recommending how to improve the quality of and confidence in the intelligence by requesting fusion with other intelligence sources such as COMINT or ELINT. The next step would be to provide that consolidated picture to a group of multi-disciplinary experts. The professional journals I was associated with included content from several disciplines. Air and Space Power Journal focuses on the full range of US Air Force operations. Any topic was fair game as long as it dealt with the operational use of airpower. Referees were very helpful here, and I did not necessarily have to rely on a contributing editors group, although I later learned that the discussions such a group normally will improve the quality of almost any publication. Strategic Studies Quarterly focuses on academic, military, and government professionals who are interested in national security strategy and international affairs. Developing content for these communities that is timely, credible, and relevant is a much more complex task. Consequently, in addition to the referee process described above, I formed a contributing editors group that meets regularly to discuss articles informed by the referee's opinions and their own expertise. The contributing editors represent a wide range of academic, military, and policy disciplines. We have had members who were former congressional staffers, full professors at Air University, retired and active military members, and Department of Defense civilians. They usually meet once each month to discuss the articles that the editor has deemed candidates for

Page 6 of 12

publication after working through the referee and revision process. In the vernacular of the IC, these articles represent processed intelligence that is ready for further evaluation. The contributing editors apply another set of analytical lenses to the material to determine its suitability for publication. Because of their focus on the journal's mission and target audiences, they will concentrate less on the mechanics of the articles and more on the effects that the journal seeks to achieve among its customers. The discussions are lively, informed, and at times heatedbut at the end of the day, the group usually reaches a consensus recommendation for the editor on whether or not to publish an article. Building on a referee-like process, I believe the IC could benefit from a contributing editors-like group that includes experts from the various specialties that make up the IC. To some degree, especially at the national level, groups like this exist to analyze various products as they move through the IC. At the operational and tactical levels, however, I believe intelligence analysis that has benefitted from a more rigorous discussion among the specialized communities will improve the reliability of and confidence in the information. Also, the awareness of a particular customer's needs that the group could bring to the table may help focus the analysis in ways that result in more relevant and actionable products. In this way, the IC gains more insight into how intelligence can best serve the immediate operational need rather than continuing to provide volumes of data that commanders have to sort through on their own. The end results could be better intelligence analysis, greater confidence in the IC on the part of operational commanders, and a more responsive intelligence production system. Both Air and Space Power Journal and Strategic Studies Quarterly benefitted from senior mentors who meet annually to discuss the effectiveness of the journal. The meetings usually lasted for an entire day. The morning was devoted to my report on the past year's

Page 7 of 12

accomplishments and a discussion of potential initiatives for the future. The afternoon usually was a free-flowing discussion among the senior mentors regarding how to make the journal more effective over the next 2-5 years. The senior mentors are retired generals, senior civilian academics, airpower analysts and researchers, and former journal editors who have the perspective and vision for expanding the journal's reach while improving its mission effectiveness. The quality of the conversation is impressivenearly every member brings a strategic focus to the conversation. The meetings always result in several ideas that result in new initiatives for the journal and that raise the bar for measuring success. In the intelligence analysis business, a senior mentor group could help commanders and senior leaders focus the efforts of the IC while providing feedback on the success of the analysis effort. If we are to judge by the critiques of intelligence from a wide variety of operations, the IC generally gets it wrong or gets it right too late to be of value to the war fighting forces. This may be due in part to the IC shooting in the dark at a moving target using too many arrows. Armed with a focused analysis requirement and a system designed to deliver products that meet the requirement, the IC may be able to meet the war fighter's needs much more regularly. The value a senior mentor group could contribute is to understand the commander's needs and help shape the IC analytical system to meet those needs. In other words, they would monitor the efficiency, performance, and relevance of the intelligence analysis system and provide recommendations to the war fighter for making strategic adjustments that result in better system-wide functioning. In conclusion, I believe that there may be parallels between publishing the best quality content in professional journals and providing the best possible intelligence analysis. The multilayered approach that I have described provides several methods and opportunities for improving intelligence integration, evaluation, and interpretation across the spectrum of IC activities.

Page 8 of 12

Imposing structure and rigor that focuses on the war fighter's specific and changing requirements will also improve the IC's contributions to overall mission effectiveness. When I was faced with the daunting task and responsibility of publishing content that enhanced professional development among Airmen, I knew that this was not a task for a single individual or for a narrowly focused group. Instead, it demanded broad participation among the communities the journals sought to serve and it demanded several levels of analysis. While I concede that many of these types of activities probably already take place within the IC community, I also believe that those activities can benefit from another look using a different set of analytical lenseswhich after all, is the purpose of this two-day event. Thank you for your kind attention.

Page 9 of 12

Article Evaluation Form See instructions on reverse Referee: Title:

Poorly written, no discernable thesis. Numerous grammar and spelling errors. Weakly organized - an obvious first draft. Paragraphs lack topic sentences. Paragraphs do not end with transitions.

Tracking Number:

Word Count:

Quality: Poor - - - - - - - - - - - - - - - - - - - - - - - - - - - - Average - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Excellent

Well written, but thesis not openly stated early in article. Spelling and grammar generally accurate. Partially polished, but still needs editing. Most paragraphs have topic sentences Paragraphs end with transitions, but may be ineffective or mechanical. Extremely well written & thesis clearly stated early in article. No spelling or grammar errors. Well organized & arguedlittle editing required. Paragraphs have clearly written topic sentences. Paragraphs end with effective and smooth transitionspulls the reader along.

Comments on Quality: [Circle the appropriate statement above; add supporting comments here.] Support: Poor - - - - - - - - - - - - - - - - - - - - - - - - - - - Average - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Excellent
Article filled with unsupported assertions. Lacks sources or sources used out of date. Article has some unsupported assertions. Most sources current and relevant to thesis. Article makes effective argument. Sources reflect most current relevant material.

Comments on Support: [Circle the appropriate statement above; add supporting comments here.] Currency: Poor - - - - - - - - - - - - - - - - - - - - - - - - - - Average - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Excellent
Article based on outdated terms/concepts. Out of step with current strategic issues. Timely, but not ahead of current ideas. Article identifies problems but solutions not clearly articulated. Synthesizes new ideas about how best to employ air, space, and cyberspace power. Develops viable, innovative solutions.

Comments on Currency: [Circle the appropriate statement above; add supporting comments here.] Relevance: Poor - - - - - - - - - - - - - - - - - - - - - - - - Average - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Excellent
Article is not relevant to contemporary strategic issues on air, space, and cyberspace power Article somewhat relevant Somewhat relates history to current issues Article deals with vital air, space, and cyberspace power Expertly presents Airmans perspective on contemporary strategic issues.

Comments on Relevance: [Circle the appropriate statement above; add supporting comments here.] Comments on Other Considerations (Author name recognition, etc.) [Add other comments here; select appropriate Overall Recommendation below.] Overall Recommendation: Reject Revise/resubmit Referee Accept as is

Page 10 of 12

Instructions: Use this form to evaluate articles submitted for publication. After reading the article, rate each assessment area. Make notes as necessary in the space provided for use in feedback to authors and for discussion in the Editorial Board. Make an overall recommendation for how to proceed with the article. At the Editorial Board meeting, combine each members assessments to derive a consensus assessment for the article. The Board will use this information to make a decision about how to proceed with the article. Attach your Evaluation Form to a hard copy of the article and file for two years.

Quality: The journal publishes the highest quality articles that advance the state of the art in air, space, and cyberspace power. While we work with authors to improve their work, not every author meets our quality standards. Characteristics that reflect quality include organization, grammar, presence or absence of a clear thesis, etc. Record your Quality rating and comments in the spaces provided. Support: Although the journal is a professional rather than an academic journal, authors must apply rigorous standards of argument and use of appropriate sources. If the author is a recognized expert in the field covered in the article, we may expect that those credentials allow relatively fewer academic trappings than one would expect to find in a student research paper. The key to evaluating how well the article is supported is to arrive at a judgment regarding the quality of the authors argument and sources. Record your Support rating and comments in the spaces provided. Currency: Articles should address current and future AF needs by articulating innovative ideas that may advance the military profession. Current and forward-thinking articles will address todays important issues and perhaps challenge accepted practices in serious, constructive, and practical ways. One hallmark of forward-thinking articles is the presence of viable proposals for change. Record your Currency rating and comments in the spaces provided. Relevance: The journal addresses air and space power topics of concern to the USAF and emphasizes the operational level of war. Articles about national strategy or general military topics may also be acceptable even if they say little about air, space, and cyberspace power. Articles that focus on air, space and cyberspace power history should glean timely lessons for todays professionals. Record your Relevance rating and comments in the spaces provided. Other Considerations: Author: Author name recognition is one of the least important article evaluation criteria. However, the editorial staff must remain sensitive to opportunities to enhance journal prestige by publishing high-quality articles written by influential authors. The staff also wants to encourage promising young writers such as NCOs and junior officers when practical. Provide a brief comment if you have information about the authors personal or professional qualifications that would help the Editorial Board make a better decision regarding proceeding with the article. Overall Recommendation: The bottom line is to decide how to proceed with the article. Weigh all the evaluation criteria and recommend what to do with the article. Check all applicable boxes. If you see little hope of publishing the article in any form, mark Reject. If you think the article shows promise, but needs substantial revision before you can render a judgment, mark, Revise/Resubmit. If you deem the article ready for review by a subject matter expert referee, mark, Referee. Mark Accept as is if you do not think the article requires a referee. (This may apply to articles by senior officials). For reference refer to article submission.

Page 11 of 12

GUIDELINES FOR REVIEWING ARTICLES

To obtain an objective evaluation of the articles submitted for publication, the editor circulates them among distinguished referees for review and comment. Here are some guidelines to help you in evaluating articles. The editor shapes the professional dialogue of the Air Force officer corps. Our editorial focus is on contributing air, space, and cyberspace perspectives on strategy, foreign affairs, international security issues, and defense policy. This journal is not for tactical/procedural material (how best to fly ones jet in a given circumstance, or site an 81mm mortar for perimeter defense). While those are important issues for functional excellence, they have little bearing on our focus (i.e., should the jet be in a given conflict in the first place, or is ground defense the Air Forces concern). Some articles are historically based. However, historical pieces forwarded only for historical reasons generally fall outside our focus. Any historical article forwarded for review should address the future of warfareespecially related to air and space power. Our target audience includes those interested in the strategic relevance of air, space, and cyberspace power. The authors you review could represent academia, military and government leaders, and the international community. Helpful Hints from the Editor: 1. Notice whether the author proves assertions with appropriate evidence. Articles are not doctoral dissertations, but they should have a reasonable level of academic credibility. Well-researched, documented, balanced, and decisive articles that nurture ideas, especially those relating to air, space, and cyberspace power, will be published ahead of others. 2. Evaluate the authors writing style. Are the ideas presented clearly? Dont bother with grammar, but let us know if you think it seriously hinders the writing. If we decide to publish the article, the AU Press editors will clean up the grammar. Rewriting is a job for the authornot the referee. 3. Sometimes we receive excellent articles that should be published somewhere but fall outside our editorial focus or arent aimed at our target audience. If you know of a periodical better suited for the article, please let us know. 4. Remain objective. Dont take issue with the author on the ideas in the article. The question is not whether you agree with the author, but whether the ideas should be discussed among members of the air, space, and cyberspace community. 5. Finally, let us know what you think about the articleshould we publish, reject, or return it to the author for rewrite? Please send us specific feedback we can send (anonymously) to the author. Your advice and counsel are extremely important to us. Obviously, our editorial staff retains responsibility for making the final publication decision.

Page 12 of 12