Data mining An Introduction to Data Mining Discovering hidden value in your data warehouse

Overview Data mining, the extraction of hidden predictive information from large databases , is a powerful new technology with great potential to help companies focus on the most important information in their data warehouses. Data mining tools predict future trends and behaviors, allowing businesses to make proactive, knowledgedriven decisions. The automated, prospective analyses offered by data mining move beyond the analyses of past events provided by retrospective tools typical of decision support systems. Data mining tools can answer business questions that traditionally were too time consuming to resolve. They scour databases for hidden patterns, finding predictive information that experts may miss because it lies outside their expectations. Most companies already collect and refine massive quantities of data. Data mining techniques can be implemented rapidly on existing software and hardware platforms to enhance the value of existing information resources, and can be integrated with new products and systems as they are brought on-line. When implemented on high performance client/server or parallel processing computers, data mining tools can analyze massive databases to deliver answers to questions such as, "Which clients are most likely to respond to my next promotional mailing, and why?" This white paper provides an introduction to the basic technologies of data mining. Examples of profitable applications illustrate its relevance to todays business environment as well as a basic description of how data warehouse architectures can evolve to deliver the value of data mining to end users. The Foundations of Data Mining Data mining techniques are the result of a long process of research and product development. This evolution began when business data was first stored on computers, continued with improvements in data access, and more recently, generated technologies that allow users to navigate through their data in real time.

Data mining takes this evolutionary process beyond retrospective data access and navigation to prospective and proactive information delivery. Data mining is ready for application in the business community because it is supported by three technologies that are now sufficiently mature:

Massive data collection Powerful multiprocessor computers Data mining algorithms

Commercial databases are growing at unprecedented rates. A recent META Group survey of data warehouse projects found that 19% of respondents are beyond the 50 gigabyte level, while 59% expect to be there by second quarter of 1996.1 In some industries, such as retail, these numbers can be much larger. The accompanying need for improved computational engines can now be met in a costeffective manner with parallel multiprocessor computer technology. Data mining algorithms embody techniques that have existed for at least 10 years, but have only recently been implemented as mature, reliable, understandable tools that consistently outperform older statistical methods. In the evolution from business data to business information, each new step has built upon the previous one. For example, dynamic data access is critical for drillthrough in data navigation applications, and the ability to store large databases is critical to data mining. From the users point of view, the four steps listed in Table 1 were revolutionary because they allowed new business questions to be answered accurately and quickly.

Evolutionary Business Step Question Data Collection (1960s) Data Access (1980s) "What was my total revenue in the last five years?" "What were unit sales in New England last March?"

Enabling Technologies Computers, tapes, disks

Product Providers IBM, CDC

Characteristics Retrospective, static data delivery Retrospective, dynamic data delivery at record level

Relational databases (RDBMS), Structured Query

Oracle, Sybase, Informix, IBM,

Language (SQL), Microsoft ODBC Data Warehousing & Decision Support (1990s) Data Mining (Emerging Today) "Whats likely to happen to Boston unit sales next month? Why?" Advanced algorithms, multiprocessor computers, massive databases Pilot, Lockheed, IBM, SGI, numerous startups (nascent industry) Prospective, proactive information delivery "What were unit sales in New England last March? Drill down to Boston." On-line analytic processing (OLAP), multidimensional databases, data warehouses Pilot, Comshare, Arbor, Cognos, Microstrategy Retrospective, dynamic data delivery at multiple levels

Table 1. Steps in the Evolution of Data Mining. The core components of data mining technology have been under development for decades, in research areas such as statistics, artificial intelligence, and machine learning. Today, the maturity of these techniques, coupled with high-performance relational database engines and broad data integration efforts, make these technologies practical for current data warehouse environments. The Scope of Data Mining Data mining derives its name from the similarities between searching for valuable business information in a large database for example, finding linked products in gigabytes of store scanner data and mining a mountain for a vein of valuable ore. Both processes require either sifting through an immense amount of material, or intelligently probing it to find exactly where the value resides. Given databases of sufficient size and quality, data mining technology can generate new business opportunities by providing these capabilities:

Automated prediction of trends and behaviors. Data mining automates the process of finding predictive information in large databases. Questions

that traditionally required extensive hands-on analysis can now be answered directly from the data quickly. A typical example of a predictive problem is targeted marketing. Data mining uses data on past promotional mailings to identify the targets most likely to maximize return on investment in future mailings. Other predictive problems include forecasting bankruptcy and other forms of default, and identifying segments of a population likely to respond similarly to given events.

Automated discovery of previously unknown patterns. Data mining tools sweep through databases and identify previously hidden patterns in one step. An example of pattern discovery is the analysis of retail sales data to identify seemingly unrelated products that are often purchased together. Other pattern discovery problems include detecting fraudulent credit card transactions and identifying anomalous data that could represent data entry keying errors.

Data mining techniques can yield the benefits of automation on existing software and hardware platforms, and can be implemented on new systems as existing platforms are upgraded and new products developed. When data mining tools are implemented on high performance parallel processing systems, they can analyze massive databases in minutes. Faster processing means that users can automatically experiment with more models to understand complex data. High speed makes it practical for users to analyze huge quantities of data. Larger databases, in turn, yield improved predictions. Databases can be larger in both depth and breadth:

More columns. Analysts must often limit the number of variables they examine when doing hands-on analysis due to time constraints. Yet variables that are discarded because they seem unimportant may carry information about unknown patterns. High performance data mining allows users to explore the full depth of a database, without preselecting a subset of variables. More rows. Larger samples yield lower estimation errors and variance, and allow users to make inferences about small but important segments of a population.

A recent Gartner Group Advanced Technology Research Note listed data mining and artificial intelligence at the top of the five key technology areas that "will clearly have a major impact across a wide range of industries within the next 3 to 5

years."2 Gartner also listed parallel architectures and data mining as two of the top 10 new technologies in which companies will invest during the next 5 years. According to a recent Gartner HPC Research Note, "With the rapid advance in data capture, transmission and storage, large-systems users will increasingly need to implement new and innovative ways to mine the after-market value of their vast stores of detail data, employing MPP [massively parallel processing] systems to create new sources of business advantage (0.9 probability)."3 The most commonly used techniques in data mining are:

Artificial neural networks: Non-linear predictive models that learn through training and resemble biological neural networks in structure. Decision trees: Tree-shaped structures that represent sets of decisions. These decisions generate rules for the classification of a dataset. Specific decision tree methods include Classification and Regression Trees (CART) and Chi Square Automatic Interaction Detection (CHAID) . Genetic algorithms: Optimization techniques that use processes such as genetic combination, mutation, and natural selection in a design based on the concepts of evolution. Nearest neighbor method: A technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 1). Sometimes called the k-nearest neighbor technique. Rule induction: The extraction of useful if-then rules from data based on statistical significance.

Many of these technologies have been in use for more than a decade in specialized analysis tools that work with relatively small volumes of data. These capabilities are now evolving to integrate directly with industry-standard data warehouse and OLAP platforms. The appendix to this white paper provides a glossary of data mining terms. How Data Mining Works How exactly is data mining able to tell you important things that you didn't know or what is going to happen next? The technique that is used to perform these feats in data mining is called modeling. Modeling is simply the act of building a model

in one situation where you know the answer and then applying it to another situation that you don't. For instance, if you were looking for a sunken Spanish galleon on the high seas the first thing you might do is to research the times when Spanish treasure had been found by others in the past. You might note that these ships often tend to be found off the coast of Bermuda and that there are certain characteristics to the ocean currents, and certain routes that have likely been taken by the ships captains in that era. You note these similarities and build a model that includes the characteristics that are common to the locations of these sunken treasures. With these models in hand you sail off looking for treasure where your model indicates it most likely might be given a similar situation in the past. Hopefully, if you've got a good model, you find your treasure. This act of model building is thus something that people have been doing for a long time, certainly before the advent of computers or data mining technology. What happens on computers, however, is not much different than the way people build models. Computers are loaded up with lots of information about a variety of situations where an answer is known and then the data mining software on the computer must run through that data and distill the characteristics of the data that should go into the model. Once the model is built it can then be used in similar situations where you don't know the answer. For example, say that you are the director of marketing for a telecommunications company and you'd like to acquire some new long distance phone customers. You could just randomly go out and mail coupons to the general population - just as you could randomly sail the seas looking for sunken treasure. In neither case would you achieve the results you desired and of course you have the opportunity to do much better than random you could use your business experience stored in your database to build a model. As the marketing director you have access to a lot of information about all of your customers: their age, sex, credit history and long distance calling usage. The good news is that you also have a lot of information about your prospective customers: their age, sex, credit history etc. Your problem is that you don't know the long distance calling usage of these prospects (since they are most likely now customers of your competition). You'd like to concentrate on those prospects who have large amounts of long distance usage. You can accomplish this by building a model. Table 2 illustrates the data used for building a model for new customer prospecting in a data warehouse.

Customers Prospects General information (e.g. demographic data) Proprietary information (e.g. customer transactions) Known Known Known Target

Table 2 - Data Mining for Prospecting

The goal in prospecting is to make some calculated guesses about the information in the lower right hand quadrant based on the model that we build going from Customer General Information to Customer Proprietary Information. For instance, a simple model for a telecommunications company might be: 98% of my customers who make more than $60,000/year spend more than $80/month on long distance This model could then be applied to the prospect data to try to tell something about the proprietary information that this telecommunications company does not currently have access to. With this model in hand new customers can be selectively targeted. Test marketing is an excellent source of data for this kind of modeling. Mining the results of a test market representing a broad but relatively small sample of prospects can provide a foundation for identifying good prospects in the overall market. Table 3 shows another common scenario for building models: predict what is going to happen in the future.

Yesterday Static information and current plans (e.g. demographic data, marketing plans) Known

Today Known

Tomorrow Known

Dynamic information (e.g. customer transactions)

Known

Known

Target

Table 3 - Data Mining for Predictions

If someone told you that he had a model that could predict customer usage how would you know if he really had a good model? The first thing you might try would be to ask him to apply his model to your customer base - where you already knew the answer. With data mining, the best way to accomplish this is by setting aside some of your data in a vault to isolate it from the mining process. Once the mining is complete, the results can be tested against the data held in the vault to confirm the models validity. If the model works, its observations should hold for the vaulted data. An Architecture for Data Mining To best apply these advanced techniques, they must be fully integrated with a data warehouse as well as flexible interactive business analysis tools. Many data mining tools currently operate outside of the warehouse, requiring extra steps for extracting, importing, and analyzing the data. Furthermore, when new insights require operational implementation, integration with the warehouse simplifies the application of results from data mining. The resulting analytic data warehouse can be applied to improve business processes throughout the organization, in areas such as promotional campaign management, fraud detection, new product rollout, and so on. Figure 1 illustrates an architecture for advanced analysis in a large data warehouse.

Figure 1 - Integrated Data Mining Architecture

The ideal starting point is a data warehouse containing a combination of internal data tracking all customer contact coupled with external market data about competitor activity. Background information on potential customers also provides an excellent basis for prospecting. This warehouse can be implemented in a variety of relational database systems: Sybase, Oracle, Redbrick, and so on, and should be optimized for flexible and fast data access. An OLAP (On-Line Analytical Processing) server enables a more sophisticated end-user business model to be applied when navigating the data warehouse. The multidimensional structures allow the user to analyze the data as they want to view their business summarizing by product line, region, and other key perspectives of their business. The Data Mining Server must be integrated with the data warehouse and the OLAP server to embed ROI-focused business analysis directly into this infrastructure. An advanced, process-centric metadata template defines the data mining objectives for specific business issues like campaign management, prospecting, and promotion optimization. Integration with the data warehouse enables operational decisions to be directly implemented and tracked. As the warehouse grows with new decisions and results, the organization can continually mine the best practices and apply them to future decisions. This design represents a fundamental shift from conventional decision support systems. Rather than simply delivering data to the end user through query and reporting software, the Advanced Analysis Server applies users business models directly to the warehouse and returns a proactive analysis of the most relevant information. These results enhance the metadata in the OLAP Server by providing a dynamic metadata layer that represents a distilled view of the data. Reporting, visualization, and other analysis tools can then be applied to plan future actions and confirm the impact of those plans. Profitable Applications A wide range of companies have deployed successful applications of data mining. While early adopters of this technology have tended to be in information-intensive industries such as financial services and direct mail marketing, the technology is applicable to any company looking to leverage a large data warehouse to better manage their customer relationships. Two critical factors for success with data mining are: a large, well-integrated data warehouse and a well-defined

understanding of the business process within which data mining is to be applied (such as customer prospecting, retention, campaign management, and so on). Some successful application areas include:

A pharmaceutical company can analyze its recent sales force activity and their results to improve targeting of high-value physicians and determine which marketing activities will have the greatest impact in the next few months. The data needs to include competitor market activity as well as information about the local health care systems. The results can be distributed to the sales force via a wide-area network that enables the representatives to review the recommendations from the perspective of the key attributes in the decision process. The ongoing, dynamic analysis of the data warehouse allows best practices from throughout the organization to be applied in specific sales situations. A credit card company can leverage its vast warehouse of customer transaction data to identify customers most likely to be interested in a new credit product. Using a small test mailing, the attributes of customers with an affinity for the product can be identified. Recent projects have indicated more than a 20-fold decrease in costs for targeted mailing campaigns over conventional approaches. A diversified transportation company with a large direct sales force can apply data mining to identify the best prospects for its services. Using data mining to analyze its own customer experience, this company can build a unique segmentation identifying the attributes of high-value prospects. Applying this segmentation to a general business database such as those provided by Dun & Bradstreet can yield a prioritized list of prospects by region. A large consumer package goods company can apply data mining to improve its sales process to retailers. Data from consumer panels, shipments, and competitor activity can be applied to understand the reasons for brand and store switching. Through this analysis, the manufacturer can select promotional strategies that best reach their target customer segments.

Each of these examples have a clear common ground. They leverage the knowledge about customers implicit in a data warehouse to reduce costs and improve the value of customer relationships. These organizations can now focus their efforts on the most important (profitable) customers and prospects, and design targeted marketing strategies to best reach them.

Conclusion Comprehensive data warehouses that integrate operational data with customer, supplier, and market information have resulted in an explosion of information. Competition requires timely and sophisticated analysis on an integrated view of the data. However, there is a growing gap between more powerful storage and retrieval systems and the users ability to effectively analyze and act on the information they contain. Both relational and OLAP technologies have tremendous capabilities for navigating massive data warehouses, but brute force navigation of data is not enough. A new technological leap is needed to structure and prioritize information for specific end-user problems. The data mining tools can make this leap. Quantifiable business benefits have been proven through the integration of data mining with current information systems, and new products are on the horizon that will bring this integration to an even wider audience of users.

META Group Application Development Strategies: "Data Mining for Data Warehouses: Uncovering Hidden Patterns.", 7/13/95 .
2 3

Gartner Group Advanced Technologies and Applications Research Note, 2/1/95. Gartner Group High Performance Computing Research Note, 1/31/95.

Glossary of Data Mining Terms analytical model anomalous data A structure and process for analyzing a dataset. For example, a decision tree is a model for the classification of a dataset. Data that result from errors (for example, data entry keying errors) or that represent unusual events. Anomalous data should be examined carefully because it may carry important information. Non-linear predictive models that learn through training and resemble biological neural networks in structure. Classification and Regression Trees. A decision tree technique

artificial neural networks CART

used for classification of a dataset. Provides a set of rules that you can apply to a new (unclassified) dataset to predict which records will have a given outcome. Segments a dataset by creating 2-way splits. Requires less data preparation than CHAID. CHAID Chi Square Automatic Interaction Detection. A decision tree technique used for classification of a dataset. Provides a set of rules that you can apply to a new (unclassified) dataset to predict which records will have a given outcome. Segments a dataset by using chi square tests to create multi-way splits. Preceded, and requires more data preparation than, CART. The process of dividing a dataset into mutually exclusive groups such that the members of each group are as "close" as possible to one another, and different groups are as "far" as possible from one another, where distance is measured with respect to specific variable(s) you are trying to predict. For example, a typical classification problem is to divide a database of companies into groups that are as homogeneous as possible with respect to a creditworthiness variable with values "Good" and "Bad." The process of dividing a dataset into mutually exclusive groups such that the members of each group are as "close" as possible to one another, and different groups are as "far" as possible from one another, where distance is measured with respect to all available variables. The process of ensuring that all values in a dataset are consistent and correctly recorded. The extraction of hidden predictive information from large databases. The process of viewing different dimensions, slices, and levels of detail of a multidimensional database. See OLAP. The visual interpretation of complex relationships in multidimensional data.

classification

clustering

data cleansing data mining data navigation data visualization

data warehouse decision tree

A system for storing and delivering massive quantities of data. A tree-shaped structure that represents a set of decisions. These decisions generate rules for the classification of a dataset. See CART and CHAID. In a flat or relational database, each field in a record represents a dimension. In a multidimensional database, a dimension is a set of similar entities; for example, a multidimensional sales database might include the dimensions Product, Time, and City. The use of graphical and descriptive statistical techniques to learn about the structure of a dataset. Optimization techniques that use processes such as genetic combination, mutation, and natural selection in a design based on the concepts of natural evolution. An analytical model that assumes linear relationships in the coefficients of the variables being studied. A statistical technique used to find the best-fitting linear relationship between a target (dependent) variable and its predictors (independent variables). A linear regression that predicts the proportions of a categorical target variable, such as type of customer, in a population.

dimension

exploratory data analysis genetic algorithms linear model linear regression

logistic regression

multidimensional A database designed for on-line analytical processing. database Structured as a multidimensional hypercube with one axis per dimension. multiprocessor computer nearest neighbor A computer that includes multiple processors connected by a network. See parallel processing. A technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 1). Sometimes called a k-

nearest neighbor technique. non-linear model OLAP An analytical model that does not assume linear relationships in the coefficients of the variables being studied. On-line analytical processing. Refers to array-oriented database applications that allow users to view, navigate through, manipulate, and analyze multidimensional databases. A data item whose value falls outside the bounds enclosing most of the other corresponding values in the sample. May indicate anomalous data. Should be examined carefully; may carry important information. The coordinated use of multiple processors to perform computational tasks. Parallel processing can occur on a multiprocessor computer or on a network of workstations or PCs. A structure and process for predicting the values of specified variables in a dataset. Data analysis that predicts future trends, behaviors, or events based on historical data. Redundant Array of Inexpensive Disks. A technology for the efficient parallel storage of data for high-performance computer systems. Data analysis that provides insights into trends, behaviors, or events that have already occurred. The extraction of useful if-then rules from data based on statistical significance. Symmetric multiprocessor. A type of multiprocessor computer in which memory is shared among the processors. One trillion bytes. The analysis of a sequence of measurements made at specified

outlier

parallel processing

predictive model prospective data analysis RAID

retrospective data analysis rule induction SMP terabyte time series

analysis

time intervals. Time is usually the dominating dimension of the data.

Data mining From Wikipedia, the free encyclopedia Not to be confused with information extraction. This article or section reads like an editorial or opinion piece and may require cleanup. Please improve this article by rewriting this article or section in an encyclopedic style to make it neutral in tone. Please see WP:No original research and WP:NOTOPINION for further details. (July 2010) Data mining (the analysis step of the Knowledge Discovery in Databases process, or KDD), a relatively young and interdisciplinary field of computer science,[1][2] is the process of extracting patterns from large data sets by combining methods from statistics and artificial intelligence with database management.[3] With recent technical advances in processing power, storage capacity, and interconnectivity of computer technology, data mining is seen as an increasingly important tool by modern business to transform unprecedented quantities of digital data into business intelligence giving an informational advantage. It is currently used in a wide range of profiling practices, such as marketing, surveillance, fraud detection, and scientific discovery. The growing consensus that data mining can bring real value has led to an explosion in demand for novel data mining technologies.[4] The related terms data dredging, data fishing and data snooping refer to the use of data mining methods to sample parts of a larger population data set that are (or may be) too small for reliable statistical inferences to be made about the validity of any patterns discovered. These methods can, however, be used in creating new hypotheses to test against the larger data populations. Contents

[hide]

1 Background o 1.1 Research and evolution 2 Process o 2.1 Pre-processing o 2.2 Data mining o 2.3 Results validation 3 Notable uses o 3.1 Games o 3.2 Business o 3.3 Science and engineering o 3.4 Spatial data mining 3.4.1 Challenges o 3.5 Surveillance 3.5.1 Pattern mining 3.5.2 Subject-based data mining 4 Privacy concerns and ethics 5 Marketplace surveys 6 Groups and associations 7 See also o 7.1 Methods and algorithms o 7.2 Applications o 7.3 Miscellaneous o 7.4 Commercial data-mining software and applications o 7.5 Free libre open source data-mining software and applications 8 References 9 Further reading 10 External links

[edit] Background The manual extraction of patterns from data has occurred for centuries. Early methods of identifying patterns in data include Bayes' theorem (1700s) and regression analysis (1800s). The proliferation, ubiquity and increasing power of computer technology has increased data collection, storage and manipulations. As data sets have grown in size and complexity, direct hands-on data analysis has increasingly been augmented with indirect, automatic data processing. This has been aided by other discoveries in computer science, such as neural networks,

clustering, genetic algorithms (1950s), decision trees (1960s) and support vector machines (1990s). Data mining is the process of applying these methods to data with the intention of uncovering hidden patterns.[5] It has been used for many years by businesses, scientists and governments to sift through volumes of data such as airline passenger trip records, census data and supermarket scanner data to produce market research reports. (Note, however, that reporting is not always considered to be data mining.) A primary reason for using data mining is to assist in the analysis of collections of observations of behavior. Such data are vulnerable to collinearity because of unknown interrelations. An unavoidable fact of data mining is that the (sub-)set(s) of data being analyzed may not be representative of the whole domain, and therefore may not contain examples of certain critical relationships and behaviors that exist across other parts of the domain. To address this sort of issue, the analysis may be augmented using experiment-based and other approaches, such as choice modelling for human-generated data. In these situations, inherent correlations can be either controlled for, or removed altogether, during the construction of the experimental design. There have been some efforts to define standards for data mining, for example the 1999 European Cross Industry Standard Process for Data Mining (CRISP-DM 1.0) and the 2004 Java Data Mining standard (JDM 1.0). These are evolving standards; later versions of these standards are under development. Independent of these standardization efforts, freely available open-source software systems like the R language, Weka, KNIME, RapidMiner, jHepWork and others have become an informal standard for defining data-mining processes. Notably, all these systems are able to import and export models in PMML (Predictive Model Markup Language) which provides a standard way to represent data mining models so that these can be shared between different statistical applications.[6] PMML is an XMLbased language developed by the Data Mining Group (DMG),[7] an independent group composed of many data mining companies. PMML version 4.0 was released in June 2009.[7][8][9] [edit] Research and evolution The premier professional body in the field is the Association for Computing Machinery's Special Interest Group on Knowledge discovery and Data Mining (SIGKDD).[citation needed] Since 1989 they have hosted an annual international conference and published its proceedings,[10] and since 1999 have published a

biannual academic journal titled "SIGKDD Explorations".[11] Other computer science conferences on data mining include:

DMIN International Conference on Data Mining[12] DMKD Research Issues on Data Mining and Knowledge Discovery ECDM European Conference on Data Mining ECML-PKDD European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases EDM International Conference on Educational Data Mining ICDM IEEE International Conference on Data Mining[13] MLDM Machine Learning and Data Mining in Pattern Recognition PAKDD The annual Pacific-Asia Conference on Knowledge Discovery and Data Mining PAW Predictive Analytics World[14] SDM SIAM International Conference on Data Mining

[edit] Process The CRoss Industry Standard Process for Data Mining (CRISP-DM)[15] is a data mining process model that describes commonly used approaches that expert data miners use to tackle problems. It defines six phases as (1) Business Understanding, (2) Data Understanding, (3) Data Preparation, (4) Modeling, (5) Evaluation, and (6) Deployment.[16] Alternatively, other process models may define three phases as (1) Pre-processing, (2) Data mining, and (3) Results validation. [edit] Pre-processing Before data mining algorithms can be used, a target data set must be assembled. As data mining can only uncover patterns already present in the data, the target dataset must be large enough to contain these patterns while remaining concise enough to be mined in an acceptable timeframe. A common source for data is a datamart or data warehouse. Pre-process is essential to analyse the multivariate datasets before data mining. The target set is then cleaned. Data cleaning removes the observations with noise and missing data. [edit] Data mining

Data mining commonly involves four classes of tasks:[17]

Association rule learning Searches for relationships between variables. For example a supermarket might gather data on customer purchasing habits. Using association rule learning, the supermarket can determine which products are frequently bought together and use this information for marketing purposes. This is sometimes referred to as market basket analysis. Clustering is the task of discovering groups and structures in the data that are in some way or another "similar", without using known structures in the data. Classification is the task of generalizing known structure to apply to new data. For example, an email program might attempt to classify an email as legitimate or spam. Common algorithms include decision tree learning, nearest neighbor, naive Bayesian classification, neural networks and support vector machines. Regression Attempts to find a function which models the data with the least error.

[edit] Results validation The final step of knowledge discovery from data is to verify the patterns produced by the data mining algorithms occur in the wider data set. Not all patterns found by the data mining algorithms are necessarily valid. It is common for the data mining algorithms to find patterns in the training set which are not present in the general data set. This is called overfitting. To overcome this, the evaluation uses a test set of data on which the data mining algorithm was not trained. The learned patterns are applied to this test set and the resulting output is compared to the desired output. For example, a data mining algorithm trying to distinguish spam from legitimate emails would be trained on a training set of sample emails. Once trained, the learned patterns would be applied to the test set of emails on which it had not been trained. The accuracy of these patterns can then be measured from how many emails they correctly classify. A number of statistical methods may be used to evaluate the algorithm such as ROC curves. If the learned patterns do not meet the desired standards, then it is necessary to reevaluate and change the pre-processing and data mining. If the learned patterns do meet the desired standards then the final step is to interpret the learned patterns and turn them into knowledge. [edit] Notable uses

[edit] Games Since the early 1960s, with the availability of oracles for certain combinatorial games, also called tablebases (e.g. for 3x3-chess) with any beginning configuration, small-board dots-and-boxes, small-board-hex, and certain endgames in chess, dots-and-boxes, and hex; a new area for data mining has been opened. This is the extraction of human-usable strategies from these oracles. Current pattern recognition approaches do not seem to fully acquire the high level of abstraction required to be applied successfully. Instead, extensive experimentation with the tablebases, combined with an intensive study of tablebase-answers to well designed problems and with knowledge of prior art, i.e. pre-tablebase knowledge, is used to yield insightful patterns. Berlekamp in dots-and-boxes etc. and John Nunn in chess endgames are notable examples of researchers doing this work, though they were not and are not involved in tablebase generation. [edit] Business Data mining in customer relationship management applications can contribute significantly to the bottom line.[citation needed] Rather than randomly contacting a prospect or customer through a call center or sending mail, a company can concentrate its efforts on prospects that are predicted to have a high likelihood of responding to an offer. More sophisticated methods may be used to optimize resources across campaigns so that one may predict to which channel and to which offer an individual is most likely to respondacross all potential offers. Additionally, sophisticated applications could be used to automate the mailing. Once the results from data mining (potential prospect/customer and channel/offer) are determined, this "sophisticated application" can either automatically send an email or regular mail. Finally, in cases where many people will take an action without an offer, uplift modeling can be used to determine which people will have the greatest increase in responding if given an offer. Data clustering can also be used to automatically discover the segments or groups within a customer data set. Businesses employing data mining may see a return on investment, but also they recognize that the number of predictive models can quickly become very large. Rather than one model to predict how many customers will churn, a business could build a separate model for each region and customer type. Then instead of sending an offer to all people that are likely to churn, it may only want to send offers to customers. Finally, it may want to determine which customers are going to be profitable over a window of time and only send the offers to those that are likely to

be profitable. In order to maintain this quantity of models, they need to manage model versions and move to automated data mining. Data mining can also be helpful to human-resources departments in identifying the characteristics of their most successful employees. Information obtained, such as universities attended by highly successful employees, can help HR focus recruiting efforts accordingly. Additionally, Strategic Enterprise Management applications help a company translate corporate-level goals, such as profit and margin share targets, into operational decisions, such as production plans and workforce levels.[18] Another example of data mining, often called the market basket analysis, relates to its use in retail sales. If a clothing store records the purchases of customers, a datamining system could identify those customers who favor silk shirts over cotton ones. Although some explanations of relationships may be difficult, taking advantage of it is easier. The example deals with association rules within transaction-based data. Not all data are transaction based and logical or inexact rules may also be present within a database. Market basket analysis has also been used to identify the purchase patterns of the Alpha consumer. Alpha Consumers are people that play a key role in connecting with the concept behind a product, then adopting that product, and finally validating it for the rest of society. Analyzing the data collected on this type of user has allowed companies to predict future buying trends and forecast supply demands.[citation needed] Data Mining is a highly effective tool in the catalog marketing industry.[citation needed] Catalogers have a rich history of customer transactions on millions of customers dating back several years. Data mining tools can identify patterns among customers and help identify the most likely customers to respond to upcoming mailing campaigns. Data Mining for business applications is a component which needs to be integrated into a complex modelling and decision making process. Reactive Business Intelligence (RBI) advocates a holistic approach that integrates data mining, modeling and interactive visualization, into an end-to-end discovery and continuous innovation process powered by human and automated learning.[19] In the area of decision making the RBI approach has been used to mine the knowledge which is progressively acquired from the decision maker and self-tune the decision method accordingly.[20]

Related to an integrated-circuit production line, an example of data mining is described in the paper "Mining IC Test Data to Optimize VLSI Testing."[21] In this paper the application of data mining and decision analysis to the problem of dielevel functional test is described. Experiments mentioned in this paper demonstrate the ability of applying a system of mining historical die-test data to create a probabilistic model of patterns of die failure. These patterns are then utilized to decide in real time which die to test next and when to stop testing. This system has been shown, based on experiments with historical test data, to have the potential to improve profits on mature IC products. [edit] Science and engineering In recent years, data mining has been used widely in the areas of science and engineering, such as bioinformatics, genetics, medicine, education and electrical power engineering. In the study of human genetics, an important goal is to understand the mapping relationship between the inter-individual variation in human DNA sequences and variability in disease susceptibility. In lay terms, it is to find out how the changes in an individual's DNA sequence affect the risk of developing common diseases such as cancer. This is very important to help improve the diagnosis, prevention and treatment of the diseases. The data mining method that is used to perform this task is known as multifactor dimensionality reduction.[22] In the area of electrical power engineering, data mining methods have been widely used for condition monitoring of high voltage electrical equipment. The purpose of condition monitoring is to obtain valuable information on the insulation's health status of the equipment. Data clustering such as self-organizing map (SOM) has been applied on the vibration monitoring and analysis of transformer on-load tapchangers(OLTCS). Using vibration monitoring, it can be observed that each tap change operation generates a signal that contains information about the condition of the tap changer contacts and the drive mechanisms. Obviously, different tap positions will generate different signals. However, there was considerable variability amongst normal condition signals for exactly the same tap position. SOM has been applied to detect abnormal conditions and to estimate the nature of the abnormalities.[23] Data mining methods have also been applied for dissolved gas analysis (DGA) on power transformers. DGA, as a diagnostics for power transformer, has been available for many years. Methods such as SOM has been applied to analyze data

and to determine trends which are not obvious to the standard DGA ratio methods such as Duval Triangle.[23] A fourth area of application for data mining in science/engineering is within educational research, where data mining has been used to study the factors leading students to choose to engage in behaviors which reduce their learning[24] and to understand the factors influencing university student retention.[25] A similar example of the social application of data mining is its use in expertise finding systems, whereby descriptors of human expertise are extracted, normalized and classified so as to facilitate the finding of experts, particularly in scientific and technical fields. In this way, data mining can facilitate Institutional memory. Other examples of applying data mining method applications are biomedical data facilitated by domain ontologies,[26] mining clinical trial data,[27] traffic analysis using SOM,[28] et cetera. In adverse drug reaction surveillance, the Uppsala Monitoring Centre has, since 1998, used data mining methods to routinely screen for reporting patterns indicative of emerging drug safety issues in the WHO global database of 4.6 million suspected adverse drug reaction incidents.[29] Recently, similar methodology has been developed to mine large collections of electronic health records for temporal patterns associating drug prescriptions to medical diagnoses.[30] [edit] Spatial data mining Spatial data mining is the application of data mining methods to spatial data. Spatial data mining follows along the same functions in data mining, with the end objective to find patterns in geography. So far, data mining and Geographic Information Systems (GIS) have existed as two separate technologies, each with its own methods, traditions and approaches to visualization and data analysis. Particularly, most contemporary GIS have only very basic spatial analysis functionality. The immense explosion in geographically referenced data occasioned by developments in IT, digital mapping, remote sensing, and the global diffusion of GIS emphasises the importance of developing data driven inductive approaches to geographical analysis and modeling. Data mining, which is the partially automated search for hidden patterns in large databases, offers great potential benefits for applied GIS-based decision-making. Recently, the task of integrating these two technologies has become critical, especially as various public and private sector organizations possessing huge

databases with thematic and geographically referenced data begin to realise the huge potential of the information hidden there. Among those organizations are:

offices requiring analysis or dissemination of geo-referenced statistical data public health services searching for explanations of disease clusters environmental agencies assessing the impact of changing land-use patterns on climate change geo-marketing companies doing customer segmentation based on spatial location.

[edit] Challenges Geospatial data repositories tend to be very large. Moreover, existing GIS datasets are often splintered into feature and attribute components, that are conventionally archived in hybrid data management systems. Algorithmic requirements differ substantially for relational (attribute) data management and for topological (feature) data management.[31] Related to this is the range and diversity of geographic data formats, that also presents unique challenges. The digital geographic data revolution is creating new types of data formats beyond the traditional "vector" and "raster" formats. Geographic data repositories increasingly include ill-structured data such as imagery and geo-referenced multi-media.[32] There are several critical research challenges in geographic knowledge discovery and data mining. Miller and Han [33] offer the following list of emerging research topics in the field:

Developing and supporting geographic data warehouses Spatial properties are often reduced to simple aspatial attributes in mainstream data warehouses. Creating an integrated GDW requires solving issues in spatial and temporal data interoperability, including differences in semantics, referencing systems, geometry, accuracy and position. Better spatio-temporal representations in geographic knowledge discovery Current geographic knowledge discovery (GKD) methods generally use very simple representations of geographic objects and spatial relationships. Geographic data mining methods should recognize more complex geographic objects (lines and polygons) and relationships (nonEuclidean distances, direction, connectivity and interaction through attributed geographic space such as terrain). Time needs to be more fully integrated into these geographic representations and relationships.

Geographic knowledge discovery using diverse data types GKD methods should be developed that can handle diverse data types beyond the traditional raster and vector models, including imagery and geo-referenced multimedia, as well as dynamic data types (video streams, animation).

In four annual surveys of data miners (2007-2010),[34][35][36][37] data mining practitioners consistently identified that they faced three key challenges more than any others:

Dirty Data Explaining Data Mining to Others Unavailability of Data / Difficult Access to Data

In the 2010 survey data miners also shared their experiences in overcoming these challenges.[38] [edit] Surveillance Prior data mining to stop terrorist programs under the U.S. government include the Total Information Awareness (TIA) program, Secure Flight (formerly known as Computer-Assisted Passenger Prescreening System (CAPPS II)), Analysis, Dissemination, Visualization, Insight, Semantic Enhancement (ADVISE),[39] and the Multi-state Anti-Terrorism Information Exchange (MATRIX).[40] These programs have been discontinued due to controversy over whether they violate the US Constitution's 4th amendment, although many programs that were formed under them continue to be funded by different organizations, or under different names.[41] Two plausible data mining methods in the context of combating terrorism include "pattern mining" and "subject-based data mining". [edit] Pattern mining "Pattern mining" is a data mining method that involves finding existing patterns in data. In this context patterns often means association rules. The original motivation for searching association rules came from the desire to analyze supermarket transaction data, that is, to examine customer behavior in terms of the purchased products. For example, an association rule "beer potato chips (80%)" states that four out of five customers that bought beer also bought potato chips.

In the context of pattern mining as a tool to identify terrorist activity, the National Research Council provides the following definition: "Pattern-based data mining looks for patterns (including anomalous data patterns) that might be associated with terrorist activity these patterns might be regarded as small signals in a large ocean of noise."[42][43][44] Pattern Mining includes new areas such a Music Information Retrieval (MIR) where patterns seen both in the temporal and non temporal domains are imported to classical knowledge discovery search methods. [edit] Subject-based data mining "Subject-based data mining" is a data mining method involving the search for associations between individuals in data. In the context of combating terrorism, the National Research Council provides the following definition: "Subject-based data mining uses an initiating individual or other datum that is considered, based on other information, to be of high interest, and the goal is to determine what other persons or financial transactions or movements, etc., are related to that initiating datum."[43] [edit] Privacy concerns and ethics Some people believe that data mining itself is ethically neutral.[45] It is important to note that the term data mining has no ethical implications. The term is often associated with the mining of information in relation to peoples' behavior. However, data mining is a statistical method that is applied to a set of information, or a data set. Associating these data sets with people is an extreme narrowing of the types of data that are available in today's technological society. Examples could range from a set of crash test data for passenger vehicles, to the performance of a group of stocks. These types of data sets make up a great proportion of the information available to be acted on by data mining methods, and rarely have ethical concerns associated with them. However, the ways in which data mining can be used can raise questions regarding privacy, legality, and ethics.[46] In particular, data mining government or commercial data sets for national security or law enforcement purposes, such as in the Total Information Awareness Program or in ADVISE, has raised privacy concerns.[47][48] Data mining requires data preparation which can uncover information or patterns which may compromise confidentiality and privacy obligations. A common way for this to occur is through data aggregation. Data aggregation is when the data are accrued, possibly from various sources, and put together so that they can be analyzed.[49] This is not data mining per se, but a result of the preparation of data

before and for the purposes of the analysis. The threat to an individual's privacy comes into play when the data, once compiled, cause the data miner, or anyone who has access to the newly compiled data set, to be able to identify specific individuals, especially when originally the data were anonymous. It is recommended that an individual is made aware of the following before data are collected:

the purpose of the data collection and any data mining projects, how the data will be used, who will be able to mine the data and use them, the security surrounding access to the data, and in addition, how collected data can be updated.[49]

In the United States, privacy concerns have been somewhat addressed by their congress via the passage of regulatory controls such as the Health Insurance Portability and Accountability Act (HIPAA). The HIPAA requires individuals to be given "informed consent" regarding any information that they provide and its intended future uses by the facility receiving that information. According to an article in Biotech Business Week, "In practice, HIPAA may not offer any greater protection than the longstanding regulations in the research arena, says the AAHC. More importantly, the rule's goal of protection through informed consent is undermined by the complexity of consent forms that are required of patients and participants, which approach a level of incomprehensibility to average individuals." [50] This underscores the necessity for data anonymity in data aggregation practices. One may additionally modify the data so that they are anonymous, so that individuals may not be readily identified.[49] However, even de-identified data sets can contain enough information to identify individuals, as occurred when journalists were able to find several individuals based on a set of search histories that were inadvertently released by AOL.[51]

Data Mining: What is Data Mining? Overview

Generally, data mining (sometimes called data or knowledge discovery) is the process of analyzing data from different perspectives and summarizing it into useful information - information that can be used to increase revenue, cuts costs, or both. Data mining software is one of a number of analytical tools for analyzing data. It allows users to analyze data from many different dimensions or angles, categorize it, and summarize the relationships identified. Technically, data mining is the process of finding correlations or patterns among dozens of fields in large relational databases. Continuous Innovation Although data mining is a relatively new term, the technology is not. Companies have used powerful computers to sift through volumes of supermarket scanner data and analyze market research reports for years. However, continuous innovations in computer processing power, disk storage, and statistical software are dramatically increasing the accuracy of analysis while driving down the cost. Example For example, one Midwest grocery chain used the data mining capacity of Oracle software to analyze local buying patterns. They discovered that when men bought diapers on Thursdays and Saturdays, they also tended to buy beer. Further analysis showed that these shoppers typically did their weekly grocery shopping on Saturdays. On Thursdays, however, they only bought a few items. The retailer concluded that they purchased the beer to have it available for the upcoming weekend. The grocery chain could use this newly discovered information in various ways to increase revenue. For example, they could move the beer display closer to the diaper display. And, they could make sure beer and diapers were sold at full price on Thursdays. Data, Information, and Knowledge Data Data are any facts, numbers, or text that can be processed by a computer. Today, organizations are accumulating vast and growing amounts of data in different formats and different databases. This includes:

operational or transactional data such as, sales, cost, inventory, payroll, and accounting

nonoperational data, such as industry sales, forecast data, and macro economic data meta data - data about the data itself, such as logical database design or data dictionary definitions

Information The patterns, associations, or relationships among all this data can provide information. For example, analysis of retail point of sale transaction data can yield information on which products are selling and when. Knowledge Information can be converted into knowledge about historical patterns and future trends. For example, summary information on retail supermarket sales can be analyzed in light of promotional efforts to provide knowledge of consumer buying behavior. Thus, a manufacturer or retailer could determine which items are most susceptible to promotional efforts. Data Warehouses Dramatic advances in data capture, processing power, data transmission, and storage capabilities are enabling organizations to integrate their various databases into data warehouses. Data warehousing is defined as a process of centralized data management and retrieval. Data warehousing, like data mining, is a relatively new term although the concept itself has been around for years. Data warehousing represents an ideal vision of maintaining a central repository of all organizational data. Centralization of data is needed to maximize user access and analysis. Dramatic technological advances are making this vision a reality for many companies. And, equally dramatic advances in data analysis software are allowing users to access this data freely. The data analysis software is what supports data mining. What can data mining do? Data mining is primarily used today by companies with a strong consumer focus retail, financial, communication, and marketing organizations. It enables these companies to determine relationships among "internal" factors such as price, product positioning, or staff skills, and "external" factors such as economic indicators, competition, and customer demographics. And, it enables them to

determine the impact on sales, customer satisfaction, and corporate profits. Finally, it enables them to "drill down" into summary information to view detail transactional data. With data mining, a retailer could use point-of-sale records of customer purchases to send targeted promotions based on an individual's purchase history. By mining demographic data from comment or warranty cards, the retailer could develop products and promotions to appeal to specific customer segments. For example, Blockbuster Entertainment mines its video rental history database to recommend rentals to individual customers. American Express can suggest products to its cardholders based on analysis of their monthly expenditures. WalMart is pioneering massive data mining to transform its supplier relationships. WalMart captures point-of-sale transactions from over 2,900 stores in 6 countries and continuously transmits this data to its massive 7.5 terabyte Teradata data warehouse. WalMart allows more than 3,500 suppliers, to access data on their products and perform data analyses. These suppliers use this data to identify customer buying patterns at the store display level. They use this information to manage local store inventory and identify new merchandising opportunities. In 1995, WalMart computers processed over 1 million complex data queries. The National Basketball Association (NBA) is exploring a data mining application that can be used in conjunction with image recordings of basketball games. The Advanced Scout software analyzes the movements of players to help coaches orchestrate plays and strategies. For example, an analysis of the play-by-play sheet of the game played between the New York Knicks and the Cleveland Cavaliers on January 6, 1995 reveals that when Mark Price played the Guard position, John Williams attempted four jump shots and made each one! Advanced Scout not only finds this pattern, but explains that it is interesting because it differs considerably from the average shooting percentage of 49.30% for the Cavaliers during that game. By using the NBA universal clock, a coach can automatically bring up the video clips showing each of the jump shots attempted by Williams with Price on the floor, without needing to comb through hours of video footage. Those clips show a very successful pick-and-roll play in which Price draws the Knick's defense and then finds Williams for an open jump shot. How does data mining work?

While large-scale information technology has been evolving separate transaction and analytical systems, data mining provides the link between the two. Data mining software analyzes relationships and patterns in stored transaction data based on open-ended user queries. Several types of analytical software are available: statistical, machine learning, and neural networks. Generally, any of four types of relationships are sought:

Classes: Stored data is used to locate data in predetermined groups. For example, a restaurant chain could mine customer purchase data to determine when customers visit and what they typically order. This information could be used to increase traffic by having daily specials. Clusters: Data items are grouped according to logical relationships or consumer preferences. For example, data can be mined to identify market segments or consumer affinities. Associations: Data can be mined to identify associations. The beer-diaper example is an example of associative mining. Sequential patterns: Data is mined to anticipate behavior patterns and trends. For example, an outdoor equipment retailer could predict the likelihood of a backpack being purchased based on a consumer's purchase of sleeping bags and hiking shoes.

Data mining consists of five major elements:

Extract, transform, and load transaction data onto the data warehouse system. Store and manage the data in a multidimensional database system. Provide data access to business analysts and information technology professionals. Analyze the data by application software. Present the data in a useful format, such as a graph or table.

Different levels of analysis are available:

Artificial neural networks: Non-linear predictive models that learn through training and resemble biological neural networks in structure. Genetic algorithms: Optimization techniques that use processes such as genetic combination, mutation, and natural selection in a design based on the concepts of natural evolution. Decision trees: Tree-shaped structures that represent sets of decisions. These decisions generate rules for the classification of a dataset. Specific decision tree methods include Classification and Regression Trees (CART) and Chi Square Automatic Interaction Detection (CHAID) . CART and CHAID are decision tree techniques used for classification of a dataset. They provide a set of rules that you can apply to a new (unclassified) dataset to predict which records will have a given outcome. CART segments a dataset by creating 2-way splits while CHAID segments using chi square tests to create multi-way splits. CART typically requires less data preparation than CHAID. Nearest neighbor method: A technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 1). Sometimes called the k-nearest neighbor technique. Rule induction: The extraction of useful if-then rules from data based on statistical significance. Data visualization: The visual interpretation of complex relationships in multidimensional data. Graphics tools are used to illustrate data relationships.

What technological infrastructure is required? Today, data mining applications are available on all size systems for mainframe, client/server, and PC platforms. System prices range from several thousand dollars for the smallest applications up to $1 million a terabyte for the largest. Enterprise-wide applications generally range in size from 10 gigabytes to over 11 terabytes. NCR has the capacity to deliver applications exceeding 100 terabytes. There are two critical technological drivers:

Size of the database: the more data being processed and maintained, the more powerful the system required. Query complexity: the more complex the queries and the greater the number of queries being processed, the more powerful the system required.

Relational database storage and management technology is adequate for many data mining applications less than 50 gigabytes. However, this infrastructure needs to be significantly enhanced to support larger applications. Some vendors have added extensive indexing capabilities to improve query performance. Others use new hardware architectures such as Massively Parallel Processors (MPP) to achieve order-of-magnitude improvements in query time. For example, MPP systems from NCR link hundreds of high-speed Pentium processors to achieve performance levels exceeding those of the largest supercomputers.

Memory

Memory
From Wikipedia, the free encyclopedia For other uses, see Memory (disambiguation).

Neuropsychology
Topics[show] Brain functions[show] People[show] Tests[show]

Mind and Brain Portal

vde

Overview of the forms and functions of memory in the sciences

In psychology, memory is an organism's ability to store, retain, and recall information and experiences. Traditional studies of memory began in the fields of philosophy, including techniques of artificially enhancing memory. During the late nineteenth and early twentieth century, scientists have put memory within the paradigm of cognitive psychology. In recent decades, it has become one of the principal pillars of a branch of science called cognitive neuroscience, an interdisciplinary link between cognitive psychology and neuroscience.

Contents
[hide]

1 Processes o 1.1 Sensory memory o 1.2 Short-term o 1.3 Long-term 2 Models o 2.1 Atkinson-Shiffrin model o 2.2 Working memory o 2.3 Levels of processing 3 Classification by information type 4 Classification by temporal direction 5 Physiology 6 Genetics 7 Disorders 8 Methods 9 Memory and aging 10 Improving memory 11 Memory tasks 12 See also 13 Footnotes 14 References 15 External links

[edit] Processes
From an information processing perspective there are three main stages in the formation and retrieval of memory:

Encoding or registration (receiving, processing and combining of received information) Storage (creation of a permanent record of the encoded information) Retrieval, recall or recollection (calling back the stored information in response to some cue for use in a process or activity)

[edit] Sensory memory

Main article: Sensory memory Sensory memory corresponds approximately to the initial 200500 milliseconds after an item is perceived. The ability to look at an item, and remember what it looked like with just a second of observation, or memorisation, is an example of sensory memory. With very short presentations, participants often report that they seem to "see" more than they can actually report. The first experiments exploring this form of sensory memory were conducted by George Sperling (1960) using the "partial report paradigm". Subjects were presented with a grid of 12 letters, arranged into three rows of four. After a brief presentation, subjects were then played either a high, medium or low tone, cuing them which of the rows to report. Based on these partial report experiments, Sperling was able to show that the capacity of sensory memory was approximately 12 items, but that it degraded very quickly (within a few hundred milliseconds). Because this form of memory degrades so quickly, participants would see the display, but be unable to report all of the items (12 in the "whole report" procedure) before they decayed. This type of memory cannot be prolonged via rehearsal.

[edit] Short-term
Main article: Short-term memory Short-term memory allows recall for a period of several seconds to a minute without rehearsal. Its capacity is also very limited: George A. Miller (1956), when working at Bell Laboratories, conducted experiments showing that the store of short-term memory was 72 items (the title of his famous paper, "The magical number 72"). Modern estimates of the capacity of short-term memory are lower, typically on the order of 45 items,[1] however, memory capacity can be increased through a process called chunking.[2] For example, in recalling a ten-digit telephone number, a person could chunk the digits into three groups: first, the area code (such as 215), then a three-digit chunk (123) and lastly a four-digit chunk (4567). This method of remembering telephone numbers is far more effective than attempting to remember a string of 10 digits; this is because we are able to chunk the information into meaningful groups of numbers. Herbert Simon showed that the ideal size for chunking letters and numbers, meaningful or not, was three.[citation needed] This may be reflected in some countries in the tendency to remember telephone numbers as several chunks of three numbers with the final four-number groups, generally broken down into two groups of two. Short-term memory is believed to rely mostly on an acoustic code for storing information, and to a lesser extent a visual code. Conrad (1964)[3] found that test subjects had more difficulty recalling collections of letters that were acoustically similar (e.g. E, P, D). Confusion with recalling acoustically similar letters rather than visually similar letters implies that the letters were encoded acoustically. Conrad's (1964) study however, deals with the encoding of written text, thus while memory of written language may rely on acoustic components, generalisations to all forms of memory cannot be made. However, some individuals have been reported to be able to remember large amounts of information, quickly, and be able to recall that information in seconds.[citation needed]

[edit] Long-term

Olin Levi Warner, Memory (1896). Library of Congress Thomas Jefferson Building, Washington, D.C. Main article: Long-term memory The storage in sensory memory and short-term memory generally have a strictly limited capacity and duration, which means that information is available only for a certain period of time, but is not retained indefinitely. By contrast, long-term memory can store much larger quantities of information for potentially unlimited duration (sometimes a whole life span). Its capacity is immeasurably large. For example, given a random seven-digit number we may remember it for only a few seconds before forgetting, suggesting it was stored in our short-term memory. On the other hand, we can remember telephone numbers for many years through repetition; this information is said to be stored in long-term memory. While short-term memory encodes information acoustically, long-term memory encodes it semantically: Baddeley (1966)[4] discovered that after 20 minutes, test subjects had the most difficulty recalling a collection of words that had similar meanings (e.g. big, large, great, huge). Short-term memory is supported by transient patterns of neuronal communication, dependent on regions of the frontal lobe (especially dorsolateral prefrontal cortex) and the parietal lobe. Longterm memories, on the other hand, are maintained by more stable and permanent changes in neural connections widely spread throughout the brain. The hippocampus is essential (for learning new information) to the consolidation of information from short-term to long-term memory, although it does not seem to store information itself. Without the hippocampus, new memories are unable to be stored into long-term memory, and there will be a very short attention span. Furthermore, it may be involved in changing neural connections for a period of three

months or more after the initial learning. One of the primary functions of sleep is thought to be improving consolidation of information, as several studies have demonstrated that memory depends on getting sufficient sleep between training and test. Additionally, data obtained from neuroimaging studies have shown activation patterns in the sleeping brain which mirror those recorded during the learning of tasks from the previous day, suggesting that new memories may be solidified through such rehearsal.

[edit] Models
Models of memory provide abstract representations of how memory is believed to work. Below are several models proposed over the years by various psychologists. Note that there is some controversy as to whether there are several memory structures, for example, Tarnow (2005) finds that it is likely that there is only one memory structure between 6 and 600 seconds.

[edit] Atkinson-Shiffrin model

See also: Memory consolidation The multi-store model (also known as Atkinson-Shiffrin memory model) was first recognised in 1968 by Atkinson and Shiffrin. The multi-store model has been criticised for being too simplistic. For instance, long-term memory is believed to be actually made up of multiple subcomponents, such as episodic and procedural memory. It also proposes that rehearsal is the only mechanism by which information eventually reaches long-term storage, but evidence shows us capable of remembering things without rehearsal. The model also shows all the memory stores as being a single unit whereas research into this shows differently. For example, short-term memory can be broken up into different units such as visual information and acoustic information. Patient KF proves this. Patient KF was brain damaged and had problems with his short term memory. He had problems with things such as spoken numbers, letters and words and with significant sounds (such as doorbells and cats meowing). Other parts of short term memory were unaffected, such as visual (pictures).[5] It also shows the sensory store as a single unit whilst we know that the sensory store is split up into several different parts such as taste, vision, and hearing.

[edit] Working memory

The working memory model. Main article: working memory In 1974 Baddeley and Hitch proposed a working memory model which replaced the concept of general short term memory with specific, active components. In this model, working memory consists of three basic stores: the central executive, the phonological loop and the visuo-spatial sketchpad. In 2000 this model was expanded with the multimodal episodic buffer.[6] The central executive essentially acts as attention. It channels information to the three component processes: the phonological loop, the visuo-spatial sketchpad, and the episodic buffer. The phonological loop stores auditory information by silently rehearsing sounds or words in a continuous loop: the articulatory process (for example the repetition of a telephone number over and over again). Then, a short list of data is easier to remember. The visuospatial sketchpad stores visual and spatial information. It is engaged when performing spatial tasks (such as judging distances) or visual ones (such as counting the windows on a house or imagining images). The episodic buffer is dedicated to linking information across domains to form integrated units of visual, spatial, and verbal information and chronological ordering (e.g., the memory of a story or a movie scene). The episodic buffer is also assumed to have links to long-term memory and semantical meaning. The working memory model explains many practical observations, such as why it is easier to do two different tasks (one verbal and one visual) than two similar tasks (e.g., two visual), and the aforementioned word-length effect. However, the concept of a central executive as noted here has been criticised as inadequate and vague.[citation needed]

[edit] Levels of processing

Main article: Levels-of-processing effect Craik and Lockhart (1972) proposed that it is the method and depth of processing that affects how an experience is stored in memory, rather than rehearsal.

Organization - Mandler (1967) gave participants a pack of word cards and asked them to sort them into any number of piles using any system of categorisation they liked. When they were later asked to recall as many of the words as they could, those who used more categories remembered more words. This study suggested that the act of organising information makes it more memorable. Distinctiveness - Eysenck and Eysenck (1980) asked participants to say words in a distinctive way, e.g. spell the words out loud. Such participants recalled the words better than those who simply read them off a list. Effort - Tyler et al. (1979) had participants solve a series of anagrams, some easy (FAHTER) and some difficult (HREFAT). The participants recalled the difficult anagrams better, presumably because they put more effort into them. Elaboration - Palmere et al. (1983) gave participants descriptive paragraphs of a fictitious African nation. There were some short paragraphs and some with extra sentences elaborating the main idea. Recall was higher for the ideas in the elaborated paragraphs.

[edit] Classification by information type

Anderson (1976)[7] divides long-term memory into declarative (explicit) and procedural (implicit) memories. Declarative memory requires conscious recall, in that some conscious process must call back the information. It is sometimes called explicit memory, since it consists of information that is explicitly stored and retrieved. Declarative memory can be further sub-divided into semantic memory, which concerns facts taken independent of context; and episodic memory, which concerns information specific to a particular context, such as a time and place. Semantic memory allows the encoding of abstract knowledge about the world, such as "Paris is the capital of France". Episodic memory, on the other hand, is used for more personal memories, such as the sensations, emotions, and personal associations of a particular place or time. Autobiographical memory - memory for particular events within one's own life - is generally viewed as either equivalent to, or a subset of, episodic memory. Visual memory is part of memory preserving some characteristics of our senses pertaining to visual experience. One is able to place in memory information that resembles objects, places, animals or people in sort of a mental image. Visual memory can result in priming and it is assumed some kind of perceptual representational system underlies this phenomenon. [2] In contrast, procedural memory (or implicit memory) is not based on the conscious recall of information, but on implicit learning. Procedural memory is primarily employed in learning motor skills and should be considered a subset of implicit memory. It is revealed when one does better in a given task due only to repetition - no new explicit memories have been formed, but one is unconsciously accessing aspects of those previous experiences. Procedural memory involved in motor learning depends on the cerebellum and basal ganglia.

Topographic memory is the ability to orient oneself in space, to recognize and follow an itinerary, or to recognize familiar places.[8] Getting lost when traveling alone is an example of the failure of topographic memory. This is often reported among elderly patients who are evaluated for dementia. The disorder could be caused by multiple impairments, including difficulties with perception, orientation, and memory.[9]

[edit] Classification by temporal direction

A further major way to distinguish different memory functions is whether the content to be remembered is in the past, retrospective memory, or whether the content is to be remembered in the future, prospective memory. Thus, retrospective memory as a category includes semantic, episodic and autobiographical memory. In contrast, prospective memory is memory for future intentions, or remembering to remember (Winograd, 1988). Prospective memory can be further broken down into event- and time-based prospective remembering. Time-based prospective memories are triggered by a time-cue, such as going to the doctor (action) at 4pm (cue). Eventbased prospective memories are intentions triggered by cues, such as remembering to post a letter (action) after seeing a mailbox (cue). Cues do not need to be related to the action (as the mailbox example is), and lists, sticky-notes, knotted handkerchiefs, or string around the finger are all examples of cues that are produced by people as a strategy to enhance prospective memory.

[edit] Physiology
Brain areas involved in the neuroanatomy of memory such as the hippocampus, the amygdala, the striatum, or the mammillary bodies are thought to be involved in specific types of memory. For example, the hippocampus is believed to be involved in spatial learning and declarative learning, while the amygdala is thought to be involved in emotional memory. Damage to certain areas in patients and animal models and subsequent memory deficits is a primary source of information. However, rather than implicating a specific area, it could be that damage to adjacent areas, or to a pathway traveling through the area is actually responsible for the observed deficit. Further, it is not sufficient to describe memory, and its counterpart, learning, as solely dependent on specific brain regions. Learning and memory are attributed to changes in neuronal synapses, thought to be mediated by long-term potentiation and long-term depression. Hebb distinguished between short-term and long-term memory. He postulated that any memory that stayed in short-term storage for a long enough time would be consolidated into a long-term memory. Later research showed this to be false. Research has shown that direct injections of cortisol or epinephrine help the storage of recent experiences. This is also true for stimulation of the amygdala. This proves that excitement enhances memory by the stimulation of hormones that affect the amygdala. Excessive or prolonged stress (with prolonged cortisol) may hurt memory storage. Patients with amygdalar damage are no more likely to remember emotionally charged words than nonemotionally charged ones. The hippocampus is important for explicit memory. The hippocampus is also important for memory consolidation. The hippocampus receives input from different parts of the cortex and sends its output out to different parts of the brain also. The input comes from secondary and tertiary sensory areas that have processed the information a lot

already. Hippocampal damage may also cause memory loss and problems with memory storage.[10]

[edit] Genetics
Study of the genetics of human memory is in its infancy. A notable initial success was the association of APOE with memory dysfunction in Alzheimer's Disease. The search for genes associated with normally-varying memory continues. One of the first candidates for normal variation in memory is the gene KIBRA[11], which appears to be associated with the rate at which material is forgotten over a delay period.

[edit] Disorders
Much of the current knowledge of memory has come from studying memory disorders. Loss of memory is known as amnesia. There are many sorts of amnesia, and by studying their different forms, it has become possible to observe apparent defects in individual sub-systems of the brain's memory systems, and thus hypothesize their function in the normally working brain. Other neurological disorders such as Alzheimer's disease and Parkinson's disease [12] can also affect memory and cognition. Hyperthymesia, or hyperthymesic syndrome, is a disorder which affects an individual's autobiographical memory, essentially meaning that they cannot forget small details that otherwise would not be stored.[13] Korsakoff's syndrome, also known as Korsakoff's psychosis, amnesic-confabulatory syndrome, is an organic brain disease that adversely affects memory. While not a disorder, a common temporary failure of word retrieval from memory is the tip-ofthe-tongue phenomenon. Sufferers of Nominal Aphasia (also called Anomia), however, do experience the tip-of-the-tongue phenomenon on an ongoing basis due to damage to the frontal and parietal lobes of the brain.

[edit] Methods
Methods to optimize memorization Memorization is a method of learning that allows an individual to recall information verbatim. Rote learning is the method most often used. Methods of memorizing things have been the subject of much discussion over the years with some writers, such as Cosmos Rossellius using visual alphabets. The spacing effect shows that an individual is more likely to remember a list of items when rehearsal is spaced over an extended period of time. In contrast to this is cramming which is intensive memorisation in a short period of time. Also relevant is the Zeigarnik effect which states that people remember uncompleted or interrupted tasks better than completed ones. The so-called Method of loci uses spatial memory to memorize non-spatial information. Interference from previous knowledge

At the Center for Cognitive Science at Ohio State University, researchers have found that memory accuracy of adults is hurt by the fact that they know more than children and tend to apply this knowledge when learning new information. The findings appeared in the August 2004 edition of the journal Psychological Science. Interference can hamper memorisation and retrieval. There is retroactive interference when learning new information causes forgetting of old information, and proactive interference where learning one piece of information makes it harder to learn similar new information.[14] Influence of odors and emotions In March 2007 German researchers found they could use odors to re-activate new memories in the brains of people while they slept and the volunteers remembered better later.[15] Emotion can have a powerful impact on memory. Numerous studies have shown that the most vivid autobiographical memories tend to be of emotional events, which are likely to be recalled more often and with more clarity and detail than neutral events.[16]

[edit] Memory and aging

Main article: Memory and aging One of the key concerns of older adults is the experience of memory loss, especially as it is one of the hallmark symptoms of Alzheimer's disease. However, memory loss is qualitatively different in normal aging from the kind of memory loss associated with a diagnosis of Alzheimer's (Budson & Price, 2005).

[edit] Improving memory

Main article: Improving memory A UCLA research study published in the June 2006 issue of the American Journal of Geriatric Psychiatry found that people can improve cognitive function and brain efficiency through simple lifestyle changes such as incorporating memory exercises, healthy eating, physical fitness and stress reduction into their daily lives. This study examined 17 subjects, (average age 53) with normal memory performance. Eight subjects were asked to follow a "brain healthy" diet, relaxation, physical, and mental exercise (brain teasers and verbal memory training techniques). After 14 days, they showed greater word fluency (not memory) compared to their baseline performance. No long term follow up was conducted, it is therefore unclear if this intervention has lasting effects on memory.[17] There are a loosely associated group of mnemonic principles and techniques that can be used to vastly improve memory known as the Art of memory. The International Longevity Center released in 2001 a report[18] which includes in pages 1416 recommendations for keeping the mind in good functionality until advanced age. Some of the

recommendations are to stay intellectually active through learning, training or reading, to keep physically active so to promote blood circulation to the brain, to socialize, to reduce stress, to keep sleep time regular, to avoid depression or emotional instability and to observe good nutrition.

[edit] Memory tasks

Paired associate learning - when one learns to associate one specific word with another. For example when given a word such as "safe" one must learn to say another specific word, such as "green". This is stimulus and response.[19] Free recall - during this task a subject would be asked to study a list of words and then sometime later they will be asked to recall or write down as many words that they can remember.[20] Recognition - subjects are asked to remember a list of words or pictures, after which point they are asked to identify the previously presented words or pictures from among a list of alternatives that were not presented in the original list.[21]

COBOL
From Wikipedia, the free encyclopedia For other uses, see COBOL (disambiguation). COBOL Paradigm Appeared in Designed by Stable release Typing discipline procedural, object-oriented 1959 Grace Hopper, William Selden, Gertrude Tierney, Howard Bromberg, Howard Discount, Vernon Reeves, Jean E. Sammet COBOL 2002 (2002) strong, static

OpenCOBOL, Micro Focus International (e.g. the EclipseMajor implementations plug-in Micro Focus Net Express) HP3000 COBOL/II, COBOL/2, IBM OS/VS COBOL, IBM COBOL/II, IBM COBOL SAA, IBM Enterprise COBOL, IBM COBOL/400, IBM ILE COBOL, Unix COBOL X/Open, Micro Focus COBOL, Microsoft COBOL, Ryan McFarland RM/COBOL, Ryan McFarland RM/COBOL-85, DOSVS COBOL, UNIVAC COBOL, Realia COBOL, Fujitsu COBOL, ICL COBOL, ACUCOBOL-GT, COBOL-IT, DEC COBOL-10, DEC VAX COBOL, Wang VS COBOL, Visual COBOL, Tandem (NonStop) COBOL85, Tandem (NonStop) SCOBOL (a COBOL74 variant for creating screens on text-based terminals) FLOW-MATIC, COMTRAN, FACT PL/I, CobolScript, ABAP

Dialects

Influenced by Influenced

COBOL at Wikibooks COBOL ( /kobl/) is one of the oldest programming languages. Its name is an acronym for COmmon Business-Oriented Language, defining its primary domain in business, finance, and administrative systems for companies and governments. The COBOL 2002 standard includes support for object-oriented programming and other modern language features.[1]

Contents
[hide]

1 History and specification o 1.1 ANS COBOL 1968 o 1.2 COBOL 1974 o 1.3 COBOL 1985 o 1.4 COBOL 2002 and object-oriented COBOL o 1.5 History of COBOL standards o 1.6 Legacy 2 Features o 2.1 Self-modifying code o 2.2 Syntactic features o 2.3 Data types o 2.4 Hello, world 3 Criticism and defense o 3.1 Lack of structurability o 3.2 Verbose syntax o 3.3 Other defenses 4 See also 5 References 6 Sources 7 External links

[edit] History and specification

The COBOL specification was created by a committee of researchers from private industry, universities, and government during the second half of 1959. The specifications were to a great extent inspired by the FLOW-MATIC language invented by Grace Hopper - commonly referred to as "the mother of the COBOL language." The IBM COMTRAN language invented by Bob Bemer was also drawn upon, but the FACT language specification from Honeywell was not distributed to committee members until late in the process and had relatively little impact.

FLOW-MATIC's status as the only language of the bunch to have actually been implemented made it particularly attractive to the committee.[2] The scene was set on April 8, 1959 at a meeting of computer manufacturers, users, and university people at the University of Pennsylvania Computing Center. The United States Department of Defense subsequently agreed to sponsor and oversee the next activities. A meeting chaired by Charles A. Phillips was held at the Pentagon on May 28 and 29 of 1959 (exactly one year after the Zrich ALGOL 58 meeting); there it was decided to set up three committees: short, intermediate and long range (the last one was never actually formed). It was the Short Range Committee, chaired by Joseph Wegstein of the US National Bureau of Standards, that during the following months created a description of the first version of COBOL.[3] The committee was formed to recommend a short range approach to a common business language. The committee was made up of members representing six computer manufacturers and three government agencies. The six computer manufacturers were Burroughs Corporation, IBM, Minneapolis-Honeywell (Honeywell Labs), RCA, Sperry Rand, and Sylvania Electric Products. The three government agencies were the US Air Force, the Navy's David Taylor Model Basin, and the National Bureau of Standards (now National Institute of Standards and Technology). The intermediate-range committee was formed but never became operational. In the end a sub-committee of the Short Range Committee developed the specifications of the COBOL language. This sub-committee was made up of six individuals:

William Selden and Gertrude Tierney of IBM Howard Bromberg and Howard Discount of RCA Vernon Reeves and Jean E. Sammet of Sylvania Electric Products[4]

The decision to use the name "COBOL" was made at a meeting of the committee held on 18 September 1959. The subcommittee completed the specifications for COBOL in December 1959. The first compilers for COBOL were subsequently implemented in 1960, and on December 6 and 7, essentially the same COBOL program ran on two different computer makes, an RCA computer and a Remington-Rand Univac computer, demonstrating that compatibility could be achieved.

[edit] ANS COBOL 1968

After 1959 COBOL underwent several modifications and improvements. In an attempt to overcome the problem of incompatibility between different versions of COBOL, the American National Standards Institute (ANSI) developed a standard form of the language in 1968. This version was known as American National Standard (ANS) COBOL.

[edit] COBOL 1974

In 1974, ANSI published a revised version of (ANS) COBOL, containing a number of features that were not in the 1968 version.

[edit] COBOL 1985

In 1985, ANSI published still another revised version that had new features not in the 1974 standard, most notably structured language constructs ("scope terminators"), including END-IF, END-PERFORM, END-READ, etc.

[edit] COBOL 2002 and object-oriented COBOL

The language continues to evolve today. In the early 1990s it was decided to add objectorientation in the next full revision of COBOL. The initial estimate was to have this revision completed by 1997 and an ISO CD (Committee Draft) was available by 1997. Some implementers (including Micro Focus, Fujitsu, Veryant, and IBM) introduced object-oriented syntax based on the 1997 or other drafts of the full revision. The final approved ISO Standard (adopted as an ANSI standard by INCITS) was approved and made available in 2002. Like the C++ and Java programming languages, object-oriented COBOL compilers are available even as the language moves toward standardization. Fujitsu and Micro Focus currently support object-oriented COBOL compilers targeting the .NET framework.[5] The 2002 (4th revision) of COBOL included many other features beyond object-orientation. These included (but are not limited to):

National Language support (including but not limited to Unicode support) Locale-based processing User-defined functions CALL (and function) prototypes (for compile-time parameter checking) Pointers and syntax for getting and freeing storage Calling conventions to and from non-COBOL languages such as C Support for execution within framework environments such as Microsoft's .NET and Java (including COBOL instantiated as Enterprise JavaBeans) Bit and Boolean support True binary support (up until this enhancement, binary items were truncated based on the (base-10) specification within the Data Division) Floating-point support Standard (or portable) arithmetic results XML generation and parsing

[edit] History of COBOL standards

The specifications approved by the full Short Range Committee were approved by the Executive Committee on January 3, 1960, and sent to the government printing office, which edited and printed these specifications as Cobol 60. The American National Standards Institute (ANSI) produced several revisions of the COBOL standard, including:

COBOL-68 COBOL-74 COBOL-85 Intrinsic Functions Amendment - 1989 Corrections Amendment - 1991

After the Amendments to the 1985 ANSI Standard (which were adopted by ISO), primary development and ownership was taken over by ISO. The following editions and TRs (Technical Reports) have been issued by ISO (and adopted as ANSI) Standards:

COBOL 2002 Finalizer Technical Report - 2003 Native XML syntax Technical Report - 2006 Object Oriented Collection Class Libraries - pending final approval...

From 2002, the ISO standard is also available to the public coded as ISO/IEC 1989. Work progresses on the next full revision of the COBOL Standard. Approval and availability was expected early 2010s. For information on this revision, to see the latest draft of this revision, or to see what other works is happening with the COBOL Standard, see the COBOL Standards Website.

[edit] Legacy
COBOL programs are in use globally in governmental and military agencies and in commercial enterprises, and are running on operating systems such as IBM's z/OS, the POSIX families (Unix/Linux etc.), and Microsoft's Windows as well as ICL's VME operating system and Unisys' OS 2200. In 1997, the Gartner Group reported that 80% of the world's business ran on COBOL with over 200 billion lines of code in existence and with an estimated 5 billion lines of new code annually.[6] Near the end of the twentieth century the year 2000 problem was the focus of significant COBOL programming effort, sometimes by the same programmers who had designed the systems decades before. The particular level of effort required for COBOL code has been attributed both to the large amount of business-oriented COBOL, as COBOL is by design a business language and business applications use dates heavily, and to constructs of the COBOL language such as the PICTURE clause, which can be used to define fixed-length numeric fields, including two-digit fields for years.[citation needed] Because of the clean-up effort put into these COBOL programs for Y2K, many of them have been kept in use for years since then.[citation needed]

[edit] Features
COBOL as defined in the original specification included a PICTURE clause for detailed field specification. It did not support local variables, recursion, dynamic memory allocation, or structured programming constructs. Support for some or all of these features has been added in

later editions of the COBOL standard. COBOL has many reserved words (over 400), called keywords.

[edit] Self-modifying code

The original COBOL specification supported self-modifying code via the infamous "ALTER X TO PROCEED TO Y" statement. X and Y are paragraph labels, and any "GOTO X" statements executed after such an ALTER statement have the meaning "GOTO Y" instead. Most[citation needed] compilers still support it, but it should not be used in new programs.

[edit] Syntactic features

COBOL provides an update-in-place syntax, for example
ADD YEARS TO AGE

The equivalent construct in many procedural languages would be

age = age + years

This syntax is similar to the compound assignment operator later adopted by C:

age += years

The abbreviated conditional expression

IF SALARY > 8000 OR SUPERVISOR-SALARY OR = PREV-SALARY

is equivalent to
IF SALARY > 8000 OR SALARY > SUPERVISOR-SALARY OR SALARY == PREV-SALARY

COBOL provides "named conditions" (so-called 88-levels). These are declared as sub-items of another item (the conditional variable). The named condition can be used in an IF statement, and tests whether the conditional variable is equal to any of the values given in the named condition's VALUE clause. The SET statement can be used to make a named condition TRUE (by assigning the first of its values to the conditional variable). COBOL allows identifiers up to 30 characters long. When COBOL was introduced, much shorter lengths (e.g., 6 characters for FORTRAN) were prevalent. COBOL introduced the concept of copybooks chunks of code that can be inserted into a larger program. COBOL does this with the COPY statement, which also allows other code to replace parts of the copybook's code with other code (using the REPLACING ... BY ... clause).

[edit] Data types

Standard COBOL provides the following data types: Data type Character Edited character Sample declaration
PIC X(20) PIC A(4)9(5)X(7) PIC X99BAXX PIC S999V99 [USAGE] COMPUTATIONAL

Numeric fixed-point binary

or
BINARY

Numeric fixed-point packed decimal

PIC S999V99 PACKED-DECIMAL

Numeric fixed-point zoned decimal Numeric floating-point Edited numeric

PIC S999V99 [USAGE DISPLAY]

PIC S9V999ES99 PIC +Z,ZZ9.99 PIC $***,**9.99CR 01 CUST-NAME. 05 CUST-LAST PIC X(20). 05 CUST-FIRST PIC X(20). OCCURS 12 TIMES

Notes Alphanumeric and alphabeticonly Single-byte character set (SBCS) Formatted and inserted characters Binary 16, 32, or 64 bits (2, 4, or 8 bytes) Signed or unsigned. Conforming compilers limit the maximum value of variables based on the picture clause and not the number of bits reserved for storage. 1 to 18 decimal digits (1 to 10 bytes) Signed or unsigned 1 to 18 decimal digits (1 to 18 bytes) Signed or unsigned Leading or trailing sign, overpunch or separate Binary floating-point Formatted characters and digits

Group (record)

Aggregated elements

Table (array) Variable-length table Renames (variant or union data) Condition name Array index

Fixed-size array, row-major order Up to 7 dimensions OCCURS 0 to 12 TIMES Variable-sized array, row-major DEPENDING ON CUSTorder COUNT Up to 7 dimensions
66 RAW-RECORD RENAMES CUSTRECORD 88 IS-RETIRED-AGE VALUES 65 THRU 150 [USAGE] INDEX

Character data overlaying other variables Boolean value dependent upon another variable Array subscript

Most vendors provide additional types, such as:

Data type Numeric floating-point single precision Numeric floating-point double precision Numeric fixed-point packed decimal Numeric fixed-point binary

Sample declaration
PIC S9V9999999ES99 [USAGE] COMPUTATIONAL-1 PIC S9V999ES99 [USAGE] COMPUTATIONAL-2 PIC S9V999 [USAGE] COMPUTATIONAL-3 PIC S999V99 [USAGE] COMPUTATIONAL-4

Notes Binary floating-point (32 bits, 7+ digits) (IBM extension) Binary floating-point (64 bits, 16+ digits) (IBM extension) same as PACKED DECIMAL (IBM extension) same as COMPUTATIONAL or BINARY (IBM extension) Binary 16, 32, or 64 bits (2, 4, or 8 bytes) Signed or unsigned. The maximum value of variables based on the number of bits reserved for storage and not on the picture clause. (IBM extension) Binary 16, 32, or 64 bits (2, 4, or 8 bytes) Signed or unsigned Binary 16, 32, or 64 bits (2, 4, or 8 bytes) Signed or unsigned Alphanumeric Double-byte character set (DBCS) Formatted and inserted wide characters Formatted characters, decimal digits, and exponent Data memory address Code memory address n can be from 1 to 64, defining an n-bit integer Signed or unsigned Binary value corresponding to an occurrence of a table element May be linked to a specific table using INDEXED BY

Numeric fixed-point binary (native binary)

PIC S999V99 [USAGE] COMPUTATIONAL-5

Numeric fixed-point binary in native byte order Numeric fixed-point binary in big-endian byte order Wide character Edited wide character Edited floating-point Data pointer Code pointer Bit field

PIC S999V99 [USAGE] COMPUTATIONAL-4 PIC S999V99 [USAGE] COMPUTATIONAL-5 PIC G(20) PIC G99BGGG PIC +9.9(6)E+99 [USAGE] POINTER [USAGE] PROCEDUREPOINTER PIC 1(n) [USAGE] COMPUTATIONAL-5

Index

[USAGE] INDEX

[edit] Hello, world

An example of the "Hello, world" program in COBOL:
IDENTIFICATION DIVISION. PROGRAM-ID. HELLO-WORLD. PROCEDURE DIVISION. DISPLAY 'Hello, world'. STOP RUN.

[edit] Criticism and defense

[edit] Lack of structurability
In his letter to an editor in 1975 titled "How do we tell truths that might hurt?" which was critical of several programming languages contemporaneous with COBOL, computer scientist and Turing Award recipient Edsger Dijkstra remarked that "The use of COBOL cripples the mind; its teaching should, therefore, be regarded as a criminal offense."[7] In his dissenting response to Dijkstra's article and the above "offensive statement," computer scientist Howard E. Tompkins defended structured COBOL: "COBOL programs with convoluted control flow indeed tend to 'cripple the mind'," but this was because "There are too many such business application programs written by programmers that have never had the benefit of structured COBOL taught well..."[8] Additionally, the introduction of OO-COBOL has added support for object-oriented code as well as user-defined functions and user-defined data types to COBOL's repertoire.

[edit] Verbose syntax

COBOL 85 was not fully compatible with earlier versions, resulting in the "cesarean birth" of COBOL 85. Joseph T. Brophy, CIO, Travelers Insurance, spearheaded an effort to inform users of COBOL of the heavy reprogramming costs of implementing the new standard. As a result the ANSI COBOL Committee received more than 3,200 letters from the public, mostly negative, requiring the committee to make changes. On the other hand, conversion to COBOL 85 was thought to increase productivity in future years, thus justifying the conversion costs.[9] COBOL syntax has often been criticized for its verbosity. However, proponents note that this was intentional in the language design, and many consider it one of COBOL's strengths. One of the design goals of COBOL was that non-programmersmanagers, supervisors, and users could read and understand the code. This is why COBOL has an English-like syntax and structural elementsincluding: nouns, verbs, clauses, sentences, sections, and divisions. Consequently, COBOL is considered by at least one source to be "The most readable, understandable and self-documenting programming language in use today. [...] Not only does this readability generally assist the maintenance process but the older a program gets the more valuable this readability becomes."[10] On the other hand, the mere ability to read and understand

a few lines of COBOL code does not grant to an executive or end user the experience and knowledge needed to design, build, and maintain large software systems.[citation needed]

[edit] Other defenses

Additionally, traditional COBOL is a simple language with a limited scope of function (with no pointers, no user-defined types, and no user-defined functions), encouraging a straightforward coding style. This has made it well-suited to its primary domain of business computingwhere the program complexity lies in the business rules that need to be encoded rather than sophisticated algorithms or data structures. And because the standard does not belong to any particular vendor, programs written in COBOL are highly portable. The language can be used on a wide variety of hardware platforms and operating systems. And the rigid hierarchical structure restricts the definition of external references to the Environment Division, which simplifies platform changes.[10]