SAP HANA Predictive Analysis Library PAL en

PUBLIC
SAP HANA Platform SPS 06 Document Version: 1.0 - 26-06-2013
SAP HANA Predictive Analysis Library (PAL)
Table of Contents
1 2 2.1 2.2 2.3 2.4 2.5 3 3.1 What is PAL?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Getting Started with PAL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Application Function Library (AFL). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Checking PAL Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 How to Call PAL Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 PAL Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Clustering Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.1 3.1.2 3.1.3 3.1.4 3.1.5 3.2 3.2.1 3.2.2 3.2.3 3.2.4 3.2.5 3.2.6 3.2.7 3.2.8 3.2.9 3.2.10 3.3 3.4 3.3.1 3.4.1 3.4.2 3.4.3 3.5 3.5.1 3.5.2 3.5.3 3.5.4 3.5.5 Anomaly Detection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 DBSCAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 K-means. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Self-Organizing Maps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 Slight Silhouette. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Bi-Variate Geometric Regression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Bi-Variate Natural Logarithmic Regression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45 C4.5 Decision Tree. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 CHAID Decision Tree. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Exponential Regression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 KNN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Logistic Regression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Multiple Linear Regression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Naive Bayes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Polynomial Regression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .98 Apriori. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Single Exponential Smoothing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Double Exponential Smoothing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 Triple Exponential Smoothing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Binning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Convert Category Type to Binary Vector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Inter-quartile Range Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Sampling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Scaling Range. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Classification Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Association Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105 Time Series Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Preprocessing Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125
PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.
SAP HANA Predictive Analysis Library (PAL) Table of Contents
3.5.6 3.6 3.7 3.6.1 3.7.1 3.7.2 4 5
Variance Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 Link Prediction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .154 ABC Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Weighted Score Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Social Network Analysis Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Miscellaneous. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
End-to-End Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
SAP HANA Predictive Analysis Library (PAL) Table of Contents
PUBLIC
2013 SAP AG or an SAP affiliate company. All rights reserved.
What is PAL?
SAP HANAs SQLScript, an extension of SQL that includes enhanced control-flow capabilities, lets developers define complex application logic inside database procedures. However, it is difficult or even impossible to describe predictive analysis logic with procedures. For example, an application may need to perform a cluster analysis in a huge customer table with 1T records. It is impossible to implement the analysis in a procedure using the simple classic Kmeans algorithms, and also impossible with the more complicated algorithms in the data-mining area. Transferring large tables to the application server to perform the K-means calculation would also be costly. The Predictive Analysis Library (PAL) defines functions that can be called from within SQLScript procedures to perform analytic algorithms. This release of PAL includes classic and universal predictive analysis algorithms in seven data-mining categories: Clustering Classification Association Time Series Preprocessing Social Network Analysis Miscellaneous
The algorithms in PAL were carefully selected based on the following criteria: The algorithms are needed for SAP HANA applications. The algorithms are the most commonly used based on market surveys (e.g. Rexer Analytics and KDnuggets polls). The algorithms are generally available in other database products.
PUBLIC
SAP HANA Predictive Analysis Library (PAL) What is PAL?
Getting Started with PAL
This section covers the information you need to know to start working with the SAP HANA Predictive Analysis Library.
2.1
Prerequisites
To use the PAL functions, you must: Install SAP HANA SPS06. Install the Application Function Library (AFL), which includes the PAL. For more information, see the SAP HANA Server Installation Guide.
Note
The revision of the AFL must match the revision of SAP HANA. Each release of the AFL has a version in the form of <revision_number>.<patch_level>. For example, AFL 40.01 refers to revision 40 and patch level 01, and it should be installed with SAP HANA revision 40 only.
2.2
Application Function Library (AFL)
You can dramatically increase performance by executing complex computations in the database instead of at the application sever level. SAP HANA provides several techniques to move application logic into the database, and one of the most important is the use of application functions. Application functions are like database procedures written in C++ and called from outside to perform data intensive and complex operations. Functions for a particular topic are grouped into an application function library (AFL), such as the Predictive Analysis Library (PAL) and the Business Function Library (BFL). Currently, all AFLs are delivered in one archive (that is, one SAR file with the name AFL<version_string>.SAR). The AFL archive is not part of the HANA appliance, and must be installed separately by the administrator.
2.3
Security
This section provides detailed security information which can help administrator and architects answer some common questions.
SAP HANA Predictive Analysis Library (PAL) Getting Started with PAL
PUBLIC
User and Schema

During startup, the system creates the user _SYS_AFL, with default schema _SYS_AFL. All AFL objects (such as areas, packages, functions, and procedures) are created under this user and schema. Therefore, all these objects have fully specified names in the form of _SYS_AFL.<object name>.
Role Assignment
For each AFL area, there is a role. You must be assigned this role to execute the functions in the library. The role for the PAL library is named: AFL__SYS_AFL_AFLPAL_EXECUTE
Note
There are 2 underscores between AFL and SYS. Once a role is created, it cannot be dropped anymore. In other words, even when an area with all its objects is dropped and re-created during system startup, the user still keeps the role originally granted.
2.4

Checking PAL Installation
To confirm that the PAL functions were installed successfully, you can check the following three public views: sys.afl_areas sys.afl_packages sys.afl_functions
These views are granted to the PUBLIC role and can be accessed by anyone. To check the views, run the SQL statements: SELECT * FROM "SYS"."AFL_AREAS" WHERE SCHEMA_NAME = '_SYS_AFL' AND AREA_NAME = 'AFLPAL' SELECT * FROM "SYS"."AFL_PACKAGES" WHERE SCHEMA_NAME = '_SYS_AFL' AND AREA_NAME = 'AFLPAL' SELECT * FROM "SYS"."AFL_FUNCTIONS" WHERE SCHEMA_NAME = '_SYS_AFL' AND AREA_NAME = 'AFLPAL' The result will tell you whether the PAL functions were successfully installed on your system.
2.5
How to Call PAL Functions
To use PAL functions, you must do the following: Create the AFL_WRAPPER_GENERATOR and AFL_WRAPPER_ERASER procedures. This only needs to be done once. From within SQLScript code, generate a procedure that wraps the PAL function. Call the procedure, for example, from an SQLScript procedure.
Step 1 Create the AFL_WRAPPER_GENERATOR and AFL_WRAPPER_ERASER Procedures

Before using any AFL function, you need to create the AFL_WRAPPER_GENERATOR procedure. It is used to generate a wrapper for the AFL functions that take tables with a variable number of columns as inputs. This procedure only needs to be created once. If you want to drop any AFL function, you need to create the AFL_WRAPPER_ERASER procedure. It is used to generate a wrapper to clear the AFL functions that were created before. This procedure only needs to be created once. 1. 2. Make sure you are the SYSTEM user. Go to /hanamnt//<SID>/HDB <instance_number>/exe/plugins/afl/ and execute the afl_wrapper_generator.sql and afl_wrapper_eraser.sql script files. Thus, the AFL_WRAPPER_GENERATOR and AFL_WRAPPER_ERASER procedures are owned by the SYSTEM user. 3. Grant the EXECUTE privilege of system.afl_wrapper_generator and system.afl_wrapper_eraser to other users. For example, if the user name is USER1, run the command: GRANT EXECUTE ON system.afl_wrapper_generator to USER1 GRANT EXECUTE ON system.afl_wrapper_eraser to USER1
Step 2 Generate a PAL Procedure

Any user granted with the EXECUTE privilege on the system.afl_wrapper_generator procedure can generate an AFLLANG procedure for a specific PAL function. The syntax is shown below: CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure_name>', '<area_name>', '<function_name>', <signature_table>); <procedure_name>: A name for the PAL procedure. This can be anything you want. <area_name>: Always set to AFLPAL. <function_name>: A PAL built-in function name. <signature_table>: A user-defined table variable. The table contains records to describe the input table type, parameter table type, and result table type.
PUBLIC
A typical table variable references a table with the following definition: Index 1 2 3 Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
Note
1. The system.afl_wrapper_generator procedure is in definer mode, which means, the user who generates a PAL procedure should grant the SELECT privilege on signature table to the SYSTEM user who is the definer of system.afl_wrapper_generator. For example, if the user name is USER1, run the command: GRANT SELECT ON user1.<signature table> to SYSTEM 2. 3. 4. 5. The records in the signature table must follow this order: first input table types, next parameter table type, and then output table types. The signature table must be created before generating the PAL procedure. The table type names are userdefined. You can find detailed table type definitions for each PAL function in Chapter 3. It is suggested that you add <schema_name> before the table type name in <signature_table>. Since all the generated procedures and the procedure parameter table types belong to the _SYS_AFL schema, their names must be unique. The procedure names are defined by users. When generating a PAL procedure, make sure you give a unique procedure name. The parameter table type names are given by the system, so it is guaranteed the names are unique. 6. If you want to drop an existing procedure and then generate it again, you need to call the system.afl_wrapper_eraser procedure to clear the existing procedure. For example: CALL SYSTEM.AFL_WRAPPER_ERASER('<procedure_name>');
Step 3 Call a PAL Procedure

After generating a PAL procedure, any user that has the AFL__SYS_AFL_AFLPAL_EXECUTE role can call the procedure, using the syntax below. CALL <procedure_name>(<data_input_table> {,}, <parameter_table>, <output_table> {, }) with overview; <procedure_name>: The procedure name specified when generating the procedure in Step 2. <data_input_table>: User-defined name(s) of the procedures input table(s). Detailed input table definitions for each procedure can be found in Chapter 3. <parameter_table>: User-defined name of the procedures parameter table. The table structure is described in Section 2.4.1. Detailed parameter table definition for each procedure can be found in Chapter 3. <output_table>: User-defined name(s) of the procedures output table(s). Detailed output table definition for each procedure can be found in Chapter 3.
Note
1. The input, parameter, and output tables must be created before calling the procedure.
2. 3.
Some PAL algorithms have more than one input table or more than one output table. All AFL objects are owned by the _SYS_AFL user and reside in the _SYS_AFL schema. To call the PAL procedure generated in Step 2, you need the AFL__SYS_AFL_AFLPAL_EXECUTE role.
PUBLIC
PAL Functions
PAL Algorithm Anomaly Detection [page 11] DBSCAN [page 15] K-means [page 19] Built-in Function Name ANOMALYDETECTION DBSCAN KMEANS VALIDATEKMEANS Self-Organizing Maps [page 29] Slight Silhouette [page 34] SELFORGMAP SLIGHTSILHOUETTE GEOREGRESSION FORECASTWITHGEOR LNREGRESSION FORECASTWITHLNR CREATEDT PREDICTWITHDT CHAID Decision Tree [page 59] CREATEDTWITHCHAID PREDICTWITHDT Exponential Regression [page 66] EXPREGRESSION FORECASTWITHEXPR KNN [page 73] Logistic Regression [page 77] KNN LOGISTICREGRESSION FORECASTWITHLOGISTICR Multiple Linear Regression [page 85] Naive Bayes [page 92] LRREGRESSION FORECASTWITHLR NBCTRAIN NBCPREDICT Polynomial Regression [page 98] POLYNOMIALREGRESSION FORECASTWITHPOLYNOMIALR
The following are the available algorithms and functions in the Predictive Analysis Library. Category Clustering
Classification
Bi-Variate Geometric Regression [page 38] Bi-Variate Natural Logarithmic Regression [page 45] C4.5 Decision Tree [page 52]
Association
Apriori [page 106]
APRIORIRULE LITEAPRIORIRULE
Time Series
Single Exponential Smoothing [page SINGLESMOOTH 113]
10
SAP HANA Predictive Analysis Library (PAL) PAL Functions
Category
PAL Algorithm Double Exponential Smoothing [page 116]
Built-in Function Name DOUBLESMOOTH
Triple Exponential Smoothing [page TRIPLESMOOTH 120] Preprocessing Binning [page 126] Convert Category Type to Binary Vector [page 131] Inter-quartile Range Test [page 134] Sampling [page 137] Scaling Range [page 144] Variance Test [page 150] Social Network Analysis Miscellaneous Link Prediction [page 154] ABC Analysis [page 157] Weighted Score Table [page 160] BINNING CONV2BINARYVECTOR IQRTEST SAMPLING SCALINGRANGE VARIANCETEST LINKPREDICTION ABC WEIGHTEDTABLE
3.1
Clustering Algorithms
This section describes the clustering algorithms that are provided by the Predictive Analysis Library.
3.1.1
Anomaly Detection
Anomaly detection is used to find the existing data objects that do not comply with the general behavior or model of the data. Such data objects, which are grossly different from or inconsistent with the remaining set of data, are called anomalies or outliers. Sometimes anomalies are also referred to as discordant observations, exceptions, aberrations, surprises, peculiarities or contaminants in different application domains. Anomalies in data can translate to significant (and often critical) actionable information in a wide variety of application domains. For example, an anomalous traffic pattern in a computer network could mean that a hacked computer is sending out sensitive data to an unauthorized destination. An anomalous MRI image may indicate presence of malignant tumors. Anomalies in credit card transaction data could indicate credit card or identity theft or anomalous readings from a space craft sensor could signify a fault in some component of the space craft. PAL uses k-means to realize anomaly detection in two steps: 1. 2. Use k-means to group the origin data into k clusters. Identify some points that are far from all cluster centers as anomalies.
PUBLIC
11
Prerequisites
The input data contains an ID column and the other columns are of integer or double data type. The input data does not contain null value. The algorithm will issue errors when encountering null values.
ANOMALYDETECTION
Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'ANOMALYDETECTION', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column Other columns Column Data Type Integer or string Integer or double Description ID Attribute data Constraint It must be the first column. Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
Parameter Table Name GROUP_NUMBER Data Type Integer Description Number of groups (k). If k is not specified, the G-means method will be used to determine the number of clusters.
12
PUBLIC
Name DISTANCE_LEVEL
Data Type Integer
Description Computes the distance between the item and the cluster center. 1 = Manhattan distance 2 = Euclidean distance 3 = Minkowski distance
OUTLIER_PERCENTAGE OUTLIER_DEFINE
Double Integer
Indicates the proportion of anomalies in the source data. Specifies which point should be defined as outlier: 1 = max distance between the point and the center it belongs to 2 = max sum distance from the point to all centers
MAX_ITERATION INIT_TYPE
Integer Integer
Maximum number of iterations. Center initialization type: 1 = first K 2 = random with replacement 3 = random without replacement 4 = one patent of selecting the init center (US 6,882,998 B1)
NORMALIZATION
Integer
Normalization type: 0 = no 1 = yes. For each point X(x1,x, ,xn), the normalized value will be X'(x1/S,x2/S,...,xn/S), where S = |x1|+|x2|+...|xn|. 2 = for each column C, get the min and max value of C, and then C[i] = (C[i]-min)/(maxmin).
THREAD_NUMBER EXIT_THRESHOLD
Integer Double
Number of threads. Threshold (actual value) for exiting the iterations.
Output Table Table Result Column 1st column Column Data Type Integer or string Description ID Constraint
PUBLIC
13
Table
Column Other columns
Column Data Type Integer or double
Description Coordinates of outliers
Constraint It must have the same type as the input data table.
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_AD_RESULT_T; CREATE TYPE PAL_AD_RESULT_T AS TABLE( "ID" INT, "V000" DOUBLE, "V001" DOUBLE); DROP TYPE PAL_AD_DATA_T; CREATE TYPE PAL_AD_DATA_T AS TABLE( "ID" INT, "V000" DOUBLE, "V001" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); -- create procedure DROP TABLE PAL_AD_PDATA_TBL; CREATE COLUMN TABLE PAL_AD_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_AD_PDATA_TBL VALUES (1, 'DM_PAL.PAL_AD_DATA_T', 'in'); INSERT INTO PAL_AD_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_AD_PDATA_TBL VALUES (3, 'DM_PAL.PAL_AD_RESULT_T', 'out'); GRANT SELECT ON DM_PAL.PAL_AD_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_ANOMALY_DETECTION', 'AFLPAL', 'ANOMALYDETECTION', PAL_AD_PDATA_TBL); DROP TABLE PAL_AD_DATA_TBL; CREATE COLUMN TABLE PAL_AD_DATA_TBL ( "ID" INT, "V000" DOUBLE, "V001" DOUBLE); INSERT INTO PAL_AD_DATA_TBL VALUES (0 , 0.5, 0.5); INSERT INTO PAL_AD_DATA_TBL VALUES (1 , 1.5, 0.5); INSERT INTO PAL_AD_DATA_TBL VALUES (2 , 1.5, 1.5); INSERT INTO PAL_AD_DATA_TBL VALUES (3 , 0.5, 1.5); INSERT INTO PAL_AD_DATA_TBL VALUES (4 , 1.1, 1.2); INSERT INTO PAL_AD_DATA_TBL VALUES (5 , 0.5, 15.5); INSERT INTO PAL_AD_DATA_TBL VALUES (6 , 1.5, 15.5); INSERT INTO PAL_AD_DATA_TBL VALUES (7 , 1.5, 16.5); INSERT INTO PAL_AD_DATA_TBL VALUES (8 , 0.5, 16.5); INSERT INTO PAL_AD_DATA_TBL VALUES (9 , 1.2, 16.1); INSERT INTO PAL_AD_DATA_TBL VALUES (10, 15.5, 15.5); INSERT INTO PAL_AD_DATA_TBL VALUES (11, 16.5, 15.5); INSERT INTO PAL_AD_DATA_TBL VALUES (12, 16.5, 16.5); INSERT INTO PAL_AD_DATA_TBL VALUES (13, 15.5, 16.5); INSERT INTO PAL_AD_DATA_TBL VALUES (14, 15.6, 16.2); INSERT INTO PAL_AD_DATA_TBL VALUES (15, 15.5, 0.5); INSERT INTO PAL_AD_DATA_TBL VALUES (16, 16.5, 0.5); INSERT INTO PAL_AD_DATA_TBL VALUES (17, 16.5, 1.5); INSERT INTO PAL_AD_DATA_TBL VALUES (18, 15.5, 1.5); INSERT INTO PAL_AD_DATA_TBL VALUES (19, 15.7, 1.6); INSERT INTO PAL_AD_DATA_TBL VALUES (20,-1.0, -1.0); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('GROUP_NUMBER',4,null,null);
14
INSERT INTO #PAL_CONTROL_TBL VALUES ('INIT_TYPE',4,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('DISTANCE_LEVEL',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MAX_ITERATION',100,null,null); DROP TABLE PAL_AD_RESULT_TBL; CREATE COLUMN TABLE PAL_AD_RESULT_TBL ( "ID" INT, "V000" DOUBLE, "V001" DOUBLE); CALL _SYS_AFL.PAL_ANOMALY_DETECTION(PAL_AD_DATA_TBL, "#PAL_CONTROL_TBL", PAL_AD_RESULT_TBL) with overview; select * from PAL_AD_RESULT_TBL; Expected Result PAL_AD_RESULT_TBL:
3.1.2
DBSCAN
DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based data clustering algorithm. It finds a number of clusters starting from the estimated density distribution of corresponding nodes. DBSCAN requires two parameters: scan radius (eps) and the minimum number of points required to form a cluster (minPts). The algorithm starts with an arbitrary starting point that has not been visited. This point's epsneighborhood is retrieved, and if the number of points it contains is equal to or greater than minPts, a cluster is started. Otherwise, the point is labeled as noise. These two parameters are very important and are usually determined by user. PAL provides a method to automatically determine these two parameters. You can choose to specify the parameters by yourself or let the system determine them for you.
Prerequisites
No missing or null data in the inputs. The data is numeric, not categorical.
DBSCAN
Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'DBSCAN', <signature table>);
PUBLIC
15
The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column Other columns Parameter Table Name THREAD_NUMBER AUTO_PARAM Data Type Integer String Description Specifies the number of threads. Specifies whether the MINPTS and RADIUS parameters are determined automatically or by user. True: automatically determines the parameters False: uses parameter values provided by user Column Data Type Integer Integer or double Description ID Attribute data Constraint This must be the first column. Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
MINPTS
Integer
When AUTO_PARAM is set to False, specifies the minimum number of points required to form a cluster. When AUTO_PARAM is set to False, specifies the scan radius (eps). Specifies the method to compute the distance between two points. 1: Manhattan 2: Euclidean 3: Minkowski
RADIUS DISTANCE_METHOD
Double Integer
16
Name
Data Type
Description 4: Chebyshev 5: Standardized Euclidean 6: Cosine
MINKOW_P Output Table Table Result Column
Integer
Specifies the Minkowski method.
Column Data Type Integer Integer
Description ID Cluster ID (from 0 to cluster_number) Note: -1 means the point is labeled as noise.
1st column 2nd column
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_DBSCAN_DATA_T; CREATE TYPE PAL_DBSCAN_DATA_T AS TABLE ( ID integer, ATTRIB1 double, ATTRIB2 double); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( NAME varchar(50), INTARGS integer, DOUBLEARGS double, STRINGARGS varchar(100)); DROP TYPE PAL_DBSCAN_RESULTS_T; CREATE TYPE PAL_DBSCAN_RESULTS_T AS TABLE( ID integer, RESULT integer); DROP TABLE PAL_DBSCAN_PDATA_TBL; CREATE COLUMN TABLE PAL_DBSCAN_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_DBSCAN_PDATA_TBL VALUES (1, 'DM_PAL.PAL_DBSCAN_DATA_T', 'in'); INSERT INTO PAL_DBSCAN_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_DBSCAN_PDATA_TBL VALUES (3, 'DM_PAL.PAL_DBSCAN_RESULTS_T', 'out'); GRANT SELECT ON DM_PAL.PAL_DBSCAN_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_DBSCAN9', 'AFLPAL', 'DBSCAN', PAL_DBSCAN_PDATA_TBL); DROP TABLE PAL_DBSCAN_DATA_TBL; CREATE COLUMN TABLE PAL_DBSCAN_DATA_TBL ( ID integer, ATTRIB1 double, ATTRIB2 double); INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(1,0.10,0.10); INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(2,0.11,0.10); INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(3,0.10,0.11); INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(4,0.11,0.11); INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(5,0.12,0.11); INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(6,0.11,0.12); INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(7,0.12,0.12); INSERT INTO PAL_DBSCAN_DATA_TBL VALUES(8,0.12,0.13);
PUBLIC
17
INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT
INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO
PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL PAL_DBSCAN_DATA_TBL
VALUES(9,0.13,0.12); VALUES(10,0.13,0.13); VALUES(11,0.13,0.14); VALUES(12,0.14,0.13); VALUES(13,10.10,10.10); VALUES(14,10.11,10.10); VALUES(15,10.10,10.11); VALUES(16,10.11,10.11); VALUES(17,10.11,10.12); VALUES(18,10.12,10.11); VALUES(19,10.12,10.12); VALUES(20,10.12,10.13); VALUES(21,10.13,10.12); VALUES(22,10.13,10.13); VALUES(23,10.13,10.14); VALUES(24,10.14,10.13); VALUES(25,4.10,4.10); VALUES(26,7.11,7.10); VALUES(27,-3.10,-3.11); VALUES(28,16.11,16.11); VALUES(29,20.11,20.12); VALUES(30,15.12,15.11);
DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL( NAME varchar(50), INTARGS integer, DOUBLEARGS double, STRINGARGS varchar(100)); INSERT INTO #PAL_CONTROL_TBL VALUES('THREAD_NUMBER',18,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES('AUTO_PARAM',null,null,'true'); INSERT INTO #PAL_CONTROL_TBL VALUES('DISTANCE_METHOD',1,null,null); DROP TABLE PAL_DBSCAN_RESULTS_TBL; CREATE COLUMN TABLE PAL_DBSCAN_RESULTS_TBL( ID integer, RESULT integer); CALL _SYS_AFL.PAL_DBSCAN9(PAL_DBSCAN_DATA_TBL, "#PAL_CONTROL_TBL", PAL_DBSCAN_RESULTS_TBL) with overview; SELECT * FROM PAL_DBSCAN_RESULTS_TBL; Expected Result PAL_DBSCAN_RESULTS_TBL:
18
PUBLIC
3.1.3
K-means
In predictive analysis, k-means clustering is a method of cluster analysis. The k-means algorithm partitions n observations or records into k clusters in which each observation belongs to the cluster with the nearest center. In marketing and customer relationship management areas, this algorithm uses customer data to track customer behavior and create strategic business initiatives. Organizations can thus divide their customers into segments based on variants such as demography, customer behavior, customer profitability, measure of risk, and lifetime value of a customer or retention probability. Clustering works to group records together according to an algorithm or mathematical formula that attempts to find centroids, or centers, around which similar records gravitate. The most common algorithm uses an iterative refinement technique. It is also referred to as Lloyd's algorithm:
PUBLIC
19
Given an initial set of k means m1, ..., mk, the algorithm proceeds by alternating between two steps: Assignment step: assigns each observation to the cluster with the closest mean. Update step: calculates the new means to be the center of the observations in the cluster.
The algorithm repeats until the assignments no longer change. The k-means implementation in PAL supports multi-thread, data normalization, different distance level measurement, and cluster quality measurement (Silhouette). The implementation does not support categorical data, but this can be managed through data transformation. The first K and random K starting methods are supported. Support for Categorical Attributes If an attribute is of category type, it will be converted to a binary vector and then be used as a numerical attribute. For example, in the below table, "Gender" is of category type. Customer ID T1 T2 Age 31 27 Income 10,000 8,000 Gender Female Male
Because "Gender" has two distinct values, it will be converted into a binary vector with two dimensions: Customer ID T1 T2 Age 31 27 Income 10,000 8,000 Gender_1 0 1 Gender_2 1 0
Thus, the Euclidean distance between T1 and T2 is:
Where is the weight to be given to the transposed categorical attributes to lessen the impact on the clustering from the 0/1 attributes. Then you can use the traditional method to update the mean of every cluster. Assuming one cluster only has T1 and T2, the mean is: Customer ID Center1 Age 29.0 Income 9000.0 Gender_1 0.5 Gender_2 0.5
The means of categorical attributes will not be outputted. Instead, the means will be replaced by the modes similar to the K-Modes algorithm. Take the below center for example: Age Center 29.0 Income 9000.0 Gender_1 0.25 Gender_2 0.75
Because "Gender_2" is the maximum value, the output will be: Age Center 29.0 Income 9000.0 Gender Female
20
Prerequisites
The input data contains an ID column and the other columns are of integer or double data type. The input data does not contain null value. The algorithm will issue errors when encountering null values.
KMEANS
This is a clustering function using the k-means algorithm. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'KMEANS', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <center point output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column Other columns Column Data Type Integer or string Integer, double, or string Description ID Attribute data Constraint This must be the first column. Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <Center Point OUTPUT table type> Direction in in out out
Parameter Table Name GROUP_NUMBER Data Type Integer Description Number of groups (k).
PUBLIC
21
Name DISTANCE_LEVEL
Data Type Integer
Description Computes the distance between the item and the cluster center. 1 = Manhattan distance 2 (default) = Euclidean distance 3 = Minkowski distance 4 = Chebyshev distance
MINKOWSKI_POWER
Double
When you use the Minkowski distance, this parameter controls the value of p. The default is 3.0.
MAX_ITERATION INIT_TYPE
Integer Integer
Maximum iterations. The default is 100. Center initialization type: 1 = first K 2 = random with replacement 3 = random without replacement 4 (default) = one patent of selecting the init center (US 6,882,998 B1)
NORMALIZATION
Integer
Normalization type: 0 (default) = no 1 = yes. For each point X (x1,x2,...,xn), the normalized value will be X'(x1/S,x2/ S,...,xn/S), where S = |x1|+|x2| +...|xn|. 2 = for each column C, get the min and max value of C, and then C[i] = (C[i]-min)/(maxmin).
THREAD_NUMBER CATEGORY_COL
Integer Integer
Number of threads. Indicates whether the column is category variable. By default, 'string' is category variable, and 'integer' or 'double' is continuous variable. Represents the weight of category attributes (). The default is 0.707.
CATEGORY_WEIGHTS
Double
22
PUBLIC
Name EXIT_THRESHOLD
Data Type Double
Description Threshold (actual value) for exiting the iterations. The default is 0.000000001.
Output Tables Table Result Column 1st column 2nd column 3rd column Column Data Type Integer or string Integer Double Description ID Clustered item assigned to class number The distance between the cluster and each point in the cluster Cluster center ID Outputs means if the column is continuous variable; outputs modes if the column is category variable.
Center Points
1st column Other columns
Integer Double or string
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; --CREATE TABLE TYPE DROP TYPE PAL_KMEANS_DATA_T; CREATE TYPE PAL_KMEANS_DATA_T AS TABLE( "ID" INTEGER, "V000" DOUBLE, "V001" VARCHAR(5), "V002" DOUBLE ); DROP TYPE PAL_KMEANS_ASSIGN_T; CREATE TYPE PAL_KMEANS_ASSIGN_T AS TABLE( "ID" INTEGER, "CENTER_ASSIGN" INTEGER, "DISTANCE" DOUBLE ); DROP TYPE PAL_KMEANS_CENTERS_T; CREATE TYPE PAL_KMEANS_CENTERS_T AS TABLE( "CENTER_ID" INTEGER, "V000" DOUBLE, "V001" VARCHAR(5),
PUBLIC
23
);
"V002" DOUBLE
DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INT_ARGS" INTEGER, "DOUBLE_ARGS" DOUBLE, "STRING_ARGS" VARCHAR (100) ); --CREATE PROCEDURE DROP TABLE PAL_KMEANS_PDATA_TBL; CREATE COLUMN TABLE PAL_KMEANS_PDATA_TBL( "ID" INTEGER, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (1, INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (2, INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (3, INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (4, 'out');
'DM_PAL.PAL_KMEANS_DATA_T', 'in'); 'DM_PAL.PAL_CONTROL_T', 'in'); 'DM_PAL.PAL_KMEANS_ASSIGN_T', 'out'); 'DM_PAL.PAL_KMEANS_CENTERS_T',
GRANT SELECT ON DM_PAL.PAL_KMEANS_PDATA_TBL TO SYSTEM; CALL "SYSTEM".afl_wrapper_generator('PAL_KMEANS_PROC', 'AFLPAL', 'KMEANS', PAL_KMEANS_PDATA_TBL); --CREATE TABLES DROP TABLE PAL_KMEANS_DATA_TBL; CREATE COLUMN TABLE PAL_KMEANS_DATA_TBL LIKE PAL_KMEANS_DATA_T; INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL PAL_KMEANS_DATA_TBL VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES (0 (1 (2 (3 (4 (5 (6 (7 (8 (9 , , , , , , , , , , 0.5, 1.5, 1.5, 0.5, 1.1, 0.5, 1.5, 1.5, 0.5, 1.2, 15.5, 16.5, 16.5, 15.5, 15.6, 15.5, 16.5, 16.5, 15.5, 15.7, 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'C', 'C', 'C', 'C', 'C', 'D', 'D', 'D', 'D', 'D', 'A', 0.5); 0.5); 1.5); 1.5); 1.2); 15.5); 15.5); 16.5); 16.5); 16.1); 15.5); 15.5); 16.5); 16.5); 16.2); 0.5); 0.5); 1.5); 1.5); 1.6);
(10, (11, (12, (13, (14, (15, (16, (17, (18, (19,
DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL LIKE PAL_CONTROL_T; INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('GROUP_NUMBER',4,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('INIT_TYPE',1,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('DISTANCE_LEVEL',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MAX_ITERATION',100,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('EXIT_THRESHOLD',null,1.0E-6,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('NORMALIZATION',0,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CATEGORY_WEIGHTS',null, 0.5, null); --INSERT INTO #PAL_CONTROL_TBL VALUES ('MINKOWSKI_POWER',null,3,null);
24
DROP TABLE PAL_KMEANS_ASSIGN_TBL; CREATE COLUMN TABLE PAL_KMEANS_ASSIGN_TBL LIKE PAL_KMEANS_ASSIGN_T; DROP TABLE PAL_KMEANS_CENTERS_TBL; CREATE COLUMN TABLE PAL_KMEANS_CENTERS_TBL LIKE PAL_KMEANS_CENTERS_T; --CALL PROCEDURE CALL _SYS_AFL.PAL_KMEANS_PROC(PAL_KMEANS_DATA_TBL, #PAL_CONTROL_TBL, PAL_KMEANS_ASSIGN_TBL, PAL_KMEANS_CENTERS_TBL) with OVERVIEW; SELECT * FROM PAL_KMEANS_CENTERS_TBL; SELECT * FROM PAL_KMEANS_ASSIGN_TBL; Expected Result PAL_KMEANS_CENTERS_TBL:
PAL_KMEANS_ASSIGN_TBL:
PUBLIC
25
VALIDATEKMEANS
This is a quality measurement function for k-means clustering. The current version of VALIDATEKMEANS does not support category attributes. You can use CONV2BINARYVECTOR to convert category attributes into binary vectors, and then use them as continuous attributes. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','VALIDATEKMEANS', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<data input table>, <type input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Data Column 1st column Other columns Type Data/ Class Data 1st column 2nd column Parameter Table Name VARIABLE_NUM THREAD_NUMBER Output Table Table Result Column 1st column Column Data Type Varchar Description Name Data Type Integer Integer Description Number of variables Number of threads Column Data Type Integer Integer or double Integer Integer Description ID Attribute data ID Class type Table Type Name <Data INPUT table type> <Type INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in in out
26
Table
Column 2nd column
Column Data Type Double
Description Measure result
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; --table type for conv2binaryvector DROP TYPE PAL_CONV2_IN_T; CREATE TYPE PAL_CONV2_IN_T AS TABLE( "ID" INTEGER, "V001" VARCHAR(5) ); DROP TYPE PAL_CONV2_OUT_T; CREATE TYPE PAL_CONV2_OUT_T AS TABLE( "ID" INTEGER, "A0" INTEGER, "A1" INTEGER, "A2" INTEGER, "A3" INTEGER ); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INT_ARGS" INTEGER, "DOUBLE_ARGS" DOUBLE, "STRING_ARGS" VARCHAR(100) ); --table type for silhouette DROP TYPE PAL_SILHOUETTE_DATA_T; CREATE TYPE PAL_SILHOUETTE_DATA_T AS TABLE( "ID" INTEGER, "V000" DOUBLE, "A0" INTEGER, "A1" INTEGER, "A2" INTEGER, "A3" INTEGER, "V002" DOUBLE ); DROP TYPE PAL_SILHOUETTE_ASSIGN_T; CREATE TYPE PAL_SILHOUETTE_ASSIGN_T AS TABLE( "ID" INTEGER, "CLASS_LABEL" INTEGER ); DROP TYPE PAL_SILHOUETTE_RESULT_T; CREATE TYPE PAL_SILHOUETTE_RESULT_T AS TABLE( "Silhouette" VARCHAR(15), "VALUE" DOUBLE ); --create procedure DROP TABLE PAL_CONV2_PDATA_TBL; CREATE COLUMN TABLE PAL_CONV2_PDATA_TBL( "ID" INTEGER,
PUBLIC
27
); INSERT INTO PAL_CONV2_PDATA_TBL VALUES (1, 'DM_PAL.PAL_CONV2_IN_T', 'in'); INSERT INTO PAL_CONV2_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_CONV2_PDATA_TBL VALUES (3, 'DM_PAL.PAL_CONV2_OUT_T', 'out'); GRANT SELECT ON DM_PAL.PAL_CONV2_PDATA_TBL TO SYSTEM; CALL "SYSTEM".afl_wrapper_generator('PAL_CONV2BINARY_PROC', 'AFLPAL', 'CONV2BINARYVECTOR', PAL_CONV2_PDATA_TBL); DROP TABLE PAL_KMEANS_PDATA_TBL; CREATE COLUMN TABLE PAL_KMEANS_PDATA_TBL( "ID" INTEGER, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (1, INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (2, 'in'); INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (3, INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (4, 'out');
"TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100)
'DM_PAL.PAL_SILHOUETTE_DATA_T', 'in'); 'DM_PAL.PAL_SILHOUETTE_ASSIGN_T', 'DM_PAL.PAL_CONTROL_T', 'in'); 'DM_PAL.PAL_SILHOUETTE_RESULT_T',
CALL "SYSTEM".afl_wrapper_generator('PAL_SILHOUETTE_PROC', 'AFLPAL', 'VALIDATEKMEANS', PAL_KMEANS_PDATA_TBL); --prepare data DROP TABLE PAL_KMEANS_DATA_TBL; CREATE COLUMN TABLE PAL_KMEANS_DATA_TBL( "ID" INTEGER, "V000" DOUBLE, "V001" VARCHAR(5), "V002" DOUBLE ); INSERT INTO PAL_KMEANS_DATA_TBL VALUES (0 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (1 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (2 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (3 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (4 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (5 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (6 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (7 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (8 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (9 , INSERT INTO PAL_KMEANS_DATA_TBL VALUES (10, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (11, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (12, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (13, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (14, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (15, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (16, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (17, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (18, INSERT INTO PAL_KMEANS_DATA_TBL VALUES (19,
0.5, 'A', 0.5); 1.5, 'A', 0.5); 1.5, 'A', 1.5); 0.5, 'A', 1.5); 1.1, 'B', 1.2); 0.5, 'B', 15.5); 1.5, 'B', 15.5); 1.5, 'B', 16.5); 0.5, 'B', 16.5); 1.2, 'C', 16.1); 15.5, 'C', 15.5); 16.5, 'C', 15.5); 16.5, 'C', 16.5); 15.5, 'C', 16.5); 15.6, 'D', 16.2); 15.5, 'D', 0.5); 16.5, 'D', 0.5); 16.5, 'D', 1.5); 15.5, 'D', 1.5); 15.7, 'A', 1.6);
DROP TABLE PAL_CONV2_OUT_TBL; CREATE COLUMN TABLE PAL_CONV2_OUT_TBL LIKE PAL_CONV2_OUT_T; DROP VIEW PAL_CONV2_IN_V; CREATE VIEW PAL_CONV2_IN_V AS SELECT "ID", "V001" FROM PAL_KMEANS_DATA_TBL; DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL LIKE PAL_CONTROL_T; INSERT INTO #PAL_CONTROL_TBL VALUES ('OUT_PUT_COLUMNS', 5, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER', 2, null, null); --call procedure
28
PUBLIC
CALL "_SYS_AFL".PAL_CONV2BINARY_PROC(PAL_CONV2_IN_V, #PAL_CONTROL_TBL, PAL_CONV2_OUT_TBL) with OVERVIEW; --prepare data DROP TABLE PAL_SILHOUETTE_ASSIGN_TBL; CREATE COLUMN TABLE PAL_SILHOUETTE_ASSIGN_TBL LIKE PAL_SILHOUETTE_ASSIGN_T; INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (0, 0); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (1, 0); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (2, 0); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (3, 0); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (4, 0); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (5, 1); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (6, 1); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (7, 1); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (8, 1); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (9, 1); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (10, 2); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (11, 2); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (12, 2); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (13, 2); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (14, 2); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (15, 3); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (16, 3); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (17, 3); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (18, 3); INSERT INTO PAL_SILHOUETTE_ASSIGN_TBL VALUES (19, 3); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL LIKE PAL_CONTROL_T; INSERT INTO #PAL_CONTROL_TBL VALUES ('VARIABLE_NUM', 6, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER', 2, null, null); DROP VIEW PAL_SILHOUETTE_V; CREATE VIEW PAL_SILHOUETTE_V AS ( SELECT A."ID", "V000", "A0", "A1", "A2", "A3", "V002" FROM PAL_KMEANS_DATA_TBL AS A JOIN PAL_CONV2_OUT_TBL AS B ON A.ID=B.ID ); DROP TABLE PAL_SILHOUETTE_RESULT_TBL; CREATE COLUMN TABLE PAL_SILHOUETTE_RESULT_TBL LIKE PAL_SILHOUETTE_RESULT_T; --call procedure CALL "_SYS_AFL".PAL_SILHOUETTE_PROC(PAL_SILHOUETTE_V, PAL_SILHOUETTE_ASSIGN_TBL, #PAL_CONTROL_TBL, PAL_SILHOUETTE_RESULT_TBL) with OVERVIEW; SELECT * FROM PAL_SILHOUETTE_RESULT_TBL; Expected Result PAL_SILHOUETTE_RESULT_TBL:
3.1.4
Self-Organizing Maps
Self-organizing feature maps (SOMs) are one of the most popular neural network methods for cluster analysis. They are sometimes referred to as Kohonen self-organizing feature maps, after their creator, Teuvo Kohonen, or as topologically ordered maps. SOMs aim to represent all points in a high-dimensional source space by points in a
PUBLIC
29
low-dimensional (usually 2-D or 3-D) target space, such that the distance and proximity relationships are preserved as much as possible. This makes SOMs useful for visualizing low-dimensional views of highdimensional data, akin to multidimensional scaling. SOMs can also be viewed as a constrained version of k-means clustering, in which the cluster centers tend to lie in low-dimensional manifold in the feature or attribute space. The learning process mainly includes three steps: 1. 2. 3. Initialize the weighted vectors in each unit. Select the Best Matching Unit (BMU) for every point and update the weighted vectors of BMU and its neighbors. Repeat Step 2 until convergence or the maximum iterations are reached.
The SOM approach has many applications such as virtualization, web document clustering, and speech recognition.
Prerequisites
The first column of the input data is an ID column and the other columns are of integer or double data type. The input data does not contain null value. The algorithm will issue errors when encountering null values.
SELFORGMAP
Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'SELFORGMAP', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <map output table>, <assign output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Type Name <INPUT table type> <PARAMETER table type> <Map OUTPUT table type> <Assign OUTPUT table type> Direction in in out out
30
Table Data
Column 1st column Other columns
Column Data Type Integer or string Integer or double
Description ID Attribute data
Constraint This must be the first column.
Parameter Table Name MAX_ITERATION NORMALIZATION Data Type Integer Integer Description Maximum number of iterations. The default is 200. Normalization type: THREAD_NUMBER RAMDOM_SEED Integer Integer 0 (default) = no 1 = transform to new range (0.0, 1.0) 2 = z-score normalization
Number of threads. -1 (default): random 0: sets every weight to zero Other value: uses this value as seed
HEIGHT_OF_MAP WIDTH_OF_MAP ALPHA
Integer Integer Double Integer
Indicates the height of the map. The default is 10. Indicates the width of the map. The default is 10. Specifies the learning rate. The default is 0.5. Indicates the shape of the grid. 1: rectangle 2: hexagon (default)
SHAPE_OF_MAP or SHAPE_OF_GRID
Output Tables Table SOM Map Column 1st column Other columns except the last one Column Data Type Integer double Description Unit cell ID. Weight vectors used to simulate the original tuples.
PUBLIC
31
Table
Column Last column
Column Data Type Integer
Description Number of original tuples that every unit cell contains. ID of original tuples ID of the unit cells
SOM Assign
1st column 2th column
Integer or string Integer
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_SOM_DATA_T; CREATE TYPE PAL_SOM_DATA_T AS TABLE( "TRANS_ID" INT, "V000" DOUBLE, "V001" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); DROP TYPE PAL_SOM_MAP_T; CREATE TYPE PAL_SOM_MAP_T AS TABLE( "CELL_ID" INT, "WEIGHT000" DOUBLE, "WEIGHT001" DOUBLE, "NUMS_TUPLE" INT); DROP TYPE PAL_SOM_RESASSIGN_T; CREATE TYPE PAL_SOM_RESASSIGN_T AS TABLE( "TRANS_ID" INT, "CELL_ID" INT); -- create procedure DROP TABLE PAL_SOM_PDATA_TBL; CREATE COLUMN TABLE PAL_SOM_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_SOM_PDATA_TBL VALUES (1, 'DM_PAL.PAL_SOM_DATA_T', 'in'); INSERT INTO PAL_SOM_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_SOM_PDATA_TBL VALUES (3, 'DM_PAL.PAL_SOM_MAP_T', 'out'); INSERT INTO PAL_SOM_PDATA_TBL VALUES (4, 'DM_PAL.PAL_SOM_RESASSIGN_T', 'out'); GRANT SELECT ON DM_PAL.PAL_SOM_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_SELF_ORG_MAP', 'AFLPAL', 'SELFORGMAP', PAL_SOM_PDATA_TBL); DROP TABLE PAL_SOM_DATA_TBL; CREATE COLUMN TABLE PAL_SOM_DATA_TBL( "TRANS_ID" INT, "V000" DOUBLE, "V001" DOUBLE); INSERT INTO PAL_SOM_DATA_TBL VALUES (0 , 0.1, 0.2); INSERT INTO PAL_SOM_DATA_TBL VALUES (1 , 0.22, 0.25); INSERT INTO PAL_SOM_DATA_TBL VALUES (2 , 0.3, 0.4); INSERT INTO PAL_SOM_DATA_TBL VALUES (3 , 0.4, 0.5); INSERT INTO PAL_SOM_DATA_TBL VALUES (4 , 0.5, 1.0); INSERT INTO PAL_SOM_DATA_TBL VALUES (5 , 1.1, 15.1); INSERT INTO PAL_SOM_DATA_TBL VALUES (6 , 2.2, 11.2); INSERT INTO PAL_SOM_DATA_TBL VALUES (7 , 1.3, 15.3); INSERT INTO PAL_SOM_DATA_TBL VALUES (8 , 1.4, 15.4); INSERT INTO PAL_SOM_DATA_TBL VALUES (9 , 3.5, 15.9); INSERT INTO PAL_SOM_DATA_TBL VALUES (10,13.1, 1.1); INSERT INTO PAL_SOM_DATA_TBL VALUES (11,16.2, 1.5); INSERT INTO PAL_SOM_DATA_TBL VALUES (12,16.3, 1.3); INSERT INTO PAL_SOM_DATA_TBL VALUES (13,12.4, 2.4); INSERT INTO PAL_SOM_DATA_TBL VALUES (14,16.9, 1.9); INSERT INTO PAL_SOM_DATA_TBL VALUES (15,49.0, 40.1);
32
PUBLIC
INSERT INSERT INSERT INSERT
INTO INTO INTO INTO
PAL_SOM_DATA_TBL PAL_SOM_DATA_TBL PAL_SOM_DATA_TBL PAL_SOM_DATA_TBL
VALUES VALUES VALUES VALUES
(16,50.1, (17,50.2, (18,55.3, (19,50.4,
50.2); 48.3); 50.4); 56.5);
DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER', 2, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MAX_ITERATION', 200, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('HEIGHT_OF_MAP', 4, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('WIDTH_OF_MAP', 4, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('NORMALIZATION', 0, null, null); DROP TABLE PAL_SOM_MAP_TBL; CREATE COLUMN TABLE PAL_SOM_MAP_TBL ( "CELL_ID" INT, "WEIGHT000" DOUBLE, "WEIGHT001" DOUBLE, "NUMS_TUPLE" INT); DROP TABLE PAL_SOM_RESASSIGN_TBL; CREATE COLUMN TABLE PAL_SOM_RESASSIGN_TBL ( "TRANS_ID" INT, "CELL_ID" INT); CALL _SYS_AFL.PAL_SELF_ORG_MAP(PAL_SOM_DATA_TBL, "#PAL_CONTROL_TBL", PAL_SOM_MAP_TBL, PAL_SOM_RESASSIGN_TBL) with overview; select * from PAL_SOM_MAP_TBL; select * from PAL_SOM_RESASSIGN_TBL; Expected Result PAL_SOM_MAP_TBL:
PAL_SOM_RESASSIGN_TBL:
PUBLIC
33
3.1.5
Slight Silhouette
Silhouette refers to a method used to validate the cluster of data. The complex of Silhouette is O(N2), where N represents the number of records. When N is very large, silhouette will cost much time. In consideration of the efficient, PAL provides a lite version of Sihouette called Slight Silhouette. Suppose you have N records. For every record i, the following is defined:
Where A(i) represents the distance between i and the center of the cluster it belongs to, and B(i) is the minimum distance between i and other cluster centers. Finally the below formula is derived:
It is clear that
. 1 indicates poor clustering result, and 1 stands for good result.
For attributes of category type, you can pre-process the input data using the method described in K-means [page 19].
34
Prerequisites
The input data does not contain null value.
SLIGHTSILHOUETTE
Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'SLIGHTSILHOUETTE', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column Last column Other columns Column Data Type Integer or string Integer, double, or string Description Class label Attribute data Constraint This must be the last column. Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> Direction in in out
Parameter Table Name DISTANCE_LEVEL Data Type Integer Description Computes the distance between the item and the cluster center. 1 = Manhattan distance 2 (default) = Euclidean distance 3 = Minkowski distance 4 = Chebyshev distance
PUBLIC
35
Name MINKOWSKI_POWER
Data Type Double
Description When you use the Minkowski distance, this parameter controls the value of p. The default is 3.0.
NORMALIZATION
Integer
Normalization type: 0 = no 1 = yes. For each point X (x1,x2,...,xn), the normalized value will be X'(x1/S,x2/ S,...,xn/S), where S = |x1|+|x2| +...|xn|. 2 = for each column C, get the min and max value of C, and then C[i] = (C[i]-min)/(maxmin).
THREAD_NUMBER CATEGORY_COL
Integer Integer
Number of threads. Indicates whether the column is category variable. By default, 'string' is category variable, and 'integer' or 'double' is continuous variable. Represents the weight of category attributes (). The default is 0.707.
CATEGORY_WEIGHTS
Double
Output Table Table Result Example DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_SLIGHT_SIL_T; CREATE TYPE PAL_SLIGHT_SIL_T AS TABLE( "V000" DOUBLE, "V001" VARCHAR(5), "V002" DOUBLE, "Cluster" INTEGER ); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INT_ARGS" INTEGER, Column 1st column Column Data Type Double Description Validation value
36
);
"DOUBLE_ARGS" DOUBLE, "STRING_ARGS" VARCHAR(100)
DROP TYPE PAL_SLIGHT_SIL_RESULT_T; CREATE TYPE PAL_SLIGHT_SIL_RESULT_T AS TABLE( "SILHOUETTE" DOUBLE ); DROP TABLE PAL_SLIGHT_SIL_PDATA_TBL; CREATE COLUMN TABLE PAL_SLIGHT_SIL_PDATA_TBL( "ID" INTEGER, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_SLIGHT_SIL_PDATA_TBL VALUES (1, 'DM_PAL.PAL_SLIGHT_SIL_T', 'in'); INSERT INTO PAL_SLIGHT_SIL_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_SLIGHT_SIL_PDATA_TBL VALUES (3, 'DM_PAL.PAL_SLIGHT_SIL_RESULT_T', 'out'); GRANT SELECT ON DM_PAL.PAL_SLIGHT_SIL_PDATA_TBL TO SYSTEM; CALL "SYSTEM".afl_wrapper_generator('PAL_SLIGHT_SIL_PROC', 'AFLPAL', 'SLIGHTSILHOUETTE', PAL_SLIGHT_SIL_PDATA_TBL); DROP TABLE PAL_SLIGHT_SIL_TBL; CREATE COLUMN TABLE PAL_SLIGHT_SIL_TBL LIKE INSERT INTO PAL_SLIGHT_SIL_TBL VALUES (0.5, INSERT INTO PAL_SLIGHT_SIL_TBL VALUES (1.5, INSERT INTO PAL_SLIGHT_SIL_TBL VALUES (1.5, INSERT INTO PAL_SLIGHT_SIL_TBL VALUES (0.5, INSERT INTO PAL_SLIGHT_SIL_TBL VALUES (1.1, INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL PAL_SLIGHT_SIL_TBL VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES (0.5, (1.5, (1.5, (0.5, (1.2, (15.5, (16.5, (16.5, (15.5, (15.6, (15.5, (16.5, (16.5, (15.5, (15.7, PAL_SLIGHT_SIL_T; 'A', 0.5, 0); 'A', 0.5, 0); 'A', 1.5, 0); 'A', 1.5, 0); 'B', 1.2, 0); 'B', 'B', 'B', 'B', 'C', 'C', 'C', 'C', 'C', 'D', 'D', 'D', 'D', 'D', 'A', 15.5, 15.5, 16.5, 16.5, 16.1, 15.5, 15.5, 16.5, 16.5, 16.2, 0.5, 0.5, 1.5, 1.5, 1.6, 1); 1); 1); 1); 1); 2); 2); 2); 2); 2); 3); 3); 3); 3); 3);
DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES
#PAL_CONTROL_TBL LIKE PAL_CONTROL_T; ('THREAD_NUMBER', 2, null, null); ('NORMALIZATION', 0, null, null); ('DISTANCE_LEVEL', 2, null, null); ('CATEGORY_WEIGHTS', null, 0.7, null); ('CATEGORY_COL', 1, null, null);
DROP TABLE PAL_SLIGHT_SIL_RESULT_TBL; CREATE COLUMN TABLE PAL_SLIGHT_SIL_RESULT_TBL LIKE PAL_SLIGHT_SIL_RESULT_T; CALL "_SYS_AFL".PAL_SLIGHT_SIL_PROC(PAL_SLIGHT_SIL_TBL, #PAL_CONTROL_TBL, PAL_SLIGHT_SIL_RESULT_TBL) with OVERVIEW; SELECT * FROM PAL_SLIGHT_SIL_RESULT_TBL; Expected Result
PUBLIC
37
PAL_SLIGHT_SIL_RESULT_TBL:
3.2
Classification Algorithms
This section describes the classification algorithms that are provided by the Predictive Analysis Library.
3.2.1
Bi-Variate Geometric Regression
Geometric regression is an approach used to model the relationship between a scalar variable y and one or more variables denoted X. In geometric regression, data is modeled using geometric functions, and unknown model parameters are estimated from the data. Such models are called geometric models. In PAL, the implementation of geometric regression is to transform to linear regression and solve it: y = 0 x 1 Where 0 and 1 are parameters that need to be calculated. The steps are: 1. 2. 3. Put natural logarithmic operation on both sides: ln(y) = ln(0 x 1) Transform it into: ln(y) = ln( 0) + 1 ln(x) Let y' = ln(y), x' = ln(x), 0' = ln(0) y' = 0' + 1 x'
Thus, y and x is a linear relationship and can be solved with the linear regression method. The implementation also supports calculating the F value and R^2 to determine statistical significance.
Prerequisites
GEOREGRESSION
This is a geometric regression function.
38
PUBLIC
Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','GEOREGRESSION', <signature table>); The signature table should contain the following records: Index 1 2 3 4 5 6 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <fitted output table>, <significance output table>, <PMML output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column 3rd column Column Data Type Integer or varchar Integer or double Integer or double Description ID Variable y Variable x Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <Fitted OUTPUT table type> <Significance OUTPUT table type> <PMML OUTPUT table type> Direction in in out out out out
Parameter Table Name THREAD_NUMBER ALG Data Type Integer Integer Description Number of threads. (Optional) Specifies decomposition method: 0 (default): Doolittle decomposition (LU) 2: Singular value decomposition (SVD)
PUBLIC
39
Name ADJUSTED_R2
Data Type Integer
Description 0 (default): does not output adjusted R square 1: outputs adjusted R square 0 (default): does not export geometric regression model in PMML. 1: exports geometric regression model in PMML in single row. 2: exports geometric regression model in several rows, each row containing a maximum of 5000 characters.
PMML_EXPORT
Integer
Output Tables Table Result Column 1st column 2nd column Column Data Type Integer Integer or double Description ID Value Ai A0: intercept A1: beta coefficient for X1 Constraint
Fitted Data
Integer or varchar Integer or double Varchar Double Integer CLOB or varchar
ID Value Yi Name Value ID Geometric regression model in PMML format (R^2 / F)
Significance
PMML Result
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_GR_DATA_T; CREATE TYPE PAL_GR_DATA_T AS TABLE( "ID" INT,"Y" DOUBLE,"X1" DOUBLE);
40
PUBLIC
DROP TYPE PAL_GR_RESULT_T; CREATE TYPE PAL_GR_RESULT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_GR_FITTED_T; CREATE TYPE PAL_GR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP TYPE PAL_GR_SIGNIFICANCE_T; CREATE TYPE PAL_GR_SIGNIFICANCE_T AS TABLE("NAME" varchar(50),"VALUE" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_GR_PMMLMODEL_T; CREATE TYPE PAL_GR_PMMLMODEL_T AS TABLE("ID" INT,"Model" varchar(5000)); DROP table PAL_GR_PDATA_TBL; CREATE column table PAL_GR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_GR_PDATA_TBL values (1,'DM_PAL.PAL_GR_DATA_T','in'); insert into PAL_GR_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_GR_PDATA_TBL values (3,'DM_PAL.PAL_GR_RESULT_T','out'); insert into PAL_GR_PDATA_TBL values (4,'DM_PAL.PAL_GR_FITTED_T','out'); insert into PAL_GR_PDATA_TBL values (5,'DM_PAL.PAL_GR_SIGNIFICANCE_T','out'); insert into PAL_GR_PDATA_TBL values (6,'DM_PAL.PAL_GR_PMMLMODEL_T','out'); GRANT SELECT ON DM_PAL.PAL_GR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palGeoR','AFLPAL','GEOREGRESSION',PAL_GR_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PMML_EXPORT',2,null,null); DROP TABLE PAL_GR_DATA_TBL; CREATE COLUMN TABLE PAL_GR_DATA_TBL ( "ID" INT,"Y" DOUBLE,"X1" DOUBLE); INSERT INTO PAL_GR_DATA_TBL VALUES (0,1.1,1); INSERT INTO PAL_GR_DATA_TBL VALUES (1,4.2,2); INSERT INTO PAL_GR_DATA_TBL VALUES (2,8.9,3); INSERT INTO PAL_GR_DATA_TBL VALUES (3,16.3,4); INSERT INTO PAL_GR_DATA_TBL VALUES (4,24,5); INSERT INTO PAL_GR_DATA_TBL VALUES (5,36,6); INSERT INTO PAL_GR_DATA_TBL VALUES (6,48,7); INSERT INTO PAL_GR_DATA_TBL VALUES (7,64,8); INSERT INTO PAL_GR_DATA_TBL VALUES (8,80,9); INSERT INTO PAL_GR_DATA_TBL VALUES (9,101,10); DROP TABLE PAL_GR_RESULTS_TBL; CREATE COLUMN TABLE PAL_GR_RESULTS_TBL ("ID" INT,"Ai" DOUBLE); DROP TABLE PAL_GR_FITTED_TBL; CREATE COLUMN TABLE PAL_GR_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); DROP TABLE PAL_GR_SIGNIFICANCE_TBL; CREATE COLUMN TABLE PAL_GR_SIGNIFICANCE_TBL ("NAME" varchar(50),"VALUE" DOUBLE); DROP TABLE PAL_GR_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_GR_PMMLMODEL_TBL ("ID" INT, "PMMLMODEL" VARCHAR(5000)); CALL _SYS_AFL.palGeoR(PAL_GR_DATA_TBL, "#PAL_CONTROL_TBL", PAL_GR_RESULTS_TBL, PAL_GR_FITTED_TBL, PAL_GR_SIGNIFICANCE_TBL, PAL_GR_PMMLMODEL_TBL) with overview; SELECT SELECT SELECT SELECT * * * * FROM FROM FROM FROM PAL_GR_RESULTS_TBL; PAL_GR_FITTED_TBL; PAL_GR_SIGNIFICANCE_TBL; PAL_GR_PMMLMODEL_TBL;
PUBLIC
41
Expected Result PAL_GR_RESULTS_TBL:
PAL_GR_FITTED_TBL:
PAL_GR_SIGNIFICANCE_TBL:
PAL_GR_PMMLMODEL_TBL:
FORECASTWITHGEOR
This function performs prediction with the geometric regression result. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','FORECASTWITHGEOR', <signature table>); The signature table should contain the following records: Index 1 Table Type Name <Predictive INPUT table type> Direction in
42
Index 2 3 4 Procedure Calling
Table Type Name <Coefficient INPUT table type> <PARAMETER table type> <OUTPUT table type>
Direction in in out
CALL <procedure name>(<predictive input table>, <coefficient input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Predictive Data Column 1st column 2nd column Coefficient 1st column 2nd column Column Data Type Integer or varchar Integer or double Integer Integer, double, varchar, or CLOB Description ID Variable X ID (start from 0) Value Ai or PMML model. Varchar and CLOB types are only valid for PMML model.
Parameter Table Name THREAD_NUMBER MODEL_FORMAT Data Type Integer Integer Description Number of threads 0 (default): Coefficients in table 1: PMML format
Output Table Table Fitted Result Column 1st column 2nd column Example Assume that: Column Data Type Integer/ varchar Integer/ double Description ID Value Yi
PUBLIC
43
DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_FGR_PREDICT_T; CREATE TYPE PAL_FGR_PREDICT_T AS TABLE( "ID" INT,"X1" DOUBLE); DROP TYPE PAL_FGR_COEFFICIENT_T; CREATE TYPE PAL_FGR_COEFFICIENT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_FGR_FITTED_T; CREATE TYPE PAL_FGR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP table PAL_FGR_PDATA_TBL; CREATE column table PAL_FGR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_FGR_PDATA_TBL values (1,'DM_PAL.PAL_FGR_PREDICT_T','in'); insert into PAL_FGR_PDATA_TBL values (2,'DM_PAL.PAL_FGR_COEFFICIENT_T','in'); insert into PAL_FGR_PDATA_TBL values (3,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_FGR_PDATA_TBL values (4,'DM_PAL.PAL_FGR_FITTED_T','out'); GRANT SELECT ON DM_PAL.PAL_FGR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palForecastWithGeoR','AFLPAL','FORECASTWITHGEOR',PAL_FGR_PDA TA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); DROP TABLE PAL_FGR_PREDICTDATA_TBL; CREATE COLUMN TABLE PAL_FGR_PREDICTDATA_TBL ( "ID" INT,"X1" DOUBLE); INSERT INTO PAL_FGR_PREDICTDATA_TBL VALUES (0,1); INSERT INTO PAL_FGR_PREDICTDATA_TBL VALUES (1,2); INSERT INTO PAL_FGR_PREDICTDATA_TBL VALUES (2,3); INSERT INTO PAL_FGR_PREDICTDATA_TBL VALUES (3,4); INSERT INTO PAL_FGR_PREDICTDATA_TBL VALUES (4,5); DROP TABLE PAL_FGR_COEFFICIENT_TBL; CREATE COLUMN TABLE PAL_FGR_COEFFICIENT_TBL ("ID" INT,"Ai" DOUBLE); INSERT INTO PAL_FGR_COEFFICIENT_TBL VALUES (0,1); INSERT INTO PAL_FGR_COEFFICIENT_TBL VALUES (1,1.99); DROP TABLE PAL_FGR_FITTED_TBL; CREATE COLUMN TABLE PAL_FGR_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); CALL _SYS_AFL.palForecastWithGeoR(PAL_FGR_PREDICTDATA_TBL, PAL_FGR_COEFFICIENT_TBL, "#PAL_CONTROL_TBL", PAL_FGR_FITTED_TBL) with overview; SELECT * FROM PAL_FGR_FITTED_TBL;
Expected Result PAL_FGR_FITTED_TBL:
44
3.2.2
Bi-Variate Natural Logarithmic Regression
Bi-variate natural logarithmic regression is an approach to modeling the relationship between a scalar variable y and one variable denoted X. In natural logarithmic regression, data is modeled using natural logarithmic functions, and unknown model parameters are estimated from the data. Such models are called natural logarithmic models. In PAL, the implementation of natural logarithmic regression is to transform to linear regression and solve it: y = 1ln(x) + 0 Where 0 and 1 are parameters that need to be calculated. Let x = ln(x) Then y = 0 + 1 x Thus, y and x is a linear relationship and can be solved with the linear regression method. The implementation also supports calculating the F value and R^2 to determine statistical significance.
Prerequisites
No missing or null data in the inputs. The data is numeric, not categorical. Given the structure as Y and X, there are more than 2 records available for analysis.
LNREGRESSION
This is a logarithmic regression function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','LNREGRESSION', <signature table>); The signature table should contain the following records:
PUBLIC
45
Index 1 2 3 4 5 6 Procedure Calling
Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <Fitted OUTPUT table type> <Significance OUTPUT table type> <PMML OUTPUT table type>
Direction in in out out out out
CALL <procedure name>(<input table>, <parameter table>, <result output table>, <fitted output table>, <significance output table>, <PMML output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column 3rd column Column Data Type Integer or varchar Integer or double Integer or double Description ID Variable y Variable X
Parameter Table Name THREAD_NUMBER ALG Data Type Integer Integer Description Number of threads (Optional) Specifies decomposition method: 0 (default): Doolittle decomposition (LU) 2: Singular value decomposition (SVD) 0 (default): does not output adjusted R square 1: outputs adjusted R square 0 (default): does not export logarithmic regression model in PMML.
ADJUSTED_R2
Integer
PMML_EXPORT
Integer
46
PUBLIC
Name
Data Type
Description 1: exports logarithmic regression model in PMML in single row. 2: exports logarithmic regression model in PMML in several rows, each row containing a maximum of 5000 characters.
Output Tables Table Result Column 1st column 2nd column Column Data Type Integer Integer or double Description ID Value Ai Fitted Data 1st column 2nd column Significance 1st column 2nd column PMML Result 1st column 2nd column Integer or varchar Integer or double Varchar Double Integer CLOB or varchar ID Value Yi Name Value ID Logarithmic regression model in PMML format (R^2 / F) A0: intercept A1: beta coefficient for X1 A2: beta coefficient for X2 Constraint
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_NLR_DATA_T; CREATE TYPE PAL_NLR_DATA_T AS TABLE( "ID" INT,"Y" DOUBLE,"X1" DOUBLE); DROP TYPE PAL_NLR_RESULT_T; CREATE TYPE PAL_NLR_RESULT_T AS TABLE("ID" INT,"Ai" DOUBLE);
PUBLIC
47
DROP TYPE PAL_NLR_FITTED_T; CREATE TYPE PAL_NLR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP TYPE PAL_NLR_SIGNIFICANCE_T; CREATE TYPE PAL_NLR_SIGNIFICANCE_T AS TABLE("NAME" varchar(50),"VALUE" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_NLR_PMMLMODEL_T; CREATE TYPE PAL_NLR_PMMLMODEL_T AS TABLE("ID" INT,"Model" varchar(5000)); DROP table PAL_NLR_PDATA_TBL; CREATE column table PAL_NLR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_NLR_PDATA_TBL values (1,'DM_PAL.PAL_NLR_DATA_T','in'); insert into PAL_NLR_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_NLR_PDATA_TBL values (3,'DM_PAL.PAL_NLR_RESULT_T','out'); insert into PAL_NLR_PDATA_TBL values (4,'DM_PAL.PAL_NLR_FITTED_T','out'); insert into PAL_NLR_PDATA_TBL values (5,'DM_PAL.PAL_NLR_SIGNIFICANCE_T','out'); insert into PAL_NLR_PDATA_TBL values (6,'DM_PAL.PAL_NLR_PMMLMODEL_T','out'); GRANT SELECT ON DM_PAL.PAL_NLR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palLnR','AFLPAL','LNREGRESSION',PAL_NLR_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PMML_EXPORT',2,null,null); DROP TABLE PAL_NLR_DATA_TBL; CREATE COLUMN TABLE PAL_NLR_DATA_TBL ( "ID" INT,"Y" DOUBLE,"X1" DOUBLE); INSERT INTO PAL_NLR_DATA_TBL VALUES (0,10,1); INSERT INTO PAL_NLR_DATA_TBL VALUES (1,80,2); INSERT INTO PAL_NLR_DATA_TBL VALUES (2,130,3); INSERT INTO PAL_NLR_DATA_TBL VALUES (3,160,4); INSERT INTO PAL_NLR_DATA_TBL VALUES (4,180,5); INSERT INTO PAL_NLR_DATA_TBL VALUES (5,190,6); INSERT INTO PAL_NLR_DATA_TBL VALUES (6,192,7); DROP TABLE PAL_NLR_RESULTS_TBL; CREATE COLUMN TABLE PAL_NLR_RESULTS_TBL ("ID" INT,"Ai" DOUBLE); DROP TABLE PAL_NLR_FITTED_TBL; CREATE COLUMN TABLE PAL_NLR_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); DROP TABLE PAL_NLR_SIGNIFICANCE_TBL; CREATE COLUMN TABLE PAL_NLR_SIGNIFICANCE_TBL ("NAME" varchar(50),"VALUE" DOUBLE); DROP TABLE PAL_NLR_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_NLR_PMMLMODEL_TBL("ID" INT, "PMMLMODEL" VARCHAR(5000)); CALL _SYS_AFL.palLnR(PAL_NLR_DATA_TBL, "#PAL_CONTROL_TBL", PAL_NLR_RESULTS_TBL, PAL_NLR_FITTED_TBL, PAL_NLR_SIGNIFICANCE_TBL, PAL_NLR_PMMLMODEL_TBL) with overview; SELECT SELECT SELECT SELECT * * * * FROM FROM FROM FROM PAL_NLR_RESULTS_TBL; PAL_NLR_FITTED_TBL; PAL_NLR_SIGNIFICANCE_TBL; PAL_NLR_PMMLMODEL_TBL;
Expected Result PAL_NLR_RESULTS_TBL:
48
PAL_NLR_FITTED_TBL:
PAL_NLR_SIGNIFICANCE_TBL:
PAL_NLR_PMMLMODEL_TBL:
FORECASTWITHLNR
This function performs prediction with the natural logarithmic regression result. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','FORECASTWITHLNR', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Table Type Name <Predictive INPUT table type> <Coefficient INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in in out
PUBLIC
49
Procedure Calling CALL <procedure name>(<predictive input table>, <coefficient input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Predictive Data Column 1st column 2nd column Coefficient 1st column 2nd column Column Data Type Integer or varchar Integer or double Integer Integer, double, varchar, or CLOB Description ID Variable X ID (start from 0) Value Ai or PMML model. Varchar and CLOB types are only valid for PMML model.
Parameter Table Name THREAD_NUMBER MODEL_FORMAT Data Type Integer Integer Description Number of threads 0 (default): coefficients in table 1: PMML format
Output Table Table Fitted Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_FNLR_PREDICT_T; CREATE TYPE PAL_FNLR_PREDICT_T AS TABLE( "ID" INT,"X1" DOUBLE); Column Data Type Integer or varchar Integer or double Description ID Value Yi
50
DROP TYPE PAL_FNLR_COEFFICIENT_T; CREATE TYPE PAL_FNLR_COEFFICIENT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_FNLR_FITTED_T; CREATE TYPE PAL_FNLR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP table PAL_FNLR_PDATA_TBL; CREATE column table PAL_FNLR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_FNLR_PDATA_TBL values (1,'DM_PAL.PAL_FNLR_PREDICT_T','in'); insert into PAL_FNLR_PDATA_TBL values (2,'DM_PAL.PAL_FNLR_COEFFICIENT_T','in'); insert into PAL_FNLR_PDATA_TBL values (3,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_FNLR_PDATA_TBL values (4,'DM_PAL.PAL_FNLR_FITTED_T','out'); GRANT SELECT ON DM_PAL.PAL_FNLR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palForecastWithLnR','AFLPAL','FORECASTWITHLNR',PAL_FNLR_PDATA _TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); DROP TABLE PAL_FNLR_PREDICTDATA_TBL; CREATE COLUMN TABLE PAL_FNLR_PREDICTDATA_TBL ( "ID" INT,"X1" DOUBLE); INSERT INTO PAL_FNLR_PREDICTDATA_TBL VALUES (0,1); INSERT INTO PAL_FNLR_PREDICTDATA_TBL VALUES (1,2); INSERT INTO PAL_FNLR_PREDICTDATA_TBL VALUES (2,3); INSERT INTO PAL_FNLR_PREDICTDATA_TBL VALUES (3,4); INSERT INTO PAL_FNLR_PREDICTDATA_TBL VALUES (4,5); INSERT INTO PAL_FNLR_PREDICTDATA_TBL VALUES (5,6); INSERT INTO PAL_FNLR_PREDICTDATA_TBL VALUES (6,7); DROP TABLE PAL_FNLR_COEFFICIENT_TBL; CREATE COLUMN TABLE PAL_FNLR_COEFFICIENT_TBL ("ID" INT,"Ai" DOUBLE); INSERT INTO PAL_FNLR_COEFFICIENT_TBL VALUES (0,14.86160299); INSERT INTO PAL_FNLR_COEFFICIENT_TBL VALUES (1,98.29359746); DROP TABLE PAL_FNLR_FITTED_TBL; CREATE COLUMN TABLE PAL_FNLR_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); CALL _SYS_AFL.palForecastWithLnR(PAL_FNLR_PREDICTDATA_TBL, PAL_FNLR_COEFFICIENT_TBL, "#PAL_CONTROL_TBL", PAL_FNLR_FITTED_TBL) with overview; SELECT * FROM PAL_FNLR_FITTED_TBL; Expected Result PAL_FNLR_FITTED_TBL:
PUBLIC
51
3.2.3
C4.5 Decision Tree
A decision tree is used as a classifier for determining an appropriate action or decision among a predetermined set of actions for a given case. A decision tree helps you to effectively identify the factors to consider and how each factor has historically been associated with different outcomes of the decision. A decision tree uses a tree like structure of conditions and their possible consequences. Each node of a decision tree can be a leaf node or a decision node. Leaf node: mentions the value of the dependent (target) variable. Decision node: contains one condition that specifies some test on an attribute value. The outcome of the condition is further divided into branches with sub-trees or leaf nodes.
As a classification algorithm, C4.5 builds decision trees from a set of training data, using the concept of information entropy. The training data is a set of already classified samples. At each node of the tree, C4.5 chooses one attribute of the data that most effectively splits it into subsets in one class or the other. Its criterion is the normalized information gain (difference in entropy) that results from choosing an attribute for splitting the data. The attribute with the highest normalized information gain is chosen to make the decision. The C4.5 algorithm then proceeds recursively until meeting some stopping criteria such as the minimum number of cases in a leaf node. The C4.5 decision tree functions implemented in PAL support both discrete and continuous values. We discrete a continuous attribute by defining fixed intervals provided by users. For example, if the salary ranges from $100 to $20000, then we can form intervals like $0 $8000, $8000 $18000, and $18000 $20000. An attribute value will fall into any one of these intervals. In PAL implementation, the REP (Reduced Error Pruning) algorithm is used as pruning method.
Prerequisites
The column order and column number of the predicted data are the same as the order and number used in tree model building. The last column of the training data is used as a predicted field and is of discrete type. The predicted data set has an ID column. The input data does not contain null value. The algorithm will issue errors when encountering null values. The table used to store the tree model is a column table.
52
CREATEDT
This function creates a decision tree from the input training data. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','CREATEDT', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <PMML output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Training / Historical Data Column Columns Column Data Type Varchar, integer, or double Description Table used to build the predictive tree model Constraint Discrete value: integer or varchar Continuous value: integer or double Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <PMML OUTPUT table type> Direction in in out out
Parameter Table Name PERCENTAGE Data Type Double Description Specifies the percentage of the input data to be used to build the tree model. For example, if you set this parameter to 0.7, 70% of the training data will be used to build the tree model, and 30% will be used to prune the tree model.
PUBLIC
53
Name
Data Type
Description Default: 1.0
MIN_NUMS_RECORDS
Integer
Specifies the stop condition: if the number of records is less than the parameter value, the algorithm will stop splitting. Default: 0
MAX_DEPTH
Integer
Specifies the stop condition: if the depth of the tree model is greater than the parameter value, the algorithm will stop splitting. Default: the number of columns in the input table which contains the training data.
THREAD_NUMBER IS_SPLIT_MODEL
Integer Integer
Number of threads. Indicates whether the string of the tree model should be split or not. If the value is not 0, the tree model will be split, and the length of each unit is 5000. Default: 0
CONTINUOUS_COL
Integer or double
Indicates which column contains continuous variables: Integer value specifies the column position (column index starts from zero) Double value specifies the interval. If this value is not specified, the algorithm will automatically split this continuous value.
Default: IS_OUTPUT_RULES Integer String or integer is discrete attribute Double is continuous attribute
If this parameter is set to 1, the algorithm will extract all decision rules from the tree model and save them to the result table which is used to save the PMML model. Default: 0
54
PUBLIC
Name PMML_EXPORT
Data Type Integer
Description 0 (default): does not export PMML tree model. 1: exports PMML tree model in single row. 2: exports PMML tree model in several rows, each row containing a maximum of 5000 characters.
Output Tables Table Tree model of JSON format Column 1st column 2nd column Column Data Type Integer CLOB or varchar Description ID Tree model saved as The table must be a a JSON string. column table. The minimum length of every unit (row) is 5000. Tree model of PMML 1st column format 2nd column Integer CLOB or varchar ID Tree model in PMML format Constraint
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_CDT_DATA_T; CREATE TYPE PAL_CDT_DATA_T AS TABLE( "REGION" VARCHAR(50), "SALESPERIOD" VARCHAR(50), "REVENUE" Double, "CLASSLABEL" VARCHAR(50) ); DROP TYPE PAL_CDT_JSONMODEL_T; CREATE TYPE PAL_CDT_JSONMODEL_T AS TABLE("ID" INTEGER, "JSONMODEL" VARCHAR(5000)); DROP TYPE PAL_CDT_PMMLMODEL_T; CREATE TYPE PAL_CDT_PMMLMODEL_T AS TABLE("ID" INTEGER, "PMMLMODEL" VARCHAR(5000)); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INT_ARGS" INTEGER, "DOUBLE_ARGS" DOUBLE,
PUBLIC
55
);
"STRING_ARGS" VARCHAR(100)
--create procedure DROP TABLE PAL_CDT_PDATA_TBL; CREATE COLUMN TABLE PAL_CDT_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100)); INSERT INTO PAL_CDT_PDATA_TBL VALUES (1, 'DM_PAL.PAL_CDT_DATA_T', 'in'); INSERT INTO PAL_CDT_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_CDT_PDATA_TBL VALUES (3, 'DM_PAL.PAL_CDT_JSONMODEL_T', 'out'); INSERT INTO PAL_CDT_PDATA_TBL VALUES (4, 'DM_PAL.PAL_CDT_PMMLMODEL_T', 'out'); GRANT SELECT ON DM_PAL.PAL_CDT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_CREATEDT', 'AFLPAL', 'CREATEDT', PAL_CDT_PDATA_TBL); DROP TABLE PAL_CDT_TRAINING_TBL; CREATE COLUMN TABLE PAL_CDT_TRAINING_TBL LIKE PAL_CDT_DATA_T; INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('South', 'Winter', 100000, 'Good'); INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('North', 'Spring', 45000, 'Average'); INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('West', 'Summer', 30000, 'Poor'); INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('East', 'Autumn', 5000, 'Poor'); INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('West', 'Spring', 5000, 'Poor'); INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('East', 'Spring', 200000, 'Good'); INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('South', 'Summer', 25000, 'Poor'); INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('South', 'Spring', 10000, 'Average'); INSERT INTO PAL_CDT_TRAINING_TBL VALUES ('North', 'Winter', 50000, 'Average'); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL LIKE PAL_CONTROL_T; INSERT INTO #PAL_CONTROL_TBL VALUES ('PERCENTAGE',null,1.0,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('IS_SPLIT_MODEL',1,null,null); --INSERT INTO #PAL_CONTROL_TBL VALUES ('IS_OUTPUT_RULES',1,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PMML_EXPORT', 2, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CONTINUOUS_COL',2,25000,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CONTINUOUS_COL',2,60000,null); DROP TABLE PAL_CDT_JSONMODEL_TBL; CREATE COLUMN TABLE PAL_CDT_JSONMODEL_TBL LIKE PAL_CDT_JSONMODEL_T; DROP TABLE PAL_CDT_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_CDT_PMMLMODEL_TBL LIKE PAL_CDT_PMMLMODEL_T; CALL _SYS_AFL.PAL_CREATEDT(PAL_CDT_TRAINING_TBL, "#PAL_CONTROL_TBL", PAL_CDT_JSONMODEL_TBL, PAL_CDT_PMMLMODEL_TBL) with overview; SELECT * FROM PAL_CDT_JSONMODEL_TBL; SELECT * FROM PAL_CDT_PMMLMODEL_TBL; Expected Result PAL_CDT_JSONMODEL_TBL:
PAL_CDT_PMMLMODEL_TBL:
56
PREDICTWITHDT
This function uses decision trees to perform prediction. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'PREDICTWITHDT', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<data input table>, <parameter table>, <model input table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Predicted Data Column 1st column Other columns Predictive Model 1st column 2nd column Column Data Type Integer Varchar Integer Varchar Description ID Data to be classified (predicted) ID Serialized tree model Table Type Name <Data INPUT table type> <PARAMETER table type> <Model INPUT table type> <OUTPUT table type> Direction in in in out
Parameter Table
PUBLIC
57
Name THREAD_NUMBER MODEL_FORMAT
Data Type Integer Integer
Description Number of threads 0 (default): Deserializes the tree model from JSON format 1: Deserializes the tree model from PMML format
Output Table Table Result (tree model) Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_PCDT_DATA_T; CREATE TYPE PAL_PCDT_DATA_T AS TABLE( "ID" INTEGER, "REGION" VARCHAR(50), "SALESPERIOD" VARCHAR(50), "REVENUE" DOUBLE ); DROP TYPE PAL_PCDT_JSONMODEL_T; CREATE TYPE PAL_PCDT_JSONMODEL_T AS TABLE("ID" INTEGER, "JSONMODEL" VARCHAR(5000)); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR(50), "INT_ARGS" INTEGER, "DOUBLE_ARGS" DOUBLE, "STRING_ARGS" VARCHAR(100) ); DROP TYPE PAL_PCDT_RESULT_T; CREATE TYPE PAL_PCDT_RESULT_T AS TABLE("ID" INTEGER, "CLASSLABEL" VARCHAR(50)); -- create procedure DROP TABLE PAL_PCDT_PDATA_TBL; CREATE COLUMN TABLE PAL_PCDT_PDATA_TBL( "ID" INTEGER, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_PCDT_PDATA_TBL VALUES (1, 'DM_PAL.PAL_PCDT_DATA_T', 'in'); INSERT INTO PAL_PCDT_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_PCDT_PDATA_TBL VALUES (3, 'DM_PAL.PAL_PCDT_JSONMODEL_T', 'in'); INSERT INTO PAL_PCDT_PDATA_TBL VALUES (4, 'DM_PAL.PAL_PCDT_RESULT_T', 'out'); GRANT SELECT ON DM_PAL.PAL_PCDT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_PREDICTWITHDT', 'AFLPAL', 'PREDICTWITHDT', PAL_PCDT_PDATA_TBL); Column Data Type Integer Varchar Description ID Predictive result
58
DROP TABLE PAL_PCDT_DATA_TBL; CREATE COLUMN TABLE PAL_PCDT_DATA_TBL LIKE PAL_PCDT_DATA_T; INSERT INTO PAL_PCDT_DATA_TBL VALUES (0,'South', 'Autumn', 60000); INSERT INTO PAL_PCDT_DATA_TBL VALUES (1,'North', 'Spring', 30000); INSERT INTO PAL_PCDT_DATA_TBL VALUES (2,'South', 'Summer', 25000); INSERT INTO PAL_PCDT_DATA_TBL VALUES (3,'West', 'Winter', 5000); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL LIKE PAL_CONTROL_T; INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER', 2, null, null); DROP TABLE PAL_PCDT_RESULT_TBL; CREATE COLUMN TABLE PAL_PCDT_RESULT_TBL LIKE PAL_PCDT_RESULT_T; CALL _SYS_AFL.PAL_PREDICTWITHDT(PAL_PCDT_DATA_TBL, "#PAL_CONTROL_TBL", PAL_CDT_JSONMODEL_TBL, PAL_PCDT_RESULT_TBL) with overview; SELECT * FROM PAL_PCDT_RESULT_TBL; Expected Result PAL_PCDT_RESULT_TBL:
3.2.4
CHAID Decision Tree
CHAID stands for CHi-squared Automatic Interaction Detection. It is similar to the C4.5 decision tree. CHAID is a classification method for building decision trees by using chi-square statistics to identify optimal splits. CHAID examines the cross tabulations between each of the input fields and the outcome, and tests for significance using a chi-square independence test. If more than one of these relations is statistically significant, CHAID will select the input field that is the most significant (smallest p value). CHAID can generate non-binary trees.
Prerequisites
The column order and column number of the predicted data are the same as the order and number used in tree model building. The last column of the training data is used as a predicted field and is of discrete type. The predicted data set has an ID column. The input data does not contain null value. The algorithm will issue errors when encountering null values. The table used to store the tree model is a column table.
PUBLIC
59
CREATEDTWITHCHAID
This function creates a decision tree from the input training data. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'CREATEDTWITHCHAID', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <PMML output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Training / Historical Data Column Columns Column Data Type Varchar, integer, or double Description Table used to build the predictive tree model Constraint Discrete value: integer or varchar Continuous value: integer or double Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <PMML OUTPUT table type> Direction in in out out
Parameter Table Name PERCENTAGE Data Type Double Description Specifies the percentage of the input data to be used to build the tree model. For example, if you set this parameter to 0.7, 70% of the training data will be used to build the tree model, and 30% will be used to prune the tree model.
60
Name
Data Type
Description Default: 1.0
MIN_NUMS_RECORDS
Integer
Specifies the stop condition: if the number of records is less than the parameter value, the algorithm will stop splitting. Default: 0
MAX_DEPTH
Integer
Specifies the stop condition: if the depth of the tree model is greater than the parameter value, the algorithm will stop splitting. Default: the number of columns in the input table which contains the training data.
THREAD_NUMBER IS_SPLIT_MODEL
Integer Integer
Number of threads. Indicates whether the string of the tree model should be split or not. If the value is not 0, the tree model will be split, and the length of each unit is 5000. Default: 0
CONTINUOUS_COL
Integer or double
Indicates which column contains continuous variables: Integer value specifies the column position (column index starts from zero) Double value specifies the interval. If this value is not specified, the algorithm will automatically split this continuous value.
Default: IS_OUTPUT_RULES Integer String or integer is discrete attribute Double is continuous attribute
If this parameter is set to 1, the algorithm will extract all decision rules from the tree model and save them to the result table which is used to save the PMML model. Default: 0
PUBLIC
61
Name PMML_EXPORT
Data Type Integer
Description 0 (default): does not export PMML tree model. 1: exports PMML tree model in single row. 2: exports PMML tree model in several rows, each row containing a maximum of 5000 characters.
Output Tables Table Tree model of JSON format Column 1st column 2nd column Column Data Type Integer Varchar or CLOB Description ID Tree model saved as The table must be a a JSON string column table. The minimum length of each unit (row) is 5000. Tree model of PMML 1st column format 2nd column Integer CLOB or varchar ID Tree model in PMML format Constraint
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_CHDDT_DATA_T; CREATE TYPE PAL_CHDDT_DATA_T AS TABLE( "REGION" VARCHAR(50), "SALESPERIOD" VARCHAR(50), "REVENUE" DOUBLE, "CLASSLABEL" VARCHAR(50) ); DROP TYPE PAL_CHDDT_PMMLMODEL_T; CREATE TYPE PAL_CHDDT_PMMLMODEL_T AS TABLE("ID" INTEGER, "PMMLMODEL" VARCHAR(5000)); DROP TYPE PAL_CHDDT_JSONMODEL_T; CREATE TYPE PAL_CHDDT_JSONMODEL_T AS TABLE( "ID" INTEGER, "JSONMODEL" VARCHAR(5000)); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR(50), "INT_ARGS" INTEGER,
62
);
"DOUBLE_ARGS" DOUBLE, "STRING_ARGS" VARCHAR (100)
--create procedure DROP TABLE PAL_CHDDT_PDATA_TBL; CREATE COLUMN TABLE PAL_CHDDT_PDATA_TBL( "ID" INTEGER, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100)); INSERT INTO PAL_CHDDT_PDATA_TBL VALUES (1, 'DM_PAL.PAL_CHDDT_DATA_T', 'in'); INSERT INTO PAL_CHDDT_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_CHDDT_PDATA_TBL VALUES (3, 'DM_PAL.PAL_CHDDT_JSONMODEL_T', 'out'); INSERT INTO PAL_CHDDT_PDATA_TBL VALUES (4, 'DM_PAL.PAL_CHDDT_PMMLMODEL_T', 'out'); GRANT SELECT ON DM_PAL.PAL_CHDDT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_CREATEDT_WITH_CHAID', 'AFLPAL', 'CREATEDTWITHCHAID', PAL_CHDDT_PDATA_TBL); DROP TABLE PAL_CHDDT_TRAINING_TBL; CREATE COLUMN TABLE PAL_CHDDT_TRAINING_TBL LIKE PAL_CHDDT_DATA_T; INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('South', 'Winter', 100000, 'Good'); INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('North', 'Spring', 45000, 'Average'); INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('West', 'Summer', 30000, 'Poor'); INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('East', 'Autumn', 5000, 'Poor'); INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('West', 'Spring', 5000, 'Poor'); INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('East', 'Spring', 200000, 'Good'); INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('South', 'Summer', 25000, 'Poor'); INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('South', 'Spring', 10000, 'Average'); INSERT INTO PAL_CHDDT_TRAINING_TBL VALUES ('North', 'Winter', 50000, 'Average'); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL LIKE PAL_CONTROL_T; INSERT INTO #PAL_CONTROL_TBL VALUES ('PERCENTAGE',null,1.0,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('IS_SPLIT_MODEL',0,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MIN_NUMS_RECORDS',1,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CONTINUOUS_COL',2,25000,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CONTINUOUS_COL',2,60000,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PMML_EXPORT',2,null,null); DROP TABLE PAL_CHDDT_JSONMODEL_TBL; CREATE COLUMN TABLE PAL_CHDDT_JSONMODEL_TBL LIKE PAL_CHDDT_JSONMODEL_T; DROP TABLE PAL_CHDDT_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_CHDDT_PMMLMODEL_TBL LIKE PAL_CHDDT_PMMLMODEL_T; CALL _SYS_AFL.PAL_CREATEDT_WITH_CHAID(PAL_CHDDT_TRAINING_TBL, "#PAL_CONTROL_TBL", PAL_CHDDT_JSONMODEL_TBL, PAL_CHDDT_PMMLMODEL_TBL) with OVERVIEW; SELECT * FROM PAL_CHDDT_JSONMODEL_TBL; SELECT * FROM PAL_CHDDT_PMMLMODEL_TBL; Expected Result PAL_CHDDT_JSONMODEL_TBL:
PAL_CHDDT_PMMLMODEL_TBL:
PUBLIC
63
PREDICTWITHDT
This function uses decision trees to perform prediction. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'PREDICTWITHDT', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<data input table>, <parameter table>, <model input table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Predicted Data Column 1st column Other columns Predictive Model 1st column 2nd column Column Data Type Integer Varchar Integer Varchar Description ID Data to be classified (predicted) ID Serialized tree model Table Type Name <Data INPUT table type> <PARAMETER table type> <Model INPUT table type> <OUTPUT table type> Direction in in in out
Parameter Table
64
PUBLIC
Name THREAD_NUMBER MODEL_FORMAT
Data Type Integer Integer
Description Number of threads 0 (default): Deserializes the tree model from JSON format 1: Deserializes the tree model from PMML format
Output Table Table Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. -- Note: Before generating this model, make sure you have created the tree model using the CREATEDWITHCHAID function. SET SCHEMA DM_PAL; DROP TYPE PAL_PCHDDT_DATA_T; CREATE TYPE PAL_PCHDDT_DATA_T AS TABLE( "ID" INTEGER, "REGION" VARCHAR(50), "SALESPERIOD" VARCHAR(50), "REVENUE" DOUBLE ); DROP TYPE PAL_PCHDDT_JSONMODEL_T; CREATE TYPE PAL_PCHDDT_JSONMODEL_T AS TABLE("ID" INTEGER, "JSONMODEL" VARCHAR(5000)); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR(50), "INT_ARGS" INTEGER, "DOUBLE_ARGS" DOUBLE, "STRING_ARGS" VARCHAR (100) ); DROP TYPE PAL_PCHDDT_RESULT_T; CREATE TYPE PAL_PCHDDT_RESULT_T AS TABLE("ID" INTEGER, "CLASSLABEL" VARCHAR(50)); -- create procedure DROP TABLE PAL_PCHDDT_PDATA_TBL; CREATE COLUMN TABLE PAL_PCHDDT_PDATA_TBL("ID" INTEGER, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_PCHDDT_PDATA_TBL VALUES (1, 'DM_PAL.PAL_PCHDDT_DATA_T', 'in'); INSERT INTO PAL_PCHDDT_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_PCHDDT_PDATA_TBL VALUES (3, 'DM_PAL.PAL_PCHDDT_JSONMODEL_T', 'in'); INSERT INTO PAL_PCHDDT_PDATA_TBL VALUES (4, 'DM_PAL.PAL_PCHDDT_RESULT_T', 'out'); Column Data Type Integer Varchar Description ID Predictive result
PUBLIC
65
GRANT SELECT ON DM_PAL.PAL_PCHDDT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_PREDICTWITHDT', 'AFLPAL', 'PREDICTWITHDT', PAL_PCHDDT_PDATA_TBL); DROP TABLE PAL_PCHDDT_DATA_TBL; CREATE COLUMN TABLE PAL_PCHDDT_DATA_TBL LIKE PAL_PCHDDT_DATA_T; INSERT INTO PAL_PCHDDT_DATA_TBL VALUES (0,'South', 'Autumn', 60000); INSERT INTO PAL_PCHDDT_DATA_TBL VALUES (1,'North', 'Spring', 30000); INSERT INTO PAL_PCHDDT_DATA_TBL VALUES (2,'South', 'Summer', 25000); INSERT INTO PAL_PCHDDT_DATA_TBL VALUES (3,'West', 'Winter', 5000); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL LIKE PAL_CONTROL_T; INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); DROP TABLE PAL_PCHDDT_RESULT_TBL; CREATE COLUMN TABLE PAL_PCHDDT_RESULT_TBL LIKE PAL_PCHDDT_RESULT_T; CALL _SYS_AFL.PAL_PREDICTWITHDT(PAL_PCHDDT_DATA_TBL, "#PAL_CONTROL_TBL", PAL_CHDDT_JSONMODEL_TBL, PAL_PCHDDT_RESULT_TBL) with OVERVIEW; SELECT * FROM PAL_PCHDDT_RESULT_TBL; Expected Result PAL_PCHDDT_RESULT_TBL:
3.2.5
Exponential Regression
Exponential regression is an approach to modeling the relationship between a scalar variable y and one or more variables denoted X. In exponential regression, data is modeled using exponential functions, and unknown model parameters are estimated from the data. Such models are called exponential models. In PAL, the implementation of exponential regression is to transform to linear regression and solve it: y = 0 exp(1 x1 + 2 x2 + + n xn) Where 0n are parameters that need to be calculated. The steps are: 1. 2. 3. Put natural logarithmic operation on both sides: ln(y) = ln(0 exp(1 x1 + 2 x2 + + n xn)) Transform it into: ln(y) = ln(0) + 1 x1 + 2 x2 + + n xn Let y = ln(y), 0 = ln(0) y = 0 + 1 x1 + 2 x2 + + n xn
66
Thus, y and x1xn is a linear relationship and can be solved using the linear regression method. The implementation also supports calculating the F value and R^2 to determine statistical significance.
Prerequisites
No missing or null data in the inputs. The data is numeric, not categorical. Given the structure as Y and X1...Xn, there are more than n+1 records available for analysis.
EXPREGRESSION
This is an exponential regression function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','EXPREGRESSION', <signature table>); The signature table should contain the following records: Index 1 2 3 4 5 6 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <fitted output table>, <significance output table>, <PMML output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column Column Data Type Integer or varchar Description ID Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <Fitted OUTPUT table type> <Significance OUTPUT table type> <PMML OUTPUT table type> Direction in in out out out out
PUBLIC
67
Table
Column 2nd column Other columns
Column Data Type Integer or double Integer or double
Description Variable y Variable Xn
Parameter Table Name VARIABLE_NUM Data Type Integer Description (Optional) Specifies the number of independent variables (Xi). Default value: all Xi columns in trainingDataTab Customized value: first n Xi in trainingDataTab
THREAD_NUMBER ALG
Integer Integer
Number of threads (Optional) Specifies decomposition method: 0 (default): Doolittle decomposition (LU) 2: Singular value decomposition (SVD) 0 (default): does not output adjusted R square 1: outputs adjusted R square 0 (default): does not export exponential regression model in PMML. 1: exports exponential regression model in PMML in single row. 2: exports exponential regression model in PMML in several rows, each row containing a maximum of 5000 characters.
ADJUSTED_R2
Integer
PMML_EXPORT
Integer
Output Tables Table Result Column 1st column 2nd column Column Data Type Integer Integer or double Description ID Value Ai Constraint
68
Table
Column
Column Data Type
Description A0: the intercept A1: the beta coefficient for X1 A2: the beta coefficient for X2
Constraint
Fitted Data
Integer or varchar Integer or double Varchar Double Integer CLOB or varchar
ID Value Yi Name Value ID Exponential regression model in PMML format (R^2 / F)
Significance
PMML Result
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_ER_DATA_T; CREATE TYPE PAL_ER_DATA_T AS TABLE( "ID" INT,"Y" DOUBLE,"X1" DOUBLE, "X2" DOUBLE); DROP TYPE PAL_ER_RESULT_T; CREATE TYPE PAL_ER_RESULT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_ER_FITTED_T; CREATE TYPE PAL_ER_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP TYPE PAL_ER_SIGNIFICANCE_T; CREATE TYPE PAL_ER_SIGNIFICANCE_T AS TABLE("NAME" varchar(50),"VALUE" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_ER_PMMLMODEL_T; CREATE TYPE PAL_ER_PMMLMODEL_T AS TABLE("ID" INT,"Model" varchar(5000)); DROP table PAL_ER_PDATA_TBL; CREATE column table PAL_ER_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_ER_PDATA_TBL values (1,'DM_PAL.PAL_ER_DATA_T','in'); insert into PAL_ER_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_ER_PDATA_TBL values (3,'DM_PAL.PAL_ER_RESULT_T','out'); insert into PAL_ER_PDATA_TBL values (4,'DM_PAL.PAL_ER_FITTED_T','out'); insert into PAL_ER_PDATA_TBL values (5,'DM_PAL.PAL_ER_SIGNIFICANCE_T','out'); insert into PAL_ER_PDATA_TBL values (6,'DM_PAL.PAL_ER_PMMLMODEL_T','out');
PUBLIC
69
GRANT SELECT ON DM_PAL.PAL_ER_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palExpR','AFLPAL','EXPREGRESSION',PAL_ER_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PMML_EXPORT',2,null,null); DROP TABLE PAL_ER_DATA_TBL; CREATE COLUMN TABLE PAL_ER_DATA_TBL ( "ID" INT,"Y" DOUBLE,"X1" DOUBLE, "X2" DOUBLE); INSERT INTO PAL_ER_DATA_TBL VALUES (0,0.5,0.13,0.33); INSERT INTO PAL_ER_DATA_TBL VALUES (1,0.15,0.14,0.34); INSERT INTO PAL_ER_DATA_TBL VALUES (2,0.25,0.15,0.36); INSERT INTO PAL_ER_DATA_TBL VALUES (3,0.35,0.16,0.35); INSERT INTO PAL_ER_DATA_TBL VALUES (4,0.45,0.17,0.37); INSERT INTO PAL_ER_DATA_TBL VALUES (5,0.55,0.18,0.38); INSERT INTO PAL_ER_DATA_TBL VALUES (6,0.65,0.19,0.39); INSERT INTO PAL_ER_DATA_TBL VALUES (7,0.75,0.19,0.31); INSERT INTO PAL_ER_DATA_TBL VALUES (8,0.85,0.11,0.32); INSERT INTO PAL_ER_DATA_TBL VALUES (9,0.95,0.12,0.33); DROP TABLE PAL_ER_RESULTS_TBL; CREATE COLUMN TABLE PAL_ER_RESULTS_TBL ("ID" INT,"Ai" DOUBLE); DROP TABLE PAL_ER_FITTED_TBL; CREATE COLUMN TABLE PAL_ER_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); DROP TABLE PAL_ER_SIGNIFICANCE_TBL; CREATE COLUMN TABLE PAL_ER_SIGNIFICANCE_TBL ("NAME" varchar(50),"VALUE" DOUBLE); DROP TABLE PAL_ER_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_ER_PMMLMODEL_TBL ("ID" INT, "PMMLMODEL" VARCHAR(5000)); CALL _SYS_AFL.palExpR(PAL_ER_DATA_TBL, "#PAL_CONTROL_TBL", PAL_ER_RESULTS_TBL, PAL_ER_FITTED_TBL, PAL_ER_SIGNIFICANCE_TBL, PAL_ER_PMMLMODEL_TBL) with overview; SELECT * FROM PAL_ER_RESULTS_TBL; SELECT * FROM PAL_ER_FITTED_TBL; SELECT * FROM PAL_ER_SIGNIFICANCE_TBL; SELECT * FROM PAL_ER_PMMLMODEL_TBL; Expected Result PAL_ER_RESULTS_TBL:
PAL_ER_FITTED_TBL:
70
PAL_ER_SIGNIFICANCE_TBL:
PAL_ER_PMMLMODEL_TBL:
FORECASTWITHEXPR
This function performs prediction with the exponential regression result. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','FORECASTWITHEXPR', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<data input table>, <coefficient input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. Table Type Name <Data INPUT table type> <Coefficient INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in in out
PUBLIC
71
The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Predictive Data Column 1st column Other columns Coefficient 1st column 2nd column Column Data Type Integer or varchar Integer or double Integer Integer, double, varchar, or CLOB Description ID Variable Xn ID (start from 0) Value Ai or PMML model. Varchar and CLOB types are only valid for PMML model.
Output Table Table Fitted Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_FER_PREDICT_T; CREATE TYPE PAL_FER_PREDICT_T AS TABLE( "ID" INT,"X1" DOUBLE, "X2" DOUBLE); DROP TYPE PAL_FER_COEFFICIENT_T; CREATE TYPE PAL_FER_COEFFICIENT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_FER_FITTED_T; Column Data Type Integer or varchar Integer or double Description ID Value Yi
72
PUBLIC
CREATE TYPE PAL_FER_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP table PAL_FER_PDATA_TBL; CREATE column table PAL_FER_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_FER_PDATA_TBL values (1,'DM_PAL.PAL_FER_PREDICT_T','in'); insert into PAL_FER_PDATA_TBL values (2,'DM_PAL.PAL_FER_COEFFICIENT_T','in'); insert into PAL_FER_PDATA_TBL values (3,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_FER_PDATA_TBL values (4,'DM_PAL.PAL_FER_FITTED_T','out'); GRANT SELECT ON DM_PAL.PAL_FER_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palForecastWithExpR','AFLPAL','FORECASTWITHEXPR',PAL_FER_PDAT A_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); DROP TABLE PAL_FER_PREDICTDATA_TBL; CREATE COLUMN TABLE PAL_FER_PREDICTDATA_TBL ("ID" INT,"X1" DOUBLE, "X2" DOUBLE); INSERT INTO PAL_FER_PREDICTDATA_TBL VALUES (0,0.5,0.3); INSERT INTO PAL_FER_PREDICTDATA_TBL VALUES (1,4,0.4); INSERT INTO PAL_FER_PREDICTDATA_TBL VALUES (2,0,1.6); INSERT INTO PAL_FER_PREDICTDATA_TBL VALUES (3,0.3,0.45); INSERT INTO PAL_FER_PREDICTDATA_TBL VALUES (4,0.4,1.7); DROP TABLE PAL_FER_COEEFICIENT_TBL; CREATE COLUMN TABLE PAL_FER_COEEFICIENT_TBL ("ID" INT,"Ai" DOUBLE); INSERT INTO PAL_FER_COEEFICIENT_TBL VALUES (0,1.7120914258645001); INSERT INTO PAL_FER_COEEFICIENT_TBL VALUES (1,0.2652771198483208); INSERT INTO PAL_FER_COEEFICIENT_TBL VALUES (2,-3.471103742302148); DROP TABLE PAL_FER_FITTED_TBL; CREATE COLUMN TABLE PAL_FER_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); CALL _SYS_AFL.palForecastWithExpR(PAL_FER_PREDICTDATA_TBL, PAL_FER_COEEFICIENT_TBL, "#PAL_CONTROL_TBL", PAL_FER_FITTED_TBL) with overview; SELECT * FROM PAL_FER_FITTED_TBL; Expected Result PAL_FER_FITTED_TBL:
3.2.6
KNN
K-Nearest Neighbor (KNN) is a memory based classification method with no explicit training phase. In the testing phase, given a query sample x, its top K nearest samples is found in the training set first, then the label of x is assigned as the most frequent label of the K nearest neighbors. In this release of PAL, the description of each sample should be real numbers.
PUBLIC
73
Prerequisites
The first column of the training data and input data is an ID column. The second column of the training data is of class type. The class type column is of integer type. Other data columns are of integer or double type. The input data does not contain null value.
KNN
This is a classification function using the KNN algorithm. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','KNN', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<training input table>, <class input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Training Data Column 1st column 2nd column Other columns Class Data 1st column Other columns Column Data Type Integer or varchar Integer or varchar Integer or double Integer or varchar Integer or double Description ID Class type Attribute data ID Attribute data Table Type Name <Training INPUT table type> <Class INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in in out
Parameter Table
74
Name K_NEAREST_NEIGHBOURS
Data Type Integer
Description Number of nearest neighbors (k). Default value: 1
ATTRIBUTE_NUM
Integer
Number of attributes. Default value: 1
VOTING_TYPE
Integer
Voting type: 0 = majority voting 1 = distance-weighted voting
Default value: 1 THREAD_NUMBER Integer Number of threads. Default value: 1
Output Table Table Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_KNN_DATA_T; CREATE TYPE PAL_KNN_DATA_T AS TABLE( "ID" INT,"TYPE" INT,"X1" DOUBLE, "X2" DOUBLE); DROP TYPE PAL_KNN_CLASSDATA_T; CREATE TYPE PAL_KNN_CLASSDATA_T AS TABLE( "ID" INT,"X1" DOUBLE, "X2" DOUBLE); DROP TYPE PAL_KNN_RESULT_T; CREATE TYPE PAL_KNN_RESULT_T AS TABLE("ID" INT,"Type" INT); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP table PAL_KNN_PDATA_TBL; CREATE column table PAL_KNN_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert insert insert insert into into into into PAL_KNN_PDATA_TBL PAL_KNN_PDATA_TBL PAL_KNN_PDATA_TBL PAL_KNN_PDATA_TBL values values values values (1,'DM_PAL.PAL_KNN_DATA_T','in'); (2,'DM_PAL.PAL_KNN_CLASSDATA_T','in'); (3,'DM_PAL.PAL_CONTROL_T','in'); (4,'DM_PAL.PAL_KNN_RESULT_T','out'); Column Data Type Integer or varchar Integer or varchar Description ID Class type
PUBLIC
75
GRANT SELECT ON DM_PAL.PAL_KNN_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palKNN','AFLPAL','KNN',PAL_KNN_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('K_NEAREST_NEIGHBOURS',3,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('ATTRIBUTE_NUM',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('VOTING_TYPE',0,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); DROP TABLE PAL_KNN_DATA_TBL; CREATE COLUMN TABLE PAL_KNN_DATA_TBL ( "ID" INT,"TYPE" INT,"X1" DOUBLE, "X2" DOUBLE); INSERT INTO PAL_KNN_DATA_TBL VALUES (0,2,1,1); INSERT INTO PAL_KNN_DATA_TBL VALUES (1,3,10,10); INSERT INTO PAL_KNN_DATA_TBL VALUES (2,3,10,11); INSERT INTO PAL_KNN_DATA_TBL VALUES (3,3,10,10); INSERT INTO PAL_KNN_DATA_TBL VALUES (4,1,1000,1000); INSERT INTO PAL_KNN_DATA_TBL VALUES (5,1,1000,1001); INSERT INTO PAL_KNN_DATA_TBL VALUES (6,1,1000,999); INSERT INTO PAL_KNN_DATA_TBL VALUES (7,1,999,999); INSERT INTO PAL_KNN_DATA_TBL VALUES (8,1,999,1000); INSERT INTO PAL_KNN_DATA_TBL VALUES (9,1,1000,1000); DROP TABLE PAL_KNN_CLASSDATA_TBL; CREATE COLUMN TABLE PAL_KNN_CLASSDATA_TBL ( "ID" INT,"X1" DOUBLE, "X2" DOUBLE); INSERT INTO PAL_KNN_CLASSDATA_TBL VALUES (0,2,1); INSERT INTO PAL_KNN_CLASSDATA_TBL VALUES (1,9,10); INSERT INTO PAL_KNN_CLASSDATA_TBL VALUES (2,9,11); INSERT INTO PAL_KNN_CLASSDATA_TBL VALUES (3,15000,15000); INSERT INTO PAL_KNN_CLASSDATA_TBL VALUES (4,1000,1000); INSERT INTO PAL_KNN_CLASSDATA_TBL VALUES (5,500,1001); INSERT INTO PAL_KNN_CLASSDATA_TBL VALUES (6,500,999); INSERT INTO PAL_KNN_CLASSDATA_TBL VALUES (7,199,999); DROP TABLE PAL_KNN_RESULTS_TBL; CREATE COLUMN TABLE PAL_KNN_RESULTS_TBL ("ID" INT,"Type" INT); CALL _SYS_AFL.palKNN(PAL_KNN_DATA_TBL, PAL_KNN_CLASSDATA_TBL, "#PAL_CONTROL_TBL", PAL_KNN_RESULTS_TBL) with overview; SELECT * FROM PAL_KNN_RESULTS_TBL; Expected Result PAL_KNN_RESULTS_TBL:
76
3.2.7
Logistic Regression
Logistic regression is used to model the linear relationship between a categorical dependent variable (also known as explained variable) and some continuous independent variables (also known as explanatory variables). Since the dependent variable is discrete, logistic regression model can also be treated as a classifier, which is able to predict the label (dependent variable) of a sample using its continuous features (independent variables). In this release of PAL, only binary categorical variable, that is, a two-class classifier, is supported. In PAL, the logistic regression model is made by: h(x) = g(T x) = 1/(1 + exp(T x)) Where T x = 0x0 + 1x1 + + nxn Assuming that there are only two class labels, {0,1}, you can get the below formula: P(y = 1 | x; ) = h(x) P(y = 0 | x; ) = 1 h(x) And merge them into: P(y | x;) = h(x)y (1 h(x))1-y Where 0, 1, , n are regression coefficients and their values can be obtained through the Maximum Likelihood Estimation (MLE) method. The log likelihood function is:
Function LOGISTICREGRESSION is used to compute the coefficients from training data. Two methods are provided to minimize the objective function: Newton method and Stochastic Gradient Descent (SGD), and you can choose one of them. For fast convergence, the Newton method is preferred; Function FORECASTWITHLOGISTICR is used to predict the labels for the testing data.
Prerequisites
No missing or null data in inputs. Data is numeric, not categorical. Given the structure as Y and X1...Xn, there must be more than n+1 records available for analysis.
PUBLIC
77
LOGISTICREGRESSION
This is a logistic regression function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL', 'LOGISTICREGRESSION', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <PMML output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column Columns Type column Column Data Type Integer or double Integer or varchar Description Variable Xn Variable TYPE Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <PMML OUTPUT table type> Direction in in out out
Parameter Table Name VARIABLE_NUM Data Type Integer Description (Optional) Specifies the number of independent variables (Xi). Default value: all Xi columns in DataTab METHOD Integer 0 (default and recommended): uses the Newton iteration method. 1: uses the gradient-decent method.
78
PUBLIC
Name STEP_SIZE
Data Type Double
Description Step size for convergence. This parameter is used only when METHOD is 1. Default value: 0.0201
EXIT_THRESHOLD
Double
Threshold (actual value) for exiting the iterations. Default value: 0.000001
THREAD_NUMBER
Integer
Number of threads. Note: It is recommended to specify this parameter to a value equal to or greater than 4.
MAX_ITERATION
Integer
Maximum number of iterations. Default value: 1
CLASS_MAP0 CLASS_MAP1 PMML_EXPORT
String String Integer
The Variable type which is mapped to 0. The Variable type which is mapped to 1. 0 (default): does not export logistic regression model in PMML. 1: exports logistic regression model in PMML in single row. 2: exports logistic regression model in PMML in several rows, each row containing a maximum of 5000 characters.
Output Table Table Result Column 1st column 2nd column Column Data Type Integer Integer or double Description ID Value Ai A0: intercept A1: beta coefficient for X1 A2: beta coefficient for X2
PUBLIC
79
Table PMML Result (logistic regression model)
Column 1st column 2nd column
Column Data Type Integer CLOB or varchar
Description ID Logistic regression model in PMML format
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_LOGISTICR_DATA_T; CREATE TYPE PAL_LOGISTICR_DATA_T AS TABLE("X1" DOUBLE,"X2" DOUBLE,"TYPE" INT); DROP TYPE PAL_LOGISTICR_RESULT_T; CREATE TYPE PAL_LOGISTICR_RESULT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "Name" VARCHAR (50),"intArgs" INTEGER,"doubleArgs" DOUBLE,"stringArgs" VARCHAR (100)); DROP TYPE PAL_LOGISTICR_PMMLMODEL_T; CREATE TYPE PAL_LOGISTICR_PMMLMODEL_T AS TABLE( "ID" INT, "PMMLMODEL" VARCHAR(5000) ); DROP table PAL_LOGISTICR_PDATA_TBL; CREATE column table PAL_LOGISTICR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_LOGISTICR_PDATA_TBL values (1,'DM_PAL.PAL_LOGISTICR_DATA_T','in'); insert into PAL_LOGISTICR_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_LOGISTICR_PDATA_TBL values (3,'DM_PAL.PAL_LOGISTICR_RESULT_T','out'); insert into PAL_LOGISTICR_PDATA_TBL values (4,'DM_PAL.PAL_LOGISTICR_PMMLMODEL_T','out'); GRANT SELECT ON DM_PAL.PAL_LOGISTICR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palLogisticR','AFLPAL','LOGISTICREGRESSION',PAL_LOGISTICR_PDAT A_TBL); DROP TABLE PAL_LOGISTICR_DATA_TBL; CREATE COLUMN TABLE PAL_LOGISTICR_DATA_TBL ("X1" DOUBLE,"X2"DOUBLE,"TYPE" INT); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (110,2.62,1); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (110,2.875,1); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (93,2.32,1); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (110,3.215,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (175,3.44,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (105,3.46,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (245,3.57,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (62,3.19,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (95,3.15,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (123,3.44,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (123,3.44,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (180,4.07,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (180,3.73,0); INSERT INTO PAL_LOGISTICR_DATA_TBL VALUES (180,3.78,0);
80
INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT
INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO
PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL PAL_LOGISTICR_DATA_TBL
VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES
(205,5.25,0); (215,5.424,0); (230,5.345,0); (66,2.2,1); (52,1.615,1); (65,1.835,1); (97,2.465,0); (150,3.52,0); (150,3.435,0); (245,3.84,0); (175,3.845,0); (66,1.935,1); (91,2.14,1); (113,1.513,1); (264,3.17,1); (175,2.77,1); (335,3.57,1); (109,2.78,1);
DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE "intArgs" INTEGER, "doubleArgs" DOUBLE, "stringArgs" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES
#PAL_CONTROL_TBL ( "Name" VARCHAR (50),
('EXIT_THRESHOLD',null,0.00001,null); ('THREAD_NUMBER',8,null,null); ('MAX_ITERATION',80,null,null); ('PMML_EXPORT', 1, null, null); ('METHOD', 0, null, null);
DROP TABLE PAL_LOGISTICR_RESULTS_TBL; CREATE COLUMN TABLE PAL_LOGISTICR_RESULTS_TBL ("ID" INT,"Ai" DOUBLE); DROP TABLE PAL_LOGISTICR_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_LOGISTICR_PMMLMODEL_TBL( "ID" INT, "PMMLMODEL" VARCHAR(5000) ); CALL _SYS_AFL.palLogisticR(PAL_LOGISTICR_DATA_TBL, "#PAL_CONTROL_TBL", PAL_LOGISTICR_RESULTS_TBL, PAL_LOGISTICR_PMMLMODEL_TBL) with overview; SELECT * FROM PAL_LOGISTICR_RESULTS_TBL; SELECT * FROM PAL_LOGISTICR_PMMLMODEL_TBL; Expected Result PAL_LOGISTICR_RESULTS_TBL:
PAL_LOGISTICR_PMMLMODEL_TBL:
PUBLIC
81
FORECASTWITHLOGISTICR
This function performs predication with logistic regression result. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL', 'FORECASTWITHLOGISTICR', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<data input table>, <parameter table>, table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Predictive Data Column 1st column Other columns Coefficient 1st column 2nd column Column Data Type Integer Integer or double Integer Integer or double Description ID Variable Xn ID Value Ai <coefficient input Table Type Name <Data INPUT table type> <PARAMETER table type> <Coefficient INPUT table type> <OUTPUT table type> Direction in in in out
Parameter Table Name THREAD_NUMBER Data Type Integer Description Number of threads
82
PUBLIC
Name CLASS_MAP0
Data Type String
Description The same value as LOGISTICREGRESSIONs parameter The same value as LOGISTICREGRESSIONs parameter
CLASS_MAP1
String
Output Table Table Fitted Result Column 1st column 2nd column 3rd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. set schema DM_PAL; DROP TYPE PAL_FLOGISTICR_PREDICT_T; CREATE TYPE PAL_FLOGISTICR_PREDICT_T AS TABLE("ID" INT,"X1" DOUBLE,"X2" DOUBLE); DROP TYPE PAL_FLOGISTICR_COEFFICIENT_T; CREATE TYPE PAL_FLOGISTICR_COEFFICIENT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_FLOGISTICR_FITTED_T; CREATE TYPE PAL_FLOGISTICR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE,"TYPE" INT); DROP table PAL_FLOGISTICR_PDATA_TBL; CREATE column table PAL_FLOGISTICR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_FLOGISTICR_PDATA_TBL values (1,'DM_PAL.PAL_FLOGISTICR_PREDICT_T','in'); insert into PAL_FLOGISTICR_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_FLOGISTICR_PDATA_TBL values (3,'DM_PAL.PAL_FLOGISTICR_COEFFICIENT_T','in'); insert into PAL_FLOGISTICR_PDATA_TBL values (4,'DM_PAL.PAL_FLOGISTICR_FITTED_T','out'); GRANT SELECT ON DM_PAL.PAL_FLOGISTICR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palForecastWithLogisticR','AFLPAL','FORECASTWITHLOGISTICR',PAL_F LOGISTICR_PDATA_TBL); DROP TABLE PAL_FLOGISTICR_PREDICTDATA_TBL; CREATE COLUMN TABLE PAL_FLOGISTICR_PREDICTDATA_TBL ( "ID" INT,"X1" DOUBLE, "X2" Column Data Type Integer Integer or double Integer or varchar Description ID Value Yi TYPE
PUBLIC
83
DOUBLE); INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO INSERT INTO
PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL PAL_FLOGISTICR_PREDICTDATA_TBL
VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES
(0,120,2.8); (1,110,2.875); (2,93,2.32); (3,110,3.215); (4,175,3.44); (5,105,3.46); (6,245,3.57); (7,62,3.19); (8,95,3.15); (9,123,3.44); (10,123,3.44); (11,180,4.07); (12,180,3.73); (13,180,3.78); (14,205,5.25); (15,215,5.424); (16,230,5.345); (17,66,2.2); (18,52,1.615); (19,65,1.835); (20,97,2.465); (21,150,3.52); (22,150,3.435); (23,245,3.84); (24,175,3.845); (25,66,1.935); (26,91,2.14); (27,113,1.513); (28,264,3.17); (29,175,2.77); (30,335,3.57); (31,109,2.78);
DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); DROP TABLE PAL_FLOGISTICR_COEEFICIENT_TBL; CREATE COLUMN TABLE PAL_FLOGISTICR_COEEFICIENT_TBL ("ID" INT,"Ai" DOUBLE); INSERT INTO PAL_FLOGISTICR_COEEFICIENT_TBL VALUES (0, 18.866298717199392); INSERT INTO PAL_FLOGISTICR_COEEFICIENT_TBL VALUES (1, 0.03625559608220791); INSERT INTO PAL_FLOGISTICR_COEEFICIENT_TBL VALUES (2, -8.08347518244258); DROP TABLE PAL_FLOGISTICR_FITTED_TBL; CREATE COLUMN TABLE PAL_FLOGISTICR_FITTED_TBL ("ID" INT, "Fitted" DOUBLE,"TYPE" INT); CALL _SYS_AFL.palForecastWithLogisticR(PAL_FLOGISTICR_PREDICTDATA_TBL, "#PAL_CONTROL_TBL", PAL_FLOGISTICR_COEEFICIENT_TBL, PAL_FLOGISTICR_FITTED_TBL) with overview; SELECT * FROM PAL_FLOGISTICR_FITTED_TBL; Expected Result PAL_FLOGISTICR_FITTED_TBL:
84
3.2.8
Multiple Linear Regression
Linear regression is an approach to modeling the linear relationship between a variable Y, usually referred to as the dependent variable, and one or more variables, usually referred to as independent variables, denoted X1, X2, X3 In linear regression, data are modeled using linear functions, and unknown model parameters are estimated from the data. Such models are called linear models. According to linear least-squares estimation, linear regression is to solve the following equation: (AT A)X = (AT y) Where A is MxN matrix, x is Nx1 matrix, and y is Mx1 matrix. The implementation also supports calculating F and R^2 to determine statistical significance.
PUBLIC
85
Prerequisites
LRREGRESSION
This is a multiple linear regression function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','LRREGRESSION', <signature table>); The signature table should contain the following records: Index 1 2 3 4 5 6 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <fitted output table>, <significance output table>, <PMML output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column Other columns Column Data Type Integer or varchar Integer or double Integer or double Description ID Variable y Variable Xn Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <Fitted OUTPUT table type> <Significance OUTPUT table type> <PMML OUTPUT table type> Direction in in out out out out
Parameter Table
86
Name VARIABLE_NUM
Data Type Integer
Description (Optional) Specifies the number of independent variables (Xi). Default value: all Xi columns in trainingDataTab Customized value: first n Xi in trainingDataTab
THREAD_NUMBER ALG
Integer Integer
Specifies the number of threads. (Optional) Specifies decomposition method: 0 (default): Doolittle decomposition (LU) 2: Singular value decomposition (SVD) 0 (default): does not output adjusted R square 1: outputs adjusted R square 0 (default): does not export multiple linear regression model in PMML. 1: exports multiple linear regression model in PMML in single row. 2: exports multiple linear regression model in PMML in several rows, each row containing a maximum of 5000 characters.
ADJUSTED_R2
Integer
PMML_EXPORT
Integer
Output Tables Table Result Column 1st column 2nd column Column Data Type Integer Integer or double Description ID Value Ai Fitted Data 1st column Integer or varchar ID A0: the intercept A1: the beta coefficient for X1 A2: the beta coefficient for X2 ... Constraint
PUBLIC
87
Table
Column 2nd column
Column Data Type Integer or double Varchar Double Integer CLOB or varchar
Description Value Yi Name Value ID Multiple linear regression model in PMML format
Constraint
Significance
(R^2 / F)
PMML Result
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_MLR_DATA_T; CREATE TYPE PAL_MLR_DATA_T AS TABLE( "ID" INT,"Y" DOUBLE,"X1" DOUBLE, "X2" DOUBLE); DROP TYPE PAL_MLR_RESULT_T; CREATE TYPE PAL_MLR_RESULT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_MLR_FITTED_T; CREATE TYPE PAL_MLR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP TYPE PAL_MLR_SIGNIFICANCE_T; CREATE TYPE PAL_MLR_SIGNIFICANCE_T AS TABLE("NAME" varchar(50),"VALUE" DOUBLE); DROP TYPE PAL_MLR_PMMLMODEL_T; CREATE TYPE PAL_MLR_PMMLMODEL_T AS TABLE("ID" INT,"Model" varchar(5000)); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP table PAL_MLR_PDATA_TBL; CREATE column table PAL_MLR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_MLR_PDATA_TBL values (1,'DM_PAL.PAL_MLR_DATA_T','in'); insert into PAL_MLR_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_MLR_PDATA_TBL values (3,'DM_PAL.PAL_MLR_RESULT_T','out'); insert into PAL_MLR_PDATA_TBL values (4,'DM_PAL.PAL_MLR_FITTED_T','out'); insert into PAL_MLR_PDATA_TBL values (5,'DM_PAL.PAL_MLR_SIGNIFICANCE_T','out'); insert into PAL_MLR_PDATA_TBL values (6,'DM_PAL.PAL_MLR_PMMLMODEL_T','out'); GRANT SELECT ON DM_PAL.PAL_MLR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palLR','AFLPAL','LRREGRESSION',PAL_MLR_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PMML_EXPORT',0,null,null); DROP TABLE PAL_MLR_DATA_TBL; CREATE COLUMN TABLE PAL_MLR_DATA_TBL ( "ID" INT,"Y" DOUBLE,"X1" DOUBLE, "X2" DOUBLE);
88
PUBLIC
INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT
INTO INTO INTO INTO INTO INTO INTO INTO INTO INTO
PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL PAL_MLR_DATA_TBL
VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES
(0,0.5,0.13,0.33); (1,0.15,0.14,0.34); (2,0.25,0.15,0.36); (3,0.35,0.16,0.35); (4,0.45,0.17,0.37); (5,0.55,0.18,0.38); (6,0.65,0.19,0.39); (7,0.75,0.19,0.31); (8,0.85,0.11,0.32); (9,0.95,0.12,0.33);
DROP TABLE PAL_MLR_RESULTS_TBL; CREATE COLUMN TABLE PAL_MLR_RESULTS_TBL ("ID" INT,"Ai" DOUBLE); DROP TABLE PAL_MLR_FITTED_TBL; CREATE COLUMN TABLE PAL_MLR_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); DROP TABLE PAL_MLR_SIGNIFICANCE_TBL; CREATE COLUMN TABLE PAL_MLR_SIGNIFICANCE_TBL ("NAME" varchar(50),"VALUE" DOUBLE); DROP TABLE PAL_MLR_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_MLR_PMMLMODEL_TBL ("ID" INT, "PMMLMODEL" VARCHAR(5000)); CALL _SYS_AFL.palLR(PAL_MLR_DATA_TBL, "#PAL_CONTROL_TBL", PAL_MLR_RESULTS_TBL, PAL_MLR_FITTED_TBL, PAL_MLR_SIGNIFICANCE_TBL, PAL_MLR_PMMLMODEL_TBL) with overview; SELECT * FROM PAL_MLR_RESULTS_TBL; SELECT * FROM PAL_MLR_FITTED_TBL; SELECT * FROM PAL_MLR_SIGNIFICANCE_TBL; Expected Result PAL_MLR_RESULTS_TBL:
PAL_MLR_FITTED_TBL:
PAL_MLR_SIGNIFICANCE_TBL:
PUBLIC
89
PAL_MLR_PMMLMODEL_TBL:
FORECASTWITHLR
This function performs prediction with the linear regression result. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','FORECASTWITHLR', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<data input table>, <coefficient input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Predictive Data Column 1st column Other columns Coefficient 1st column 2nd column Column Data Type Integer or Varchar Integer or double Integer Integer, double, varchar, or CLOB Description ID Variable Xn ID (start from 0) Value Ai or PMML model. Varchar and CLOB types are only valid for PMML model. Table Type Name <Data INPUT table type> <Coefficient INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in in out
90
Parameter Table Name THREAD_NUMBER MODEL_FORMAT Data Type Integer Integer Description Number of threads Output Table Table Fitted Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_FMLR_PREDICT_T; CREATE TYPE PAL_FMLR_PREDICT_T AS TABLE( "ID" INT,"X1" DOUBLE, "X2" DOUBLE); DROP TYPE PAL_FMLR_COEFFICIENT_T; CREATE TYPE PAL_FMLR_COEFFICIENT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_FMLR_FITTED_T; CREATE TYPE PAL_FMLR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP table PAL_FMLR_PDATA_TBL; CREATE column table PAL_FMLR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_FMLR_PDATA_TBL values (1,'DM_PAL.PAL_FMLR_PREDICT_T','in'); insert into PAL_FMLR_PDATA_TBL values (2,'DM_PAL.PAL_FMLR_COEFFICIENT_T','in'); insert into PAL_FMLR_PDATA_TBL values (3,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_FMLR_PDATA_TBL values (4,'DM_PAL.PAL_FMLR_FITTED_T','out'); GRANT SELECT ON DM_PAL.PAL_FMLR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palForecastWithLR','AFLPAL','FORECASTWITHLR',PAL_FMLR_PDATA_T BL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); DROP TABLE PAL_FMLR_PREDICTDATA_TBL; CREATE COLUMN TABLE PAL_FMLR_PREDICTDATA_TBL ("ID" INT,"X1" DOUBLE, "X2" DOUBLE); INSERT INTO PAL_FMLR_PREDICTDATA_TBL VALUES (0,0.5,0.3); INSERT INTO PAL_FMLR_PREDICTDATA_TBL VALUES (1,4,0.4); INSERT INTO PAL_FMLR_PREDICTDATA_TBL VALUES (2,0,1.6); INSERT INTO PAL_FMLR_PREDICTDATA_TBL VALUES (3,0.3,0.45); INSERT INTO PAL_FMLR_PREDICTDATA_TBL VALUES (4,0.4,1.7); Column Data Type Integer or varchar Integer or double Description ID Value Yi 0 (default): coefficients in table 1: PMML format
PUBLIC
91
DROP TABLE PAL_FMLR_COEEFICIENT_TBL; CREATE COLUMN TABLE PAL_FMLR_COEEFICIENT_TBL ("ID" INT,"Ai" DOUBLE); INSERT INTO PAL_FMLR_COEEFICIENT_TBL VALUES (0,1.7120914258645001); INSERT INTO PAL_FMLR_COEEFICIENT_TBL VALUES (1,0.2652771198483208); INSERT INTO PAL_FMLR_COEEFICIENT_TBL VALUES (2,-3.471103742302148); DROP TABLE PAL_FMLR_FITTED_TBL; CREATE COLUMN TABLE PAL_FMLR_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); CALL _SYS_AFL.palForecastWithLR(PAL_FMLR_PREDICTDATA_TBL, PAL_FMLR_COEEFICIENT_TBL, "#PAL_CONTROL_TBL", PAL_FMLR_FITTED_TBL) with overview; SELECT * FROM PAL_FMLR_FITTED_TBL; Expected Result PAL_FMLR_FITTED_TBL:
3.2.9
Naive Bayes
Naive Bayes is a classification algorithm based on Bayes theorem. It estimates the class-conditional probability by assuming that the attributes are conditionally independent of one another. Given the class label y and a dependent feature vector x1 through xn, the conditional independence assumption can be formally stated as follows:
Using the naive independence assumption that P(xi|y, x1, ..., xi-1, xi+1, ..., xn) = P(xi|y) for all i, this relationship is simplified to
Since P(x1, ..., xn) is constant given the input, we can use the following classification rule:
We can use Maximum a posteriori (MAP) estimation to estimate P(y) and P(xi|y). The former is then the relative frequency of class y in the training set.
92
The different Naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of P(xi|y). Despite its simplicity, Naive Bayes works quite well in areas like document classification and spam filtering, and it only requires a small amount of training data to estimate the parameters necessary for classification. The Naive Bayes algorithm in PAL includes two functions: NBCTRAIN for generating training model; and NBCPREDICT for making prediction based on the training model.
Prerequisites
The input data is of any data type, but the last column cannot be of double type. The input data does not contain null value.
NBCTRAIN
This function reads input data and generates training model with the Naive Bayes algorithm. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','NBCTRAIN', <signature table>); The signature table should contain the following records: Index 1 2 3 Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> Direction in in out
Procedure Calling CALL <procedure name>(<training input table>, <parameter table>, <result output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Training / Historical Data Column Other columns Column Data Type Varchar, integer, or double Description Attribute columns Constraint Discrete value: integer or varchar
PUBLIC
93
Table
Column
Column Data Type
Description
Constraint Continuous value: double
Last column
Varchar or integer
Class column
Discrete value: integer or varchar
Parameter Table Name THREAD_NUMBER IS_SPLIT_MODEL Data Type Integer Integer Description Number of threads. Indicates whether to split the string of the model. 0: does not split the model Other value: splits the model. The maximum length of each unit is 5000.
LAPLACE
Double
Enables or disables Laplace smoothing. 0 (default): disables Laplace smoothing Positive value: enables Laplace smoothing for discrete values
Output Table Table Result Column 1st column 2nd column Column Data Type Integer Varchar Description ID Model saved as a JSON string. The table must be a column table. The maximum length is 5000. If the maximum length of the model is predicted to exceed 5000, set IS_SPLIT_MODEL to a value not equal to 0. Constraint
Example
94
PUBLIC
Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_NBCTRAIN_DATA_T; CREATE TYPE PAL_NBCTRAIN_DATA_T AS TABLE( "HomeOwner" VARCHAR (100), "MaritalStatus" VARCHAR (100), "AnnualIncome" DOUBLE, "DefaultedBorrower" VARCHAR (100) ); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "Name" VARCHAR (50), "intArgs" INTEGER, "doubleArgs" DOUBLE, "stringArgs" VARCHAR (100) ); DROP TYPE PAL_NBC_MODEL_T; CREATE TYPE PAL_NBC_MODEL_T AS TABLE("ID" INT, "JsonString" VARCHAR(5000)); DROP table PAL_NBCTRAIN_PDATA_TBL; CREATE column table PAL_NBCTRAIN_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); insert into PAL_NBCTRAIN_PDATA_TBL values (1,'DM_PAL.PAL_NBCTRAIN_DATA_T','in'); insert into PAL_NBCTRAIN_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_NBCTRAIN_PDATA_TBL values (3,'DM_PAL.PAL_NBC_MODEL_T','out'); GRANT SELECT ON DM_PAL.PAL_NBCTRAIN_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_NBCTRAIN','AFLPAL','NBCTRAIN',PAL_NBCTRAIN_PDATA_TBL); DROP TABLE PAL_NBCTRAIN_DATA_TBL; CREATE COLUMN TABLE PAL_NBCTRAIN_DATA_TBL ( "HomeOwner" VARCHAR(100), "MaritalStatus" VARCHAR(100), "AnnualIncome" DOUBLE, "DefaultedBorrower" VARCHAR(100) ); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('YES','Single',125,'NO'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('NO','Married',100,'NO'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('NO','Single',70,'NO'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('YES','Married',120,'NO'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('NO','Divorced',95,'YES'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('NO','Married',60,'NO'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('YES','Divorced',220,'NO'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('NO','Single',85,'YES'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('NO','Married',75,'NO'); INSERT INTO PAL_NBCTRAIN_DATA_TBL VALUES ('NO','Single',90,'YES'); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL( "Name" VARCHAR(50), "intArgs" INTEGER, "doubleArgs" DOUBLE, "stringArgs" VARCHAR(100) );
PUBLIC
95
INSERT INSERT INSERT INSERT
INTO INTO INTO INTO
#PAL_CONTROL_TBL #PAL_CONTROL_TBL #PAL_CONTROL_TBL #PAL_CONTROL_TBL
VALUES VALUES VALUES VALUES
('THREAD_NUMBER',2,null,null); ('IS_SPLIT_MODEL',0,null,null); ('LAPLACE', null,0.01,null); ('TRACE', 1,null,null);
DROP TABLE PAL_NBC_MODEL_TBL; CREATE COLUMN TABLE PAL_NBC_MODEL_TBL ("ID" INT, "JsonString" VARCHAR(5000)); CALL _SYS_AFL.PAL_NBCTRAIN(PAL_NBCTRAIN_DATA_TBL, "#PAL_CONTROL_TBL", PAL_NBC_MODEL_TBL) with overview; SELECT * FROM PAL_NBC_MODEL_TBL; Expected Result PAL_NBC_MODEL_TBL:
NBCPREDICT
This function uses the training model generated by NBCTRAIN to make predictive analysis. Procedure Generation
CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','NBCPREDICT', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Table Type Name <Data INPUT table type> <PARAMETER table type> <Model INPUT table type> <Result Output table type> Direction in in in out
Procedure Calling CALL <procedure name>(<predict input table>, <parameter table>, <model input table>, <result output table>) with overview;
The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature
96
Input Tables Table Predicted Data Column 1st column Other columns Predictive Model 1st column 2nd column Column Data Type Integer Integer, varchar, or double Integer Varchar Description ID Data to be classified (predicted) ID JSON string predictive model
Output Table Table Result Column 1st column 2nd column Column Data Type Integer Varchar Description ID Predictive result
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_NBCPREDICT_PREDATA_T; CREATE TYPE PAL_NBCPREDICT_PREDATA_T AS TABLE( "ID" INT, "HomeOwner" VARCHAR(100), "MaritalStatus" VARCHAR(100), "AnnualIncome" DOUBLE ); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "Name" VARCHAR (50), "intArgs" INTEGER, "doubleArgs" DOUBLE, "stringArgs" VARCHAR(100) ); DROP TYPE PAL_NBC_MODEL_T; CREATE TYPE PAL_NBC_MODEL_T AS TABLE("ID" INT, "JsonString" VARCHAR(5000));
PUBLIC
97
DROP TYPE PAL_NBCPREDICT_RESULT_T; CREATE TYPE PAL_NBCPREDICT_RESULT_T AS TABLE("ID" INTEGER, "Class" VARCHAR(100)); DROP table PAL_NBCPREDICT_PDATA_TBL; CREATE column table PAL_NBCPREDICT_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_NBCPREDICT_PDATA_TBL values (1,'DM_PAL.PAL_NBCPREDICT_PREDATA_T','in'); insert into PAL_NBCPREDICT_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_NBCPREDICT_PDATA_TBL values (3,'DM_PAL.PAL_NBC_MODEL_T','in'); insert into PAL_NBCPREDICT_PDATA_TBL values (4,'DM_PAL.PAL_NBCPREDICT_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_NBCPREDICT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_NBCPREDICT','AFLPAL','NBCPREDICT',PAL_NBCPREDICT_PDATA _TBL); DROP TABLE PAL_NBCPREDICT_PREDATA_TBL; CREATE COLUMN TABLE PAL_NBCPREDICT_PREDATA_TBL ( "ID" INT,"HomeOwner" VARCHAR(100), "MaritalStatus" VARCHAR(100), "AnnualIncome" DOUBLE ); INSERT INTO PAL_NBCPREDICT_PREDATA_TBL VALUES (0,'NO','Married',120); INSERT INTO PAL_NBCPREDICT_PREDATA_TBL VALUES (1,'YES','Married',180); INSERT INTO PAL_NBCPREDICT_PREDATA_TBL VALUES (2,'NO','Single',90); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL( "Name" VARCHAR(50), "intArgs" INTEGER, "doubleArgs" DOUBLE, "stringArgs" VARCHAR(100) ); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',1,null,null); DROP TABLE PAL_NBCPREDICT_RESULTS_TBL; CREATE COLUMN TABLE PAL_NBCPREDICT_RESULTS_TBL ("ID" INTEGER, "Class" VARCHAR(100)); CALL _SYS_AFL.PAL_NBCPREDICT(PAL_NBCPREDICT_PREDATA_TBL, "#PAL_CONTROL_TBL", PAL_NBC_MODEL_TBL, PAL_NBCPREDICT_RESULTS_TBL) with overview; SELECT * FROM PAL_NBCPREDICT_RESULTS_TBL; Expected Result PAL_NBCPREDICT_RESULTS_TBL:
3.2.10 Polynomial Regression

Polynomial regression is an approach to modeling the relationship between a scalar variable y and one or more variables denoted X. In polynomial regression, data is modeled using polynomial functions, and unknown model parameters are estimated from the data. Such models are called polynomial models.
98
In PAL, the implementation of exponential regression is to transform to linear regression and solve it: y = 0 + 1 x + 2 x2 + + n xn Where 0n are parameters that need to be calculated. Let x = x1, x2 = x2, , xn = xn, and then y = 0 + 1 x1 + 2 x2 + + n xn So, y and x1xn is a linear relationship and can be solved using the linear regression method. The implementation also supports calculating the F value and R^2 to determine statistical significance.
Prerequisites
POLYNOMIALREGRESSION
This is a polynomial regression function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL', 'POLYNOMIALREGRESSION', <signature table>); The signature table should contain the following records: Index 1 2 3 4 5 6 Procedure Calling CALL <procedure name> (<input table>, <parameter table>, <result output table>, <fitted output table>, <significance output table>, <PMML output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <Fitted OUTPUT table type> <Significance OUTPUT table type> <PMML OUTPUT table type> Direction in in out out out out
PUBLIC
99
Signature Input Table Table Data Column 1st column 2nd column 3rd column Column Data Type Integer or varchar Integer or double Integer or double Description ID Variable y Variable X
Parameter Table Name POLYNOMIAL_NUM Data Type Integer Description This is a mandatory parameter to create a polynomial of degree POLYNOMIAL_NUM model. Note: POLYNOMIAL_NUM replaces VARIABLE_NUM. THREAD_NUMBER ALG Integer Integer Number of threads. (Optional) Specifies decomposition method: 0 (default): Doolittle decomposition (LU) 2: Singular value decomposition (SVD) 0 (default): does not output adjusted R square 1: outputs adjusted R square 0 (default): does not export polynomial regression model in PMML. 1: exports polynomial regression model in PMML in single row. 2: exports polynomial regression model in PMML in several rows, each row containing a maximum of 5000 characters.
ADJUSTED_R2
Integer
PMML_EXPORT
Integer
Output Tables
100
Table Result
Column 1st column 2nd column
Column Data Type Integer Integer or double
Description ID Value Ai A0: the intercept A1: the beta coefficient for X1 A2: the beta coefficient for X2 ...
Constraint
Fitted Data
Integer or varchar Integer or double VARCHAR Double Integer CLOB or varchar
ID Value Yi Name Value ID Polynomial regression model in PMML format (R^2 / F)
Significance
PMML Result
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_PR_DATA_T; CREATE TYPE PAL_PR_DATA_T AS TABLE( "ID" INT,"Y" DOUBLE,"X1" DOUBLE); DROP TYPE PAL_PR_RESULT_T; CREATE TYPE PAL_PR_RESULT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_PR_FITTED_T; CREATE TYPE PAL_PR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP TYPE PAL_PR_SIGNIFICANCE_T; CREATE TYPE PAL_PR_SIGNIFICANCE_T AS TABLE("NAME" varchar(50),"VALUE" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_PR_PMMLMODEL_T; CREATE TYPE PAL_PR_PMMLMODEL_T AS TABLE("ID" INT,"Model" varchar(5000)); DROP table PAL_PR_PDATA_TBL; CREATE column table PAL_PR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_PR_PDATA_TBL values (1,'DM_PAL.PAL_PR_DATA_T','in'); insert into PAL_PR_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_PR_PDATA_TBL values (3,'DM_PAL.PAL_PR_RESULT_T','out');
PUBLIC
101
insert into PAL_PR_PDATA_TBL values (4,'DM_PAL.PAL_PR_FITTED_T','out'); insert into PAL_PR_PDATA_TBL values (5,'DM_PAL.PAL_PR_SIGNIFICANCE_T','out'); insert into PAL_PR_PDATA_TBL values (6,'DM_PAL.PAL_PR_PMMLMODEL_T','out'); GRANT SELECT ON DM_PAL.PAL_PR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palPolynomialR','AFLPAL','POLYNOMIALREGRESSION',PAL_PR_PDATA _TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('POLYNOMIAL_NUM',3,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PMML_EXPORT',2,null,null); DROP TABLE PAL_PR_DATA_TBL; CREATE COLUMN TABLE PAL_PR_DATA_TBL ( "ID" INT,"Y" DOUBLE,"X1" DOUBLE); INSERT INTO PAL_PR_DATA_TBL VALUES (0,5,1); INSERT INTO PAL_PR_DATA_TBL VALUES (1,20,2); INSERT INTO PAL_PR_DATA_TBL VALUES (2,43,3); INSERT INTO PAL_PR_DATA_TBL VALUES (3,89,4); INSERT INTO PAL_PR_DATA_TBL VALUES (4,166,5); INSERT INTO PAL_PR_DATA_TBL VALUES (5,247,6); INSERT INTO PAL_PR_DATA_TBL VALUES (6,403,7); DROP TABLE PAL_PR_RESULTS_TBL; CREATE COLUMN TABLE PAL_PR_RESULTS_TBL ("ID" INT,"Ai" DOUBLE); DROP TABLE PAL_PR_FITTED_TBL; CREATE COLUMN TABLE PAL_PR_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); DROP TABLE PAL_PR_SIGNIFICANCE_TBL; CREATE COLUMN TABLE PAL_PR_SIGNIFICANCE_TBL ("NAME" varchar(50),"VALUE" DOUBLE); DROP TABLE PAL_PR_MODEL_TBL; CREATE COLUMN TABLE PAL_PR_MODEL_TBL ("ID" INT, "PMMLMODEL" VARCHAR(5000)); CALL _SYS_AFL.palPolynomialR(PAL_PR_DATA_TBL, "#PAL_CONTROL_TBL", PAL_PR_RESULTS_TBL, PAL_PR_FITTED_TBL, PAL_PR_SIGNIFICANCE_TBL, PAL_PR_MODEL_TBL) with overview; SELECT * FROM PAL_PR_RESULTS_TBL; SELECT * FROM PAL_PR_FITTED_TBL; SELECT * FROM PAL_PR_SIGNIFICANCE_TBL; SELECT * FROM PAL_PR_MODEL_TBL; Expected Result PAL_PR_RESULTS_TBL:
PAL_PR_FITTED_TBL:
102
PUBLIC
PAL_PR_SIGNIFICANCE_TBL:
PAL_PR_MODEL_TBL:
FORECASTWITHPOLYNOMIALR
This function performs prediction with the polynomial regression result. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL', 'FORECASTWITHPOLYNOMIALR', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<data input table>, <coefficient input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Table Type Name <Data INPUT table type> <Coefficient INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in in out
PUBLIC
103
Input Tables Table Predictive Data Column 1st column 2nd column Coefficient 1st column 2nd column Column Data Type Integer or varchar Integer or double Integer Integer, double, varchar, or CLOB Description ID Variable X ID (start from 0) Value Ai or PMML model. Varchar and CLOB types are only valid for PMML model.
Output Table Table Fitted Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_FPR_PREDICT_T; CREATE TYPE PAL_FPR_PREDICT_T AS TABLE( "ID" INT,"X1" DOUBLE); DROP TYPE PAL_FPR_COEFFICIENT_T; CREATE TYPE PAL_FPR_COEFFICIENT_T AS TABLE("ID" INT,"Ai" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_FPR_FITTED_T; CREATE TYPE PAL_FPR_FITTED_T AS TABLE("ID" INT,"Fitted" DOUBLE); DROP table PAL_FPR_PDATA_TBL; CREATE column table PAL_FPR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" Column Data Type Integer or varchar Integer or double Description ID Value Yi
104
VARCHAR(100)); insert into PAL_FPR_PDATA_TBL insert into PAL_FPR_PDATA_TBL insert into PAL_FPR_PDATA_TBL insert into PAL_FPR_PDATA_TBL
values values values values
(1,'DM_PAL.PAL_FPR_PREDICT_T','in'); (2,'DM_PAL.PAL_FPR_COEFFICIENT_T','in'); (3,'DM_PAL.PAL_CONTROL_T','in'); (4,'DM_PAL.PAL_FPR_FITTED_T','out');
GRANT SELECT ON DM_PAL.PAL_FPR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palForecastWithPolynomialR','AFLPAL','FORECASTWITHPOLYNOMIALR ',PAL_FPR_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); DROP TABLE PAL_FPR_PREDICTDATA_TBL; CREATE COLUMN TABLE PAL_FPR_PREDICTDATA_TBL ( "ID" INT,"X1" DOUBLE); INSERT INTO PAL_FPR_PREDICTDATA_TBL VALUES (0,0.3); INSERT INTO PAL_FPR_PREDICTDATA_TBL VALUES (1,4.0); INSERT INTO PAL_FPR_PREDICTDATA_TBL VALUES (2,1.6); INSERT INTO PAL_FPR_PREDICTDATA_TBL VALUES (3,0.45); INSERT INTO PAL_FPR_PREDICTDATA_TBL VALUES (4,1.7); DROP TABLE PAL_FPR_COEEFICIENT_TBL; CREATE COLUMN TABLE PAL_FPR_COEEFICIENT_TBL ("ID" INT,"Ai" DOUBLE); INSERT INTO PAL_FPR_COEEFICIENT_TBL VALUES (0,4.0); INSERT INTO PAL_FPR_COEEFICIENT_TBL VALUES (1,3.0); INSERT INTO PAL_FPR_COEEFICIENT_TBL VALUES (2,2.0); INSERT INTO PAL_FPR_COEEFICIENT_TBL VALUES (3,1.0); DROP TABLE PAL_FPR_FITTED_TBL; CREATE COLUMN TABLE PAL_FPR_FITTED_TBL ("ID" INT,"Fitted" DOUBLE); CALL _SYS_AFL.palForecastWithPolynomialR(PAL_FPR_PREDICTDATA_TBL, PAL_FPR_COEEFICIENT_TBL, "#PAL_CONTROL_TBL", PAL_FPR_FITTED_TBL) with overview; SELECT * FROM PAL_FPR_FITTED_TBL; Expected Result PAL_FPR_FITTED_TBL:
3.3
Association Algorithms
This section describes the association algorithms that are provided by the Predictive Analysis Library.
PUBLIC
105
3.3.1
Apriori
Apriori is a classic predictive analysis algorithm for finding association rules used in association analysis. Association analysis uncovers the hidden patterns, correlations or casual structures among a set of items or objects. For example, association analysis enables you to understand what products and services customers tend to purchase at the same time. By analyzing the purchasing trends of your customers with association analysis, you can predict their future behavior. Apriori is designed to operate on databases containing transactions. As is common in association rule mining, given a set of items, the algorithm attempts to find subsets which are common to at least a minimum number of the item sets. Apriori uses a bottom up approach, where frequent subsets are extended one item at a time, a step known as candidate generation, and groups of candidates are tested against the data. The algorithm terminates when no further successful extensions are found. Apriori uses breadth-first search and a tree structure to count candidate item sets efficiently. It generates candidate item sets of length k from item sets of length k-1, and then prunes the candidates which have an infrequent sub pattern. The candidate set contains all frequent k-length item sets. After that, it scans the transaction database to determine frequent item sets among the candidates. The Apriori function in PAL uses vertical data format to store the transaction data in memory. The function can take varchar or integer transaction ID and item ID as input. It supports the output of confidence, support, and lift value, but does not limit the number of output rules. However, you can use SQL script to select the number of output rules, for example: SELECT TOP 2000 FROM RULE_RESULTS where lift > 0.5
Prerequisites
The input data does not contain null value. There are no duplicated items in each transaction.
APRIORIRULE
This function reads input transaction data and generates association rules by the Apriori algorithm. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','APRIORIRULE', <signature table>); The signature table should contain the following records: Index 1 2 3 Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> Direction in in out
106
Index 4 Procedure Calling
Table Type Name <PMML OUTPUT table type>
Direction out
CALL <procedure name>(<input table>, <parameter table>, <result output table>, <PMML output table>) WITH overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Dataset/ Historical Data Column 1st column Item column Column Data Type Integer or varchar Integer or varchar Description Transaction ID Item ID
Parameter Table Name MIN_SUPPORT MIN_CONFIDENCE THREAD_NUMBER MAXITEMLENGTH Data Type Double Double Integer Integer Description User-specified minimum support (actual value). User-specified minimum confidence (actual value). Number of threads. Total length of leading items and dependent items in the output. The default is 5. 0 (default): does not export Apriori model in PMML. 1: exports Apriori model in PMML in single row. 2: exports Apriori model in PMML in several rows, each row containing a maximum of 5000 characters.
PMML_EXPORT
Integer
Output Tables Table Result Column 1st column 2nd column 3rd column Column Data Type Varchar Varchar Double Description Leading items Dependent items Support value
PUBLIC
107
Table
Column 4th column 5th column
Column Data Type Double Double Integer CLOB or varchar
Description Confidence value Lift value ID Apriori model in PMML format
PMML Result
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_APRIORI_DATA_T; CREATE TYPE PAL_APRIORI_DATA_T AS TABLE( "CUSTOMER" INT, "ITEM" VARCHAR(20)); DROP TYPE PAL_APRIORI_RESULT_T ; CREATE TYPE PAL_APRIORI_RESULT_T AS TABLE( "PRERULE" VARCHAR(500), "POSTRULE" VARCHAR(500), "SUPPORT" DOUBLE, "CONFIDENCE" DOUBLE, "LIFT" DOUBLE); DROP TYPE PAL_APRIORI_PMMLMODEL_T; CREATE TYPE PAL_APRIORI_PMMLMODEL_T AS TABLE( "ID" INT, "PMMLMODEL" VARCHAR(5000)); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); DROP TABLE PAL_APRIORI_PDATA_TBL; CREATE COLUMN TABLE PAL_APRIORI_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_APRIORI_PDATA_TBL VALUES (1, 'DM_PAL.PAL_APRIORI_DATA_T', 'in'); INSERT INTO PAL_APRIORI_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_APRIORI_PDATA_TBL VALUES (3, 'DM_PAL.PAL_APRIORI_RESULT_T', 'out'); INSERT INTO PAL_APRIORI_PDATA_TBL VALUES (4, 'DM_PAL.PAL_APRIORI_PMMLMODEL_T', 'out'); GRANT SELECT ON DM_PAL.PAL_APRIORI_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_APRIORI_RULE', 'AFLPAL', 'APRIORIRULE', PAL_APRIORI_PDATA_TBL); DROP TABLE PAL_APRIORI_TRANS_TBL; CREATE COLUMN TABLE PAL_APRIORI_TRANS_TBL( "CUSTOMER" INT, "ITEM" VARCHAR(20) ); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (2, 'item2'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (2, 'item3'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (3, 'item1'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (3, 'item2'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (3, 'item4'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (4,'item1'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (4,'item3'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (5, 'item2'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (5, 'item3'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (6, 'item1'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (6, 'item3'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (0, 'item1'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (0, 'item2'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (0, 'item5'); INSERT INTO PAL_APRIORI_TRANS_TBL VALUES (1, 'item2');
108
PUBLIC
INSERT INSERT INSERT INSERT INSERT INSERT INSERT INSERT
INTO INTO INTO INTO INTO INTO INTO INTO
PAL_APRIORI_TRANS_TBL PAL_APRIORI_TRANS_TBL PAL_APRIORI_TRANS_TBL PAL_APRIORI_TRANS_TBL PAL_APRIORI_TRANS_TBL PAL_APRIORI_TRANS_TBL PAL_APRIORI_TRANS_TBL PAL_APRIORI_TRANS_TBL
VALUES VALUES VALUES VALUES VALUES VALUES VALUES VALUES
(1, (7, (7, (7, (7, (8, (8, (8,
'item4'); 'item1'); 'item2'); 'item3'); 'item5'); 'item1'); 'item2'); 'item3');
DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE,"STRINGARGS" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER', 2, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MIN_SUPPORT', null, 0.2, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MIN_CONFIDENCE', null, 0.4, null); DROP TABLE PAL_APRIORI_RESULT_TBL; CREATE COLUMN TABLE PAL_APRIORI_RESULT_TBL( "PRERULE" VARCHAR(500), "POSTRULE" VARCHAR(500), "SUPPORT" Double, "CONFIDENCE" Double, "LIFT" DOUBLE ); DROP TABLE PAL_APRIORI_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_APRIORI_PMMLMODEL_TBL( "ID" INT, "PMMLMODEL" VARCHAR(5000)); CALL _SYS_AFL.PAL_APRIORI_RULE(PAL_APRIORI_TRANS_TBL, "#PAL_CONTROL_TBL", PAL_APRIORI_RESULT_TBL, PAL_APRIORI_PMMLMODEL_TBL) WITH overview; SELECT * FROM PAL_APRIORI_RESULT_TBL; SELECT * FROM PAL_APRIORI_PMMLMODEL_TBL; Expected Result: PAL_APRIORI_RESULT_TBL:
PAL_APRIORI_PMMLMODEL_TBL:
PUBLIC
109
LITEAPRIORIRULE
This is a light association rule mining algorithm to realize the Apriori algorithm. It only calculates two large item sets. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','LITEAPRIORIRULE', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <PMML output table>) WITH overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Dataset/ Historical Data Column 1st column 2nd column Column Data Type Integer or varchar Integer or varchar Description Transaction ID Item ID Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <PMML OUTPUT table type> Direction in in out out
Parameter Table Name MIN_SUPPORT MIN_CONFIDENCE THREAD_NUMBER OPTIMIZATION_TYPE Data Type Double Double Integer Integer or double Description User-specified minimum support (actual value). User-specified minimum confidence (actual value). Number of threads. If you want to use the entire data, set the integer value to 0 (default).
110
PUBLIC
Name
Data Type
Description If you want to sample the source input data, specify a double value as the sampling percentage. The default is 0.5.
IS_RECALCULATE
Integer
If you sample the input data, this parameter controls whether to use the remaining data or not. 1 (default): uses the remaining data to update the support, confidence, and lift. 0: does not use the remaining data 0 (default): does not export liteApriori model in PMML. 1: exports liteApriori model in PMML in single row. 2: exports liteApriori model in PMML in several rows, each row containing a maximum of 5000 characters.
PMML_EXPORT
Integer
Output Tables Table Result Column 1st column 2nd column 3rd column 4th column 5th column PMML Result 1st column 2nd column Column Data Type Varchar Varchar Double Double Double Integer CLOB or varchar Description Leading items Dependent items Support value Confidence value Lift value ID liteApriori model in PMML format
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL;
PUBLIC
111
DROP TYPE PAL_LITEAPRIORI_DATA_T; CREATE TYPE PAL_LITEAPRIORI_DATA_T AS TABLE( "CUSTOMER" INT, "ITEM" VARCHAR(20) ); DROP TYPE PAL_LITEAPRIORI_RESULT_T; CREATE TYPE PAL_LITEAPRIORI_RESULT_T AS TABLE( "PRERULE" VARCHAR(500), "POSTRULE" VARCHAR(500), "SUPPORT" DOUBLE, "CONFIDENCE" DOUBLE, "LIFT" DOUBLE ); DROP TYPE PAL_LITEAPRIORI_PMMLMODEL_T; CREATE TYPE PAL_LITEAPRIORI_PMMLMODEL_T AS TABLE( "ID" INT, "PMMLMODEL" VARCHAR(5000)); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100) ); DROP TABLE PAL_LITEAPRIORI_PDATA_TBL; CREATE COLUMN TABLE PAL_LITEAPRIORI_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_LITEAPRIORI_PDATA_TBL VALUES (1, 'DM_PAL.PAL_LITEAPRIORI_DATA_T', 'in'); INSERT INTO PAL_LITEAPRIORI_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_LITEAPRIORI_PDATA_TBL VALUES (3, 'DM_PAL.PAL_LITEAPRIORI_RESULT_T', 'out'); INSERT INTO PAL_LITEAPRIORI_PDATA_TBL VALUES (4, 'DM_PAL.PAL_LITEAPRIORI_PMMLMODEL_T', 'out'); GRANT SELECT ON DM_PAL.PAL_LITEAPRIORI_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_LITE_APRIORI_RULE', 'AFLPAL', 'LITEAPRIORIRULE', PAL_LITEAPRIORI_PDATA_TBL); DROP TABLE PAL_LITEAPRIORI_TRANS_TBL; CREATE COLUMN TABLE PAL_LITEAPRIORI_TRANS_TBL( "CUSTOMER" INT, "ITEM" VARCHAR(20) ); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (2, 'item2'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (2, 'item3'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (3, 'item1'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (3, 'item2'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (3, 'item4'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (4,'item1'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (4,'item3'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (5, 'item2'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (5, 'item3'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (6, 'item1'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (6, 'item3'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (0, 'item1'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (0, 'item2'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (0, 'item5'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (1, 'item2'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (1, 'item4'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (7, 'item1'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (7, 'item2'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (7, 'item3'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (7, 'item5'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (8, 'item1'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (8, 'item2'); INSERT INTO PAL_LITEAPRIORI_TRANS_TBL VALUES (8, 'item3'); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER', 2, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MIN_SUPPORT', null, 0.3, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MIN_CONFIDENCE', null, 0.4, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('OPTIMIZATION_TYPE', 0, 0.7, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('IS_RECALCULATE', 1, null, null); DROP TABLE PAL_LITEAPRIORI_RESULT_TBL; CREATE COLUMN TABLE PAL_LITEAPRIORI_RESULT_TBL( "PRERULE" VARCHAR(500), "POSTRULE" VARCHAR(500), "SUPPORT" Double, "CONFIDENCE" Double, "LIFT" DOUBLE);
112
DROP TABLE PAL_LITEAPRIORI_PMMLMODEL_TBL; CREATE COLUMN TABLE PAL_LITEAPRIORI_PMMLMODEL_TBL( "ID" INT, "PMMLMODEL" VARCHAR(5000)); CALL _SYS_AFL.PAL_LITE_APRIORI_RULE(PAL_LITEAPRIORI_TRANS_TBL, "#PAL_CONTROL_TBL", PAL_LITEAPRIORI_RESULT_TBL, PAL_LITEAPRIORI_PMMLMODEL_TBL) WITH overview; SELECT * FROM PAL_LITEAPRIORI_RESULT_TBL; SELECT * FROM PAL_LITEAPRIORI_PMMLMODEL_TBL; Expected Result PAL_LITEAPRIORI_RESULT_TBL:
PAL_LITEAPRIORI_PMMLMODEL_TBL:
3.4
Time Series Algorithms
Financial market data or economic data usually comes with time stamps. Predicting the future values, such as stock value for tomorrow, is of great interest in many business scenarios. Quantity over time is called time series, and predicting the future value based on existing time series is also known as forecasting. In this release of PAL, three smoothing based time series models are implemented. These models can be used to smooth the existing time series and forecast. In the time series algorithms, let xt be the observed values for the t-th time period, and T be the total number of time periods.
3.4.1
Single Exponential Smoothing
Single Exponential Smoothing model is suitable to model the time series without trend and seasonality. In the model, the smoothed value is the weighted sum of previous smoothed value and previous observed value. Let St be the smoothed value for the t-th time period. Mathematically, S1 = x0 St = xt1 + (1a)St1 Where (0,1) is a user specified parameter. Forecast is made by: ST+1 = xT + (1a)ST
PUBLIC
113
It is worth nothing that when t T+2, the smoothed value St, that is, the forecast value, is always ST+1 (xt1 is not available and St1 is used instead).
Prerequisites
SINGLESMOOTH
This is a single exponential smoothing function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','SINGLESMOOTH', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column Column Data Type Integer Integer or double Description ID Raw data Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
Parameter Table
114
Name RAW_DATA_COL
Data Type Integer
Description Column number of the column that contains the raw data. Default value: 1
ALPHA
Double
Value of the smoothing constant alpha (0 < < 1). Default value: 0.1
FORECAST_NUM
Integer
Number of values to be forecast. When it is set to 1, the algorithm only forecasts one value. Default value: 0
STARTTIME
Integer
Start time of raw data sequence. Default value: 1
Output Table Table Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_SINGLESMOOTH_DATA_T; CREATE TYPE PAL_SINGLESMOOTH_DATA_T AS TABLE("ID" INT, "RAWDATA" DOUBLE); DROP TYPE PAL_SINGLESMOOTH_RESULT_T; CREATE TYPE PAL_SINGLESMOOTH_RESULT_T AS TABLE("TIME" INT, "OUTPUT" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE, "strArgs" VARCHAR(100)); DROP table PAL_SINGLESMOOTH_PDATA_TBL; CREATE column table PAL_SINGLESMOOTH_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_SINGLESMOOTH_PDATA_TBL values (1,'DM_PAL.PAL_SINGLESMOOTH_DATA_T','in'); insert into PAL_SINGLESMOOTH_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_SINGLESMOOTH_PDATA_TBL values (3,'DM_PAL.PAL_SINGLESMOOTH_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_SINGLESMOOTH_PDATA_TBL to SYSTEM; call Column Data Type Integer Integer or double Description ID Output result
PUBLIC
115
SYSTEM.afl_wrapper_generator('SINGLESMOOTH_TEST','AFLPAL','SINGLESMOOTH',PAL_SINGLESMOO TH_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE "intArgs" INT, "doubleArgs" DOUBLE, INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES #PAL_CONTROL_TBL ("Name" VARCHAR(100), "strArgs" VARCHAR(100)); ('RAW_DATA_COL',1,null,null); ('ALPHA',null,0.1,null); ('FORECAST_NUM',1,null,null); ('STARTTIME',2000,null,null);
DROP TABLE PAL_SINGLESMOOTH_DATA_TBL; CREATE COLUMN TABLE PAL_SINGLESMOOTH_DATA_TBL ("ID" INT, "RAWDATA" DOUBLE); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (0,200.0); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (1,135.0); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (2,195.0); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (3,197.5); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (4,310.0); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (5,175.0); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (6,155.0); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (7,130.0); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (8,220.0); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (9,277.5); INSERT INTO PAL_SINGLESMOOTH_DATA_TBL VALUES (10,235.0); DROP TABLE PAL_SINGLESMOOTH_RESULT_TBL; CREATE COLUMN TABLE PAL_SINGLESMOOTH_RESULT_TBL ("TIME" INT, "OUTPUT" DOUBLE); CALL _SYS_AFL.SINGLESMOOTH_TEST(PAL_SINGLESMOOTH_DATA_TBL, "#PAL_CONTROL_TBL", PAL_SINGLESMOOTH_RESULT_TBL) with overview; SELECT * FROM PAL_SINGLESMOOTH_RESULT_TBL; Expected Result PAL_SINGLESMOOTH_RESULT_TBL:
3.4.2
Double Exponential Smoothing
Double Exponential Smoothing model is suitable to model the time series with trend but without seasonality. In the model there are two kinds of smoothed quantities: smoothed signal and smoothed trend. Let St and bt be the smoothed value and smoothed trend for the (t+1)-th time period, respectively. The following rules are satisfied:
116
PUBLIC
S0 = x0 b0 = x1 x0 St = xt + (1 ) (St-1 + bt-1) bt = (St St1) + (1 ) bt-1 Where , (0,1) are two user specified parameters. The model can be understood as two coupled Single Exponential Smoothing models, and forecast can be made by the following equation: FT+m = ST + mbT
Note
F0 is not defined because there is no estimation for time 0. According to the definition, you can get F1 = S0 + b0 and so on.
Prerequisites
DOUBLESMOOTH
This is a double exponential smoothing function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','DOUBLESMOOTH', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
PUBLIC
117
Input Table Table Data Column 1st column 2nd column Parameter Table Name RAW_DATA_COL Data Type Integer Description Column number of the column that contains the raw data. Default value: 1 ALPHA Double Value of the smoothing constant alpha (0 < < 1). Default value: 0.1 BETA Double Value of the smoothing constant beta (0 < < 1). Default value: 0.1 FORECAST_NUM Integer Number of values to be forecast (num > 0). Default value: 0 STARTTIME Integer Start time of raw data sequence. Default value: 1 Output Table Table Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_DOUBLESMOOTH_DATA_T; CREATE TYPE PAL_DOUBLESMOOTH_DATA_T AS TABLE("ID" INT, "RAWDATA" DOUBLE); DROP TYPE PAL_DOUBLESMOOTH_RESULT_T; CREATE TYPE PAL_DOUBLESMOOTH_RESULT_T AS TABLE("TIME" INT, "OUTPUT" DOUBLE); DROP TYPE PAL_CONTROL_T; Column Data Type Integer Integer or double Description ID Output result Column Data Type Integer Integer or double Description ID Raw data
118
CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE, "strArgs" VARCHAR(100)); DROP table PAL_DOUBLESMOOTH_PDATA_TBL; CREATE column table PAL_DOUBLESMOOTH_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_DOUBLESMOOTH_PDATA_TBL values (1,'DM_PAL.PAL_DOUBLESMOOTH_DATA_T','in'); insert into PAL_DOUBLESMOOTH_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_DOUBLESMOOTH_PDATA_TBL values (3,'DM_PAL.PAL_DOUBLESMOOTH_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_DOUBLESMOOTH_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('DOUBLESMOOTH_TEST','AFLPAL','DOUBLESMOOTH',PAL_DOUBLESM OOTH_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE "intArgs" INT, "doubleArgs" DOUBLE, INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES #PAL_CONTROL_TBL ("Name" VARCHAR(100), "strArgs" VARCHAR(100)); ('RAW_DATA_COL',1,null,null); ('ALPHA',null,0.501,null); ('BETA',null,0.072,null); ('FORECAST_NUM',6,null,null); ('STARTTIME',2000,null,null);
DROP TABLE PAL_DOUBLESMOOTH_DATA_TBL; CREATE COLUMN TABLE PAL_DOUBLESMOOTH_DATA_TBL ("ID" INT, "RAWDATA" DOUBLE); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (0,143.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (1,152.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (2,161.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (3,139.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (4,137.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (5,174.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (6,142.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (7,141.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (8,162.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (9,180.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (10,164.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (11,171.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (12,206.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (13,193.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (14,207.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (15,218.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (16,229.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (17,225.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (18,204.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (19,227.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (20,223.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (21,242.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (22,239.0); INSERT INTO PAL_DOUBLESMOOTH_DATA_TBL VALUES (23,266.0); DROP TABLE PAL_DOUBLESMOOTH_RESULT_TBL; CREATE COLUMN TABLE PAL_DOUBLESMOOTH_RESULT_TBL ("TIME" INT, "OUTPUT" DOUBLE); CALL _SYS_AFL.DOUBLESMOOTH_TEST(PAL_DOUBLESMOOTH_DATA_TBL, "#PAL_CONTROL_TBL", PAL_DOUBLESMOOTH_RESULT_TBL) with overview; SELECT * FROM PAL_DOUBLESMOOTH_RESULT_TBL; Expected Result PAL_DOUBLESMOOTH_RESULT_TBL:
PUBLIC
119
3.4.3
Triple Exponential Smoothing
Triple exponential smoothing is used to handle the time series data containing a seasonal component. This method is based on three smoothing equations: Stationary Component, Trend, and Seasonal. Both Seasonal and Trend can be additive or multiplicative. In PAL, the algorithm is finished with multiplicative and triple exponential smoothing is given by the formula below:
120
PUBLIC
Where: X S B C F t Data smoothing factor. The range is 0 < <1. Trend smoothing factor. The range is 0 < < 1. Seasonal change smoothing factor. The range is 0 < <1. Observation Smoothed observation Trend factor Seasonal index The forecast at m periods ahead The index that denotes a time period
Note
, , and are the constants that must be estimated in such a way that the MSE of the error is minimized. The formula for the initial trend estimate BL-1 is:
Setting the initial estimates for the seasonal indices Ci for i = 0,1,...,L-1 is a bit more involved, then:
Where
Note
SL-1 is the average value of x in the L cycle of your data.
PUBLIC
121
Prerequisites
TRIPLESMOOTH
This is a triple exponential smoothing function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','TRIPLESMOOTH', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column Column Data Type Integer Integer or double Description ID Raw data Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
Parameter Table Name RAW_DATA_COL Data Type Integer Description Column number of the column that contains the raw data. Default value: 1
122
Name ALPHA
Data Type Double
Description Value of the smoothing constant alpha (0 < < 1). Default value: 0.1
BETA
Double
Value of the smoothing constant beta (0 < < 1). Default value: 0.1
GAMMA
Double
Value of the smoothing constant gamma ( 0 < < 1). Default value: 0.1
CYCLE
Integer
A cycle of length L (L > 1). For example, quarterly data cycle is 4, and monthly data cycle is 12. Default value: 2
FORECAST_NUM
Integer
Number of values to be forecast (num > 0). Default value: 0
STARTTIME
Integer
Start time of raw data sequence. Default value: 1
Output Table Table Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_TRIPLESMOOTH_DATA_T; CREATE TYPE PAL_TRIPLESMOOTH_DATA_T AS TABLE("ID" INT, "RAWDATA" DOUBLE); DROP TYPE PAL_TRIPLESMOOTH_RESULT_T; CREATE TYPE PAL_TRIPLESMOOTH_RESULT_T AS TABLE("TIME" INT, "OUTPUT" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE, "strArgs" VARCHAR(100)); DROP table PAL_TRIPLESMOOTH_PDATA_TBL; Column Data Type Integer Integer or double Description ID Output result
PUBLIC
123
CREATE column table PAL_TRIPLESMOOTH_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_TRIPLESMOOTH_PDATA_TBL values (1,'DM_PAL.PAL_TRIPLESMOOTH_DATA_T','in'); insert into PAL_TRIPLESMOOTH_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_TRIPLESMOOTH_PDATA_TBL values (3,'DM_PAL.PAL_TRIPLESMOOTH_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_TRIPLESMOOTH_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('TRIPLESMOOTH_TEST','AFLPAL','TRIPLESMOOTH',PAL_TRIPLESMOOT H_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE, "strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('RAW_DATA_COL',1,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('ALPHA',null,0.822,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('BETA',null,0.055,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('GAMMA',null,0.055,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CYCLE',4,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('STARTTIME',2000,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('FORECAST_NUM',6,null,null); DROP TABLE PAL_TRIPLESMOOTH_DATA_TBL; CREATE COLUMN TABLE PAL_TRIPLESMOOTH_DATA_TBL ("ID" INT, "RAWDATA" DOUBLE); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (0,362.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (1,385.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (2,432.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (3,341.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (4,382.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (5,409.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (6,498.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (7,387.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (8,473.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (9,513.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (10,582.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (11,474.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (12,544.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (13,582.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (14,681.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (15,557.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (16,628.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (17,707.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (18,773.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (19,592.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (20,627.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (21,725.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (22,854.0); INSERT INTO PAL_TRIPLESMOOTH_DATA_TBL VALUES (23,661.0); DROP TABLE PAL_TRIPLESMOOTH_RESULT_TBL; CREATE COLUMN TABLE PAL_TRIPLESMOOTH_RESULT_TBL ("TIME" INT, "OUTPUT" DOUBLE); CALL _SYS_AFL.TRIPLESMOOTH_TEST(PAL_TRIPLESMOOTH_DATA_TBL, "#PAL_CONTROL_TBL", PAL_TRIPLESMOOTH_RESULT_TBL) with overview; SELECT * FROM PAL_TRIPLESMOOTH_RESULT_TBL; Expected Result PAL_TRIPLESMOOTH_RESULT_TBL:
124
3.5
Preprocessing Algorithms
The records in business database are usually not directly ready for predictive analysis due to the following reasons: Some data come in large amount, which may exceed the capacity of an algorithm. Some data contains noisy observations which may hurt the accuracy of an algorithm. Some attributes are badly scaled, which can make an algorithm unstable.
To address the above challenges, PAL provides several convenient algorithms for data preprocessing.
PUBLIC
125
3.5.1
Binning
Binning data is a common requirement prior to running certain predictive algorithms. It generally reduces the complexity of the model, for example, the model in a decision tree. Binning methods replace a value by a "bin number" defined by all elements of its neighborhood, that is, the bin it belongs to. The ordered values are distributed into a number of bins. Because binning methods consult the neighborhood of values, they perform local smoothing.
Note
Binning can only be used on a table with only one attribute. Binning Methods There are four binning methods: Equal widths based on the number of bins Specify an integer to determine the number of equal width bins and calculate the range values by: BandWidth = (MaxValue - MinValue) / K Where MaxValue is the biggest value of every column, MinValue is the smallest value of every column, and K is the number of bins. For example, according to this rule: MinValue + BinWidth > Values in Bin 1 MinValue MinValue + 2 * BinWidth > Values in Bin 2 MinValue + BinWidth
Equal bin widths defined as a parameter Specify the bin width and calculate the start and end of bin intervals by: Start of bin intervals = Minimum data value 0.5 * Bin width End of bin intervals = Maximum data value + 0.5 * Bin width For example, assuming the data has a range from 6 to 38 and the bin width is 10: Start of bin intervals = 6 0.5 * 10 = 1 End of bin intervals = 38 + 0.5 * 10 = 43 Hence, the generated bins would be the following: Bin Bin 1 Bin 2 Bin 3 Bin 4 Bin 5 Value Range [1, 10) [10, 20) [20, 30) [30, 40) [40, 43]
Equal number of records per bin Assign an equal number of records to each bin. For example: 2 bins, each containing 50% of the cases (below the median / above the median) 4 bins, each containing 25% of the cases (grouped by the quartiles) 5 bins, each containing 20% of the cases (grouped by the quintiles) 10 bins, each containing 10% of the cases (grouped by the deciles)
126
PUBLIC
20 bins, each containing 5% of the cases (grouped by the vingtiles) 100 bins, each containing 1% of the cases (grouped by the percentiles)
A tie condition results when the values on either side of a cut point are identical. In this case we move the tied values up to the next bin. Mean / standard deviation bin boundaries The mean and standard deviation can be used to create bins which are above or below the mean. The rules are as follows: + and 1 standard deviation, so Bin 1 contains values less than 1 standard deviation from the mean Bin 2 contains values between 1 and +1 standard deviation from the mean Bin 3 contains values greater than +1 standard deviation from the mean + and 2 standard deviation, so Bin 1 contains values less than 2*standard deviation from the mean Bin 2 contains values less than 1 standard deviation from the mean Bin 3 contains values between 1 and +1 standard deviation from the mean Bin 4 contains values greater than +1 standard deviation from the mean Bin 5 contains values greater than +2*standard deviation from the mean + and 3 standard deviation, so Bin 1 contains values less than 3*standard deviation from the mean Bin 2 contains values less than 2*standard deviation from the mean Bin 3 contains values less than 1 standard deviation from the mean Bin 4 contains values between 1 and +1 standard deviation from the mean Bin 5 contains values greater than +1 standard deviation from the mean Bin 6 contains values greater than +2*standard deviation from the mean Bin 7 contains values greater than +3*standard deviation from the mean
Smoothing Methods There are three methods for smoothing: Smoothing by bin means: each value within a bin is replaced by the average of all the values belonging to the same bin. Smoothing by bin medians: each value in a bin is replaced by the median of all the values belonging to the same bin. Smoothing by bin boundaries: the minimum and maximum values in a given bin are identified as the bin boundaries. Each value in the bin is then replaced by its closest boundary value.
Note
When the value is equal to both sides, it will be replaced by the front boundary value.
Prerequisites
The input data does not contain null value. The data is numeric, not categorical.
PUBLIC
127
BINNING
This function preprocesses the data. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','BINNING', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column Column Data Type Integer or String Integer or double Description ID Variable temperature Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
Parameter Table Name BINNING_METHOD Data Type Integer Description Binning methods: 0: equal widths based on the number of bins 1: equal widths based on the bin width 2: equal number of records per bin 3: mean/ standard deviation bin boundaries
SMOOTH_METHOD
Integer
Smoothing methods: 0: smoothing by bin means
128
Name
Data Type
Description 1: smoothing by bin medians 2: smoothing by bin boundaries
BIN_NUMBER
Integer
Number of needed bins. Default value: 2
BIN_DISTANCE
Integer
Specifies the distance for binning. This is required only when you have set BINNING_METHOD to 1. Default value: 10
SD
Integer
Specifies the standard deviation method. This is required only when you have set BINNING_METHOD to 3. Examples: 1 S.D.; 2 S.D.; 3 S.D. Default value: 1
Output Table Table Result Column 1st column 2nd column 3rd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_BINNING_DATA_T ; CREATE TYPE PAL_BINNING_DATA_T AS TABLE("ID" INT, "TEMPERATURE" DOUBLE) ; DROP TYPE PAL_CONTROL_T ; CREATE TYPE PAL_CONTROL_T AS TABLE( "Name" VARCHAR (50),"intArgs" INTEGER,"doubleArgs" DOUBLE,"stringArgs" VARCHAR (100)); DROP TYPE PAL_BINNING_RESULT_T ; CREATE TYPE PAL_BINNING_RESULT_T AS TABLE("ID" INT, "BIN_NUMBER" DOUBLE); DROP table PAL_BINNING_PDATA_TBL; CREATE column table PAL_BINNING_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); INT, "PRE_RESULT" Column Data Type Integer or string Integer Integer or double Description ID Variable TYPE Variable PRE_RESULT
PUBLIC
129
insert into PAL_BINNING_PDATA_TBL values (1,'DM_PAL.PAL_BINNING_DATA_T','in'); insert into PAL_BINNING_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_BINNING_PDATA_TBL values (3,'DM_PAL.PAL_BINNING_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_BINNING_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('BINNING_TEST','AFLPAL','BINNING',PAL_BINNING_PDATA_TBL); DROP TABLE PAL_BINNING_DATA_TBL; CREATE COLUMN TABLE PAL_BINNING_DATA_TBL ("ID" INT, "TEMPERATURE" INSERT INTO PAL_BINNING_DATA_TBL VALUES (0, 6.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (1, 12.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (2, 13.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (3, 15.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (4, 10.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (5, 23.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (6, 24.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (7, 30.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (8, 32.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (9, 25.0) ; INSERT INTO PAL_BINNING_DATA_TBL VALUES (10, 38.0) ; DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE "intArgs" INTEGER, "doubleArgs" DOUBLE, "stringArgs" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES DOUBLE) ;
#PAL_CONTROL_TBL ( "Name" VARCHAR (50),
('BINNING_METHOD',1,null,null); ('SMOOTH_METHOD',0,null,null); ('BIN_NUMBER',4,null,null); ('BIN_DISTANCE',10,null,null); ('SD',1,null,null);
DROP TABLE PAL_BINNING_RESULT_TBL; CREATE COLUMN TABLE PAL_BINNING_RESULT_TBL ("ID" INT, "BIN_NUMBER" INT, "PRE_RESULT" DOUBLE) ; CALL _SYS_AFL.BINNING_TEST(PAL_BINNING_DATA_TBL, "#PAL_CONTROL_TBL", PAL_BINNING_RESULT_TBL) with overview; SELECT * FROM PAL_BINNING_RESULT_TBL; Expected Result PAL_BINNING_RESULT_TBL:
130
PUBLIC
3.5.2
Convert Category Type to Binary Vector
This function converts category type to binary vector with numerical columns. Assume that you have a Gender attribute which has two distinct values: Female and Male. You can convert it into: Gender Female Male Female Gender_1 1 0 1 Gender_2 0 1 0
Prerequisites
The input data must contain an ID column, and the ID column must be the first column. The other columns of the input table must be of the integer or string type. The input data does not contain any null value.
CONV2BINARYVECTOR
Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'CONV2BINARYVECTOR', <signature table>); The signature table should contain the following records. Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> Direction in in out
PUBLIC
131
Table Data
Column Data Type Integer or string Integer or string
Description ID Attribute data
Constraint This must be the first column.
Parameter Table Name THREAD_NUMBER OUT_PUT_COLUMNS Output Table Table Result Column 1st column Other columns Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_CONV2_IN_T; CREATE TYPE PAL_CONV2_IN_T AS TABLE( "ID" INTEGER, "REGION" VARCHAR(50), "SALESPERIOD" VARCHAR(50), "REVENUE" INTEGER, "CLASSLABEL" VARCHAR(50) ); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INT_ARGS" INTEGER, "DOUBLE_ARGS" DOUBLE, "STRING_ARGS" VARCHAR(100) ); DROP TYPE PAL_CONV2_OUT_T; CREATE TYPE PAL_CONV2_OUT_T AS TABLE( "ID" INTEGER, "V0" INTEGER, "V1" INTEGER, "V2" INTEGER, "V3" INTEGER, "V4" INTEGER, "V5" INTEGER, "V6" INTEGER, "V7" INTEGER, "V8" INTEGER, "V9" INTEGER, Column Data Type Integer or string Integer Description IDs of original tuples Binary vectors Data Type Integer Integer Description Number of threads. Number of output columns.
132
);
"V10" "V11" "V12" "V13"
INTEGER, INTEGER, INTEGER, INTEGER
DROP TABLE PAL_CONV2_PDATA_TBL; CREATE COLUMN TABLE PAL_CONV2_PDATA_TBL( "ID" INTEGER, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_CONV2_PDATA_TBL VALUES (1, 'DM_PAL.PAL_CONV2_IN_T', 'in'); INSERT INTO PAL_CONV2_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_CONV2_PDATA_TBL VALUES (3, 'DM_PAL.PAL_CONV2_OUT_T', 'out'); GRANT SELECT ON DM_PAL.PAL_CONV2_PDATA_TBL TO SYSTEM; CALL "SYSTEM".afl_wrapper_generator('PAL_CONV2BINARY_PROC', 'AFLPAL', 'CONV2BINARYVECTOR', PAL_CONV2_PDATA_TBL); DROP TABLE PAL_CONV2_IN_TBL; CREATE COLUMN TABLE PAL_CONV2_IN_TBL LIKE PAL_CONV2_IN_T; INSERT INTO PAL_CONV2_IN_TBL VALUES (1, 'South', 'Winter', 1, 'Good'); INSERT INTO PAL_CONV2_IN_TBL VALUES (2, 'North', 'Spring', 2, 'Average'); INSERT INTO PAL_CONV2_IN_TBL VALUES (3, 'West', 'Summer', 2, 'Poor'); INSERT INTO PAL_CONV2_IN_TBL VALUES (4, 'East', 'Autumn', 3, 'Poor'); INSERT INTO PAL_CONV2_IN_TBL VALUES (5, 'West', 'Spring', 3, 'Poor'); INSERT INTO PAL_CONV2_IN_TBL VALUES (6, 'East', 'Spring', 1, 'Good'); INSERT INTO PAL_CONV2_IN_TBL VALUES (7, 'South', 'Summer', 3, 'Poor'); INSERT INTO PAL_CONV2_IN_TBL VALUES (8, 'South', 'Spring', 3, 'Average'); INSERT INTO PAL_CONV2_IN_TBL VALUES (9, 'North', 'Winter', 2, 'Average'); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL LIKE PAL_CONTROL_T; INSERT INTO #PAL_CONTROL_TBL VALUES ('OUT_PUT_COLUMNS', 15, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER', 2, null, null); DROP TABLE PAL_CONV2_OUT_TBL; CREATE COLUMN TABLE PAL_CONV2_OUT_TBL LIKE PAL_CONV2_OUT_T; CALL "_SYS_AFL".PAL_CONV2BINARY_PROC(PAL_CONV2_IN_TBL, #PAL_CONTROL_TBL, PAL_CONV2_OUT_TBL) with OVERVIEW; SELECT * FROM PAL_CONV2_OUT_TBL; Expected Result PAL_CONV2_OUT_TBL:
PUBLIC
133
3.5.3
Inter-quartile Range Test
Given a series of numeric data, the inter-quartile range (IQR) is the difference between the third quartile (Q3) and the first quartile (Q1) of the data. IQR = Q3 Q1 Q1 is equal to 25th percentile and Q3 is equal to 75th percentile. The p-th percentile of a numeric vector is a number, which is greater than or equal to p% of all the values of this numeric vector. IQR Test is a method to test the outliers of a series of numeric data. The algorithm performs the following tasks: 1. 2. Calculates Q1, Q3, and IQR. Set upper and lower bounds as follows: Upper-bound = Q3 + 1.5 IQR Lower-bound = Q1 1.5 IQR Tests all the values of a numeric vector to determine if it is in the range. The value outside the range is marked as an outlier, meaning it does not pass the IQR test.
3.
Prerequisites
IQRTEST
This function performs the inter-quartile range test and outputs the test results. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','IQRTEST', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <IQR output table>, <test output table>) with overview; Table Type Name <INPUT table type> <PARAMETER table type> <IQR OUTPUT table type> <Test OUTPUT table type> Direction in in out out
134
The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column Column Data Type Integer or varchar Integer or double Description ID Data that needs to be tested
Parameter Table Name MULTIPLIER Data Type Double Description The multiplier used in the IQR test. The default is 1.5.
Output Tables Table IQR Values Column 1st column 2nd column Test Result 1st column 2nd column Column Data Type Double Double Integer Integer or double Description Q1 value Q3 value ID Test result: 0: a value is in the range 1: a value is out of range
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_IQR_DATA_T; CREATE TYPE PAL_IQR_DATA_T AS TABLE( "ID" VARCHAR(10),"VAL" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE, "stringArgs" VARCHAR (100)); DROP TYPE PAL_IQR_T;
PUBLIC
135
CREATE TYPE PAL_IQR_T AS TABLE("Q1" DOUBLE, "Q3" DOUBLE); DROP TYPE PAL_IQR_RESULT_T; CREATE TYPE PAL_IQR_RESULT_T AS TABLE("ID" VARCHAR(10), "TEST" INT); DROP table PAL_IQR_PDATA_TBL; CREATE column table PAL_IQR_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_IQR_PDATA_TBL values (1,'DM_PAL.PAL_IQR_DATA_T','in'); insert into PAL_IQR_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_IQR_PDATA_TBL values (3,'DM_PAL.PAL_IQR_T','out'); insert into PAL_IQR_PDATA_TBL values (4,'DM_PAL.PAL_IQR_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_IQR_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palIQR','AFLPAL','IQRTEST',PAL_IQR_PDATA_TBL); DROP TABLE PAL_IQR_TESTDT_TBL; CREATE COLUMN TABLE PAL_IQR_TESTDT_TBL("ID" VARCHAR(10),"VAL" DOUBLE); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P1', 10); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P2', 11); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P3', 10); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P4', 9); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P5', 10); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P6', 24); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P7', 11); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P8', 12); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P9', 10); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P10', 9); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P11', 1); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P12', 11); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P13', 12); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P14', 13); INSERT INTO PAL_IQR_TESTDT_TBL VALUES ('P15', 12); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE, "stringArgs" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('MULTIPLIER',null,1.5,null); DROP TABLE PAL_IQR_TBL; CREATE COLUMN TABLE PAL_IQR_TBL ("Q1" DOUBLE, "Q3" DOUBLE); DROP TABLE PAL_IQR_RESULTS_TBL; CREATE COLUMN TABLE PAL_IQR_RESULTS_TBL ("ID" VARCHAR(10), "TEST" INT); CALL _SYS_AFL.palIQR(PAL_IQR_TESTDT_TBL, "#PAL_CONTROL_TBL", PAL_IQR_TBL, PAL_IQR_RESULTS_TBL) with overview; SELECT * FROM PAL_IQR_TBL; SELECT * FROM PAL_IQR_RESULTS_TBL; Expected Result IQR value:
Test result:
136
PUBLIC
3.5.4
Sampling
In business scenarios the number of records in the database is usually quite large, and it is common to use a small portion of the records as representatives, so that a rough impression of the dataset can be given by analyzing sampling. This release of PAL provides eight sampling methods, including: First_N Middle_N Last_N Every_Nth SimpleRandom_WithReplacement SimpleRandom_WithoutReplacement Systematic Stratified
Prerequisites
PUBLIC
137
SAMPLING
This function takes samples from a population. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','SAMPLING', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Parameter Table Name SAMPLING_METHOD Data Type Integer Description Sampling method: 0 : First_N 1 : Middle_N 2 : Last_N 3 : Every_Nth 4: SimpleRandom_WithReplacem ent 5: SimpleRandom_WithoutReplac ement 6 : Systematic 7 : Stratified Column Columns Column Data Type Description Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
Integer, double, or varchar Any data users need
138
Name
Data Type
Description Note: For the random methods (method 4, 5, 6 in the above list), the system time is used for the seed.
SAMPLING_SIZE
Integer
Number of the samples. Use this parameter when PERCENTAGE is not set. Default value: 1
PERCENTAGE
Double
Percentage of the samples. Use this parameter when SAMPLING_SIZE is not set. Default value: 0.1 Note: If both SAMPLING_SIZE and PERCENTAGE are specified, PERCENTAGE takes precedence.
THREAD_NUMBER INTERVAL
Integer Integer
Number of threads The interval between two samples Note: This parameter is only required for the Every_Nth method. If this parameter is not specified, the SAMPLING_SIZE parameter will be used.
STRATA_NUM
Integer
The number of the sub-populations. Note: This parameter is only required for the stratified method. In this example a population with three strata is sampled.
STRATA1_COUNT STRATA2_COUNT STRATA3_COUNT
Integer Integer Integer
The needed numbers of the first strata. The needed numbers of the second strata. The needed numbers of the third strata.
Output Table Table Result Column Columns Column Data Type Description
Integer, double, or varchar The Output Table has the same structure as defined in the Input Table.
PUBLIC
139
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_SAMPLING_DATA_T ; CREATE TYPE PAL_SAMPLING_DATA_T AS TABLE("EMPNO" INT, "INCOME" DOUBLE) ; "GENDER" VARCHAR (50),
DROP TYPE PAL_CONTROL_T ; CREATE TYPE PAL_CONTROL_T AS TABLE( "Name" VARCHAR (50),"intArgs" INTEGER,"doubleArgs" DOUBLE,"stringArgs" VARCHAR (100)); DROP TYPE PAL_SAMPLING_RESULT_T ; CREATE TYPE PAL_SAMPLING_RESULT_T AS TABLE("RESULT_EMPNO" INT, "RESULT_GENDER" VARCHAR (50), "RESULT_INCOME" DOUBLE); DROP table PAL_SAMPLING_PDATA_TBL; CREATE column table PAL_SAMPLING_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_SAMPLING_PDATA_TBL values (1,'DM_PAL.PAL_SAMPLING_DATA_T','in'); insert into PAL_SAMPLING_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_SAMPLING_PDATA_TBL values (3,'DM_PAL.PAL_SAMPLING_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_SAMPLING_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('SAMPLING_TEST','AFLPAL','SAMPLING',PAL_SAMPLING_PDATA_TBL); DROP TABLE PAL_SAMPLING_DATA_TBL; CREATE COLUMN TABLE PAL_SAMPLING_DATA_TBL ("EMPNO" INT, "GENDER" VARCHAR (50), "INCOME" DOUBLE) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (1, 'male', 4000.5) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (2, 'male', 5000.7) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (3, 'female', 5100.8) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (4, 'male', 5400.9) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (5, 'female', 5500.2) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (6, 'male', 5540.4) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (7, 'male', 4500.9) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (8, 'female', 6000.8) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (9, 'male', 7120.8) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (10, 'female', 8120.9) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (11, 'female', 7453.9) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (12, 'male', 7643.8) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (13, 'male', 6754.3) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (14, 'male', 6759.9) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (15, 'male', 9876.5) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (16, 'female', 9873.2) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (17, 'male', 9889.9) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (18, 'male', 9910.4) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (19, 'male', 7809.3) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (20, 'female', 8705.7) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (21, 'male', 8756.0) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (22, 'female', 7843.2) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (23, 'male', 8576.9) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (24, 'male', 9560.9) ; INSERT INTO PAL_SAMPLING_DATA_TBL VALUES (25, 'female', 8794.9) ; DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ( "Name" VARCHAR (50), "intArgs" INTEGER, "doubleArgs" DOUBLE,
140
"stringArgs" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('SAMPLING_METHOD',0,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('SAMPLING_SIZE',8,null,null); --INSERT INTO #PAL_CONTROL_TBL VALUES ('PERCENTAGE',NULL,0.1,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('INTERVAL',5,null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('STRATA_NUM',3,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('STRATA1_COUNT',9,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('STRATA2_COUNT',9,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('STRATA3_COUNT',7,null,null); DROP TABLE PAL_SAMPLING_RESULT_TBL; CREATE COLUMN TABLE PAL_SAMPLING_RESULT_TBL ("RESULT_EMPNO" INT, "RESULT_GENDER" VARCHAR (50), "RESULT_INCOME" DOUBLE ) ; CALL _SYS_AFL.SAMPLING_TEST(PAL_SAMPLING_DATA_TBL, "#PAL_CONTROL_TBL", PAL_SAMPLING_RESULT_TBL) WITH OVERVIEW; SELECT * FROM PAL_SAMPLING_RESULT_TBL; Expected Result If method is 0 and SAMPLING_SIZE is 8:
If method is 1 and SAMPLING_SIZE is 8:
PUBLIC
141
If method is 3 and INTERVAL is 5:
142
If method is 0 and PERCENTAGE is 0.1:
PUBLIC
143
3.5.5
Scaling Range
In real world scenarios the collected continuous attributes are usually distributed within different ranges. It is a common practice to have the data well scaled so that data mining algorithms like neural networks, nearest neighbor classification and clustering can give more reliable results. This release of PAL provides three scaling range methods described below. In the following, Xip and Yip are the original value and transformed value of the i-th record and p-th attribute, respectively. 1. Min-Max Normalization Each transformed value is within the range [new_minA, new_maxA], where new_minA and new_maxA are usespecified parameters. Supposing that minA and maxA are the minimum and maximum values of attribute A, we get the following calculation formula: Yip = (Xip minA) (new_maxA new_minA) / (maxA minA) + new_minA Z-Score Normalization (or zero-mean normalization). PAL uses three z-score methods. Mean-Standard Deviation The transformed values have mean 0 and standard deviation 1. The transformation is made as follows:
2.
Where p and p are mean and standard deviations of the original values of the p-th attributes. Mean-Mean Absolute Deviation The transformation is made as follows:
Median-Median Absolute Deviation The transformation is made as follows:
144
PUBLIC
3.
Normalization by Decimal Scaling This method transforms the data by moving the decimal point of the values, so that the maximum absolute value for each attribute is less than or equal to 1. Mathematically, Yip = Xip 10Kp for each attribute p, where Kp is selected so that max(|Y1p|, |Y2p|, ..., |Ynp|) 1
Prerequisites
The input data does not contain null value. The data is numeric, not categorical.
SCALINGRANGE
This function normalizes the data. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','SCALINGRANGE', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
PUBLIC
145
Table Data
Column Data Type Integer or string Integer or double
Description ID Variable Xn
Parameter Table Name SCALING_METHOD Data Type Integer Description Scaling method: 0: Min-max normalization 1: Z-Score normalization 2: Decimal scaling normalization
Z-SCORE_METHOD
Integer
This parameter is used only when SCALING_METHOD is 1. 0: Mean-Standard deviation 1: Mean-Mean absolute deviation 2: Median-Median absolute deviation
THREAD_NUMBER NEW_MAX NEW_MIN
Integer Double or integer Double or integer
Number of threads The new maximum value of the minmax normalization method The new minimum value of min-max normalization method
Output Table Table Result Column 1st column Other columns Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_SCALING_DATA_T ; CREATE TYPE PAL_SCALING_DATA_T AS TABLE("ID" INT, "X1" DOUBLE, "X2" DOUBLE) ; DROP TYPE PAL_CONTROL_T ; CREATE TYPE PAL_CONTROL_T AS TABLE( "Name" VARCHAR (50),"intArgs" Column Data Type Integer or string Integer or double Description ID Variable Xn
146
INTEGER,"doubleArgs" DOUBLE,"stringArgs" VARCHAR (100)); DROP TYPE PAL_SCALING_RESULT_T ; CREATE TYPE PAL_SCALING_RESULT_T AS TABLE("ID" INT, "PRE_X1" DOUBLE); DROP table PAL_SCALING_PDATA_TBL; CREATE column table PAL_SCALING_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_SCALING_PDATA_TBL values (1,'DM_PAL.PAL_SCALING_DATA_T','in'); insert into PAL_SCALING_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_SCALING_PDATA_TBL values (3,'DM_PAL.PAL_SCALING_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_SCALING_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('SCALINGRANGE_TEST','AFLPAL','SCALINGRANGE',PAL_SCALING_PDA TA_TBL); DROP TABLE PAL_SCALING_DATA_TBL; CREATE COLUMN TABLE PAL_SCALING_DATA_TBL ("ID" INT, "X1" DOUBLE, "X2" DOUBLE) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (0, 6.0, 9.0) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (1, 12.1, 8.3) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (2, 13.5, 15.3) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (3, 15.4, 18.7) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (4, 10.2, 19.8) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (5, 23.3, 20.6) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (6, 24.4,24.3) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (7, 30.6, 25.3) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (8, 32.5, 27.6) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (9, 25.6, 28.5) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (10, 38.7, 29.4) ; INSERT INTO PAL_SCALING_DATA_TBL VALUES (11, 38.7, 29.4) ; DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE "intArgs" INTEGER, "doubleArgs" DOUBLE, "stringArgs" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES #PAL_CONTROL_TBL ( "Name" VARCHAR (50), DOUBLE, "PRE_X2"
('SCALING_METHOD',0,null,null); ('Z-SCORE_METHOD',0,null,null); ('THREAD_NUMBER',2,null,null); ('NEW_MAX',null,1.0,null); ('NEW_MIN',null,0.0,null);
DROP TABLE PAL_SCALING_RESULT_TBL; CREATE COLUMN TABLE PAL_SCALING_RESULT_TBL ("ID" INT, "PRE_X1" DOUBLE, "PRE_X2" DOUBLE) ; CALL _SYS_AFL.SCALINGRANGE_TEST(PAL_SCALING_DATA_TBL, "#PAL_CONTROL_TBL", PAL_SCALING_RESULT_TBL) with overview; SELECT * FROM PAL_SCALING_RESULT_TBL; Expected Result If Scaling method is 0:
PUBLIC
147
If Scaling method is 1 and z-score method is 0:
148
If Scaling method is 2:
PUBLIC
149
3.5.6
Variance Test
Variance Test is a method to identify the outliers of n number of numeric data {xi} where 0 < i < n+1, using the mean {} and the standard deviation of {} of n number of numeric data {xi}. Below is the algorithm for Variance Test: 1. Calculate the mean () and the standard deviation ():
2.
Set the upper and lower bounds as follows: Upper-bound = + multiplier * Lower-bound = multiplier * Where the multiplier is a double type coefficient provided by the user to test whether all the values of a numeric vector are in the range. If a value is outside the range, it means it doesn't pass the Variance Test. The value is marked as an outlier.
Prerequisites
150
VARIANCETEST
This is a variance test function. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','VARIANCETEST', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <test output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column Column Data Type Integer or varchar Integer or double Description ID Raw data Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <Test OUTPUT table type> Direction in in out out
Parameter Table Name SIGMA_NUM THREAD_NUMBER Data Type Double Integer Description Multiplier for sigma Number of threads
Output Tables Table Result Column 1st column 2nd column Test 1st column Column Data Type Double Double Integer or varchar Description Mean value Standard deviation ID Constraint
PUBLIC
151
Table
Column 2nd column
Column Data Type Integer
Description Result output
Constraint 0: in bounds 1: out of bounds
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_VT_DATA_T; CREATE TYPE PAL_VT_DATA_T AS TABLE("ID" INT,"X" DOUBLE); DROP TYPE PAL_VT_RESULT_T; CREATE TYPE PAL_VT_RESULT_T AS TABLE("MEAN" DOUBLE,"SD" DOUBLE); DROP TYPE PAL_VT_TEST_T; CREATE TYPE PAL_VT_TEST_T AS TABLE("ID" INT,"Test" INT); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP table PAL_VT_PDATA_TBL; CREATE column table PAL_VT_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_VT_PDATA_TBL values (1,'DM_PAL.PAL_VT_DATA_T','in'); insert into PAL_VT_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_VT_PDATA_TBL values (3,'DM_PAL.PAL_VT_RESULT_T','out'); insert into PAL_VT_PDATA_TBL values (4,'DM_PAL.PAL_VT_TEST_T','out'); GRANT SELECT ON DM_PAL.PAL_VT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('palVarianceTest','AFLPAL','VARIANCETEST',PAL_VT_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('SIGMA_NUM',null,3.0,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',8,null,null); DROP TABLE PAL_VT_DATA_TBL; CREATE COLUMN TABLE PAL_VT_DATA_TBL ( "ID" INT,"X" DOUBLE); INSERT INTO PAL_VT_DATA_TBL VALUES (0,25); INSERT INTO PAL_VT_DATA_TBL VALUES (1,20); INSERT INTO PAL_VT_DATA_TBL VALUES (2,23); INSERT INTO PAL_VT_DATA_TBL VALUES (3,29); INSERT INTO PAL_VT_DATA_TBL VALUES (4,26); INSERT INTO PAL_VT_DATA_TBL VALUES (5,23); INSERT INTO PAL_VT_DATA_TBL VALUES (6,22); INSERT INTO PAL_VT_DATA_TBL VALUES (7,21); INSERT INTO PAL_VT_DATA_TBL VALUES (8,22); INSERT INTO PAL_VT_DATA_TBL VALUES (9,25); INSERT INTO PAL_VT_DATA_TBL VALUES (10,26); INSERT INTO PAL_VT_DATA_TBL VALUES (11,28); INSERT INTO PAL_VT_DATA_TBL VALUES (12,29); INSERT INTO PAL_VT_DATA_TBL VALUES (13,27); INSERT INTO PAL_VT_DATA_TBL VALUES (14,26); INSERT INTO PAL_VT_DATA_TBL VALUES (15,23); INSERT INTO PAL_VT_DATA_TBL VALUES (16,22);
152
PUBLIC
INSERT INTO PAL_VT_DATA_TBL VALUES (17,23); INSERT INTO PAL_VT_DATA_TBL VALUES (18,25); INSERT INTO PAL_VT_DATA_TBL VALUES (19,103); DROP TABLE PAL_VT_RESULT_TBL; CREATE COLUMN TABLE PAL_VT_RESULT_TBL ("MEAN" DOUBLE,"SD" DOUBLE); DROP TABLE PAL_VT_TEST_TBL; CREATE COLUMN TABLE PAL_VT_TEST_TBL ("ID" INT,"Test" INT); CALL _SYS_AFL.palVarianceTest(PAL_VT_DATA_TBL, "#PAL_CONTROL_TBL", PAL_VT_RESULT_TBL, PAL_VT_TEST_TBL) with overview; SELECT * FROM PAL_VT_RESULT_TBL; SELECT * FROM PAL_VT_TEST_TBL; Expected Result PAL_VT_RESULT_TBL:
PAL_VT_TEST_TBL:
3.6
Social Network Analysis Algorithms
This section describes the algorithms provided by the PAL that are mainly used for social network analysis.
PUBLIC
153
3.6.1
Link Prediction
Predicting missing links is a common task in social network analysis. The Link Prediction algorithm in PAL provides four methods to compute the distance of any two nodes using existing links in a social network, and make prediction on the missing links based on these distances. Let x and y be two nodes in a social network, and be the set containing the neighbor nodes of x, the four methods to compute the distance of x and y are briefly described as follows. Common Neighbors The quantity is computed as the number of common neighbors of x and y:
Then, it is normalized by the total number of nodes. Jaccard's Coefficient The quantity is just a slight modification of the common neighbors:
Adamic/Adar The quantity is computed as the sum of inverse log degree over all the common neighbors:
Katz The quantity is computed as a weighted sum of the number of paths of length l connecting x and y:
Where is the user-specified parameter, and from node x and ends at node y.
is the number of paths with length l which starts
Prerequisites
The input data does not contain any null value.
154
LINKPREDICTION
Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','LINKPREDICTION', <signature table>); The signature table should contain the following records: Index 1 2 3 4 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <result output table>, <test output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column Column Data Type Integer or varchar Integer or varchar Description Node 1 in existing edge (Node1 Node2) Node 2 in existing edge (Node1 Node2) Table Type Name <INPUT table type> <PARAMETER table type> <Result OUTPUT table type> <Test OUTPUT table type> Direction in in out out
Parameter Table Name THREAD_NUMBER METHOD Data Type Integer Integer Description Number of threads Prediction method: BETA Double 1: Common Neighbors 2: Jaccard's Coefficient 3: Adamic/Adar 4: Katz
Parameter for the Katz method. BETA should be between 0 and 1. A smaller BETA is preferred.
Output Table
PUBLIC
155
Table Result
Column 1st column 2nd column 3rd column
Column Data Type Integer Integer Double
Description Node 1 in missing edge (Node1 Node2) Node 2 in missing edge (Node1 Node2) Prediction score
Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_LP_DATA_T; CREATE TYPE PAL_LP_DATA_T AS TABLE("NODE1" INTEGER, "NODE2" INTEGER); DROP TYPE PAL_LP_RESULT_T; CREATE TYPE PAL_LP_RESULT_T AS TABLE("NODE1" INTEGER, "NODE2" INTEGER, "SCORE" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR(100), "INT_ARGS" INTEGER, "DOUBLE_ARGS" DOUBLE, "STRING_ARGS" VARCHAR(100) ); DROP TABLE PAL_LP_PDATA_TBL; CREATE COLUMN TABLE PAL_LP_PDATA_TBL( "ID" INTEGER, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_LP_PDATA_TBL VALUES (1,'DM_PAL.PAL_LP_DATA_T','in'); INSERT INTO PAL_LP_PDATA_TBL VALUES (2,'DM_PAL.PAL_CONTROL_T','in'); INSERT INTO PAL_LP_PDATA_TBL VALUES (3,'DM_PAL.PAL_LP_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_LP_PDATA_TBL TO SYSTEM; CALL SYSTEM.afl_wrapper_generator('PAL_LINK_PREDICTION_PROC','AFLPAL','LINKPREDICTION', PAL_LP_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES INSERT INTO #PAL_CONTROL_TBL VALUES
#PAL_CONTROL_TBL LIKE PAL_CONTROL_T; ('THREAD_NUMBER', 1, null, null); ('METHOD', 1, null, null); ('BETA', null, 0.005, null);
DROP TABLE PAL_LP_DATA_TBL; CREATE COLUMN TABLE PAL_LP_DATA_TBL LIKE PAL_LP_DATA_T; INSERT INTO PAL_LP_DATA_TBL VALUES ('1','2'); INSERT INTO PAL_LP_DATA_TBL VALUES ('1','4'); INSERT INTO PAL_LP_DATA_TBL VALUES ('2','3'); INSERT INTO PAL_LP_DATA_TBL VALUES ('3','4'); INSERT INTO PAL_LP_DATA_TBL VALUES ('5','1'); INSERT INTO PAL_LP_DATA_TBL VALUES ('6','2'); INSERT INTO PAL_LP_DATA_TBL VALUES ('7','4');
156
INSERT INTO PAL_LP_DATA_TBL VALUES ('7','5'); INSERT INTO PAL_LP_DATA_TBL VALUES ('6','7'); INSERT INTO PAL_LP_DATA_TBL VALUES ('5','4'); DROP TABLE PAL_LP_RESULT_TBL; CREATE COLUMN TABLE PAL_LP_RESULT_TBL LIKE PAL_LP_RESULT_T; CALL _SYS_AFL.PAL_LINK_PREDICTION_PROC(PAL_LP_DATA_TBL, #PAL_CONTROL_TBL, PAL_LP_RESULT_TBL) with overview; SELECT * FROM PAL_LP_RESULT_TBL ORDER BY NODE1, NODE2; Expected Result PAL_LP_RESULT_TBL:
3.7
Miscellaneous
This section describes the ABC Analysis and Weighted Score Table algorithms that are provided by the Predictive Analysis Library.
3.7.1
ABC Analysis
This algorithm is used to classify objects (such as customers, employees, or products) based on a particular measure (such as revenue or profit). It suggests that inventories of an organization are not of equal value, thus can be grouped into three categories (A, B, and C) by their estimated importance. A items are very important for an organization. B items are of medium importance, that is, less important than A items and more important than C items. C items are of the least importance. An example of ABC classification is as follows: A items 20% of the items (customers) accounts for 70% of the revenue. B items 30% of the items (customers) accounts for 20% of the revenue.
PUBLIC
157
C items 50% of the items (customers) accounts for 10% of the revenue.
Prerequisites
Input data cannot contain null value. The item names in the Input table must be of string data type and be unique.
ABC
This function performs the ABC analysis algorithm. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>','AFLPAL','ABC', <signature table>); The signature table should contain the following records: Index 1 2 3 Procedure Calling CALL <procedure name>(<input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Table Table Data Column 1st column 2nd column Column Data Type VARCHAR Double Description Item name Value Table Type Name <INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in out
158
PUBLIC
Name PERCENT_A PERCENT_B PERCENT_C
Data Type Double Double Double
Description Interval for A class Interval for B class Interval for C class
Output Table Table Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_ABC_DATA_T; CREATE TYPE PAL_ABC_DATA_T AS TABLE("ITEM" VARCHAR(100),"VALUE" DOUBLE); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); DROP TYPE PAL_ABC_RESULT_T; CREATE TYPE PAL_ABC_RESULT_T AS TABLE("ABC" VARCHAR(10),"ITEM" VARCHAR(100)); DROP table PAL_ABC_PDATA_TBL; CREATE column table PAL_ABC_PDATA_TBL("ID" INT,"TYPENAME" VARCHAR(100),"DIRECTION" VARCHAR(100)); insert into PAL_ABC_PDATA_TBL values (1,'DM_PAL.PAL_ABC_DATA_T','in'); insert into PAL_ABC_PDATA_TBL values (2,'DM_PAL.PAL_CONTROL_T','in'); insert into PAL_ABC_PDATA_TBL values (3,'DM_PAL.PAL_ABC_RESULT_T','out'); GRANT SELECT ON DM_PAL.PAL_ABC_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_ABC','AFLPAL','ABC',PAL_ABC_PDATA_TBL); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ("Name" VARCHAR(100), "intArgs" INT, "doubleArgs" DOUBLE,"strArgs" VARCHAR(100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',1,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PERCENT_A',null,0.7,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PERCENT_B',null,0.2,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PERCENT_C',null,0.1,null); DROP TABLE PAL_ABC_DATA_TBL; CREATE COLUMN TABLE PAL_ABC_DATA_TBL("ITEM" VARCHAR(100),"VALUE" DOUBLE); INSERT INTO PAL_ABC_DATA_TBL VALUES ('item1', 15.4); INSERT INTO PAL_ABC_DATA_TBL VALUES ('item2', 200.4); INSERT INTO PAL_ABC_DATA_TBL VALUES ('item3', 280.4); INSERT INTO PAL_ABC_DATA_TBL VALUES ('item4', 100.9); INSERT INTO PAL_ABC_DATA_TBL VALUES ('item5', 40.4); Column Data Type VARCHAR VARCHAR Description ABC class Items
PUBLIC
159
INSERT INSERT INSERT INSERT INSERT
INTO INTO INTO INTO INTO
PAL_ABC_DATA_TBL PAL_ABC_DATA_TBL PAL_ABC_DATA_TBL PAL_ABC_DATA_TBL PAL_ABC_DATA_TBL
VALUES VALUES VALUES VALUES VALUES
('item6', 25.6); ('item7', 18.4); ('item8', 10.5); ('item9', 96.15); ('item10', 9.4);
DROP TABLE PAL_ABC_RESULT_TBL; CREATE COLUMN TABLE PAL_ABC_RESULT_TBL("ABC" VARCHAR(10),"ITEM" VARCHAR(100)); CALL _SYS_AFL.PAL_ABC(PAL_ABC_DATA_TBL, "#PAL_CONTROL_TBL", PAL_ABC_RESULT_TBL) with overview; select * from PAL_ABC_RESULT_TBL; Expected Result PAL_ABC_RESULT_TBL:
3.7.2
Weighted Score Table
A weighted score table is a method of evaluating alternatives when the importance of each criterion differs. In a weighted score table, each alternative is given a score for each criterion. These scores are then weighted by the importance of each criterion. All of an alternative's weighted scores are then added together to calculate its total weighted score. The alternative with the highest total score should be the best alternative. You can use weighted score tables to make predictions about future customer behavior. You first create a model based on historical data in the data mining application, and then apply the model to new data to make the prediction. The prediction, that is, the output of the model, is called a score. You can create a single score for your customers by taking into account different dimensions. A function defined by weighted score tables is a linear combination of functions of a variable. f(x1,,xn) = w1 f1(x1) + + wn fn(xn)
Prerequisites
The input data does not contain null value. The column of the Map Function table is sorted by the attribute order of the Input Data table.
160
WEIGHTEDTABLE
This function performs weighted table calculation. It is similar to the Volume Driver function in the Business Function Library (BFL). Volume Driver calculates only one column, but weightedTable calculates multiple columns at the same time. Procedure Generation CALL SYSTEM.AFL_WRAPPER_GENERATOR('<procedure name>', 'AFLPAL', 'WEIGHTEDTABLE', <signature table>); The signature table should contain the following records: Index 1 2 3 4 5 Procedure Calling CALL <procedure name>(<data input table>, <map input table>, <control input table>, <parameter table>, <output table>) with overview; The procedure name is the same as specified in the procedure generation. The input, parameter, and output tables must be of the types specified in the signature table. Signature Input Tables Table Target/ Input Data Column Columns Column Data Type Varchar, integer, or double Description Specifies which will be used to calculate the scores Constraint Discrete value: integer, string, double Continuous value: integer, double An ID column is mandatory. Its data type should be integer. Map Function Columns Varchar, integer, or double Creates the map function Every attribute (except ID) in the Input Data table maps to two columns in the Map Function table: Key Table Type Name <Data INPUT table type> <Map INPUT table type> <Control INPUT table type> <PARAMETER table type> <OUTPUT table type> Direction in in in in out
PUBLIC
161
Table
Column
Column Data Type
Description
Constraint column and Value column. The Value column must be of double type.
Control
Columns
Integer or double
This table has three columns. When the Input Data table has n attributes (except ID), the Weight Table will have n rows.
Output Table Table Result Column 1st column 2nd column Example Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role. SET SCHEMA DM_PAL; DROP TYPE PAL_WT_DATA_T; CREATE TYPE PAL_WT_DATA_T AS TABLE( "ID" INT, "GENDER" VARCHAR(10), "INCOME" INT, "HEIGHT" DOUBLE); DROP TYPE PAL_WT_MAP_FUN_T; CREATE TYPE PAL_WT_MAP_FUN_T AS TABLE( "GENDER" VARCHAR(10), "VAL1" DOUBLE, "INCOME" INT, "VAL2" DOUBLE, "HEIGHT" DOUBLE,"VAL3" DOUBLE); DROP TYPE PAL_WT_PARA_T; CREATE TYPE PAL_WT_PARA_T AS TABLE( "WEIGHT" DOUBLE, "ISDIS" INT, "ROWNUM" INT); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); DROP TYPE PAL_WT_RESULT_T; CREATE TYPE PAL_WT_RESULT_T AS TABLE( "ID" INT, "RESULT" DOUBLE); Column Data Type Integer Double Description ID Result value
162
PUBLIC
-- create procedure DROP TABLE PAL_WT_PDATA_TBL; CREATE COLUMN TABLE PAL_WT_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_WT_PDATA_TBL VALUES (1, 'DM_PAL.PAL_WT_DATA_T', 'in'); INSERT INTO PAL_WT_PDATA_TBL VALUES (2, 'DM_PAL.PAL_WT_MAP_FUN_T', 'in'); INSERT INTO PAL_WT_PDATA_TBL VALUES (3, 'DM_PAL.PAL_WT_PARA_T', 'in'); INSERT INTO PAL_WT_PDATA_TBL VALUES (4, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_WT_PDATA_TBL VALUES (5, 'DM_PAL.PAL_WT_RESULT_T', 'out'); GRANT SELECT ON DM_PAL.PAL_WT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_WEIGHTEDTABLE', 'AFLPAL', 'WEIGHTEDTABLE', PAL_WT_PDATA_TBL); DROP TABLE PAL_WT_DATA_TBL; CREATE COLUMN TABLE PAL_WT_DATA_TBL ( "ID" INT, "GENDER" VARCHAR(10), "INCOME" INT, "HEIGHT" DOUBLE); INSERT INTO PAL_WT_DATA_TBL VALUES (0,'male',5000,1.73); INSERT INTO PAL_WT_DATA_TBL VALUES (1,'male',9000,1.80); INSERT INTO PAL_WT_DATA_TBL VALUES (2,'female',6000,1.55); INSERT INTO PAL_WT_DATA_TBL VALUES (3,'male',15000,1.65); INSERT INTO PAL_WT_DATA_TBL VALUES (4,'female',2000,1.70); INSERT INTO PAL_WT_DATA_TBL VALUES (5,'female',12000,1.65); INSERT INTO PAL_WT_DATA_TBL VALUES (6,'male',1000,1.65); INSERT INTO PAL_WT_DATA_TBL VALUES (7,'male',8000,1.60); INSERT INTO PAL_WT_DATA_TBL VALUES (8,'female',5500,1.85); INSERT INTO PAL_WT_DATA_TBL VALUES (9,'female',9500,1.85); DROP TABLE PAL_WT_MAP_FUN_TBL; CREATE COLUMN TABLE PAL_WT_MAP_FUN_TBL ( "GENDER" VARCHAR(10), "VAL1" DOUBLE, "INCOME" INT, "VAL2" DOUBLE, "HEIGHT" DOUBLE, "VAL3" DOUBLE); INSERT INTO PAL_WT_MAP_FUN_TBL VALUES ('male',2.0, 0,0.0, 1.5,0.0); INSERT INTO PAL_WT_MAP_FUN_TBL VALUES ('female',1.5, 5500,1.0, 1.6,1.0); INSERT INTO PAL_WT_MAP_FUN_TBL VALUES (null,0.0, 9000,2.0, 1.71,2.0); INSERT INTO PAL_WT_MAP_FUN_TBL VALUES (null,0.0, 12000,3.0, 1.80,3.0); DROP TABLE PAL_WT_PARA_TBL; CREATE COLUMN TABLE PAL_WT_PARA_TBL ( "WEIGHT" DOUBLE, "ISDIS" INT, "ROWNUM" INT); INSERT INTO PAL_WT_PARA_TBL VALUES (0.5,1,2); INSERT INTO PAL_WT_PARA_TBL VALUES (2.0,-1,4); INSERT INTO PAL_WT_PARA_TBL VALUES (1.0,-1,4); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); DROP TABLE PAL_WT_RESULT_TBL; CREATE COLUMN TABLE PAL_WT_RESULT_TBL( "ID" INT, "RESULT" DOUBLE); CALL _SYS_AFL.PAL_WEIGHTEDTABLE(PAL_WT_DATA_TBL, PAL_WT_MAP_FUN_TBL, PAL_WT_PARA_TBL, "#PAL_CONTROL_TBL", PAL_WT_RESULT_TBL) with overview; SELECT * FROM PAL_WT_RESULT_TBL; Expected Result PAL_WT_RESULT_TBL:
PUBLIC
163
164
End-to-End Scenarios
Scenario
You want to predict segmentation/clustering of new customers for a supermarket. First use the K-means function in PAL to perform segmentation / clustering for existing customers in the supermarket. The output can then be used as the training data for the C4.5 Decision Tree function to predict new customers segmentation/clustering.
Technology Background
K-means clustering is a method of cluster analysis whereby the algorithm partitions N observations or records into K clusters, in which each observation belongs to the cluster with the nearest center. It is one of the most commonly used algorithms in clustering method. Decision trees are powerful and popular tools for classification and prediction. Decision tree learning, used in statistics, data mining, and machine learning uses a decision tree as a predictive model which maps the observations about an item to the conclusions about the item's target value.
Implementation Steps
Assume that: DM_PAL is a schema belonging to USER1, who has been granted the privilege of executing SYSTEM.afl_wrapper_generator; and USER1 has been assigned the AFL__SYS_AFL_AFLPAL_EXECUTE role.
Step 1 Input customer data and use the K-means function to partition the data set into K clusters. In this example, nine rows of data will be input. K equals 3, which means the customers will be partitioned into three levels. SET SCHEMA DM_PAL; DROP TYPE PAL_KMEANS_RESASSIGN_T; CREATE TYPE PAL_KMEANS_RESASSIGN_T AS TABLE( "ID" INT, "CENTER_ASSIGN" INT, "DISTANCE" DOUBLE); DROP TYPE PAL_KMEANS_DATA_T; CREATE TYPE PAL_KMEANS_DATA_T AS TABLE( "ID" INT, "AGE" DOUBLE, "INCOME" DOUBLE, primary key("ID")); DROP TYPE PAL_KMEANS_CENTERS_T; CREATE TYPE PAL_KMEANS_CENTERS_T AS TABLE( "CENTER_ID" INT, "V000" DOUBLE, "V001" DOUBLE );
SAP HANA Predictive Analysis Library (PAL) End-to-End Scenarios
PUBLIC
165
DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100) ); -- create kmeans procedure DROP TABLE PAL_KMEANS_PDATA_TBL; CREATE COLUMN TABLE PAL_KMEANS_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (1, INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (2, INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (3, INSERT INTO PAL_KMEANS_PDATA_TBL VALUES (4,
'DM_PAL.PAL_KMEANS_DATA_T', 'in'); 'DM_PAL.PAL_CONTROL_T', 'in'); 'DM_PAL.PAL_KMEANS_RESASSIGN_T', 'out'); 'DM_PAL.PAL_KMEANS_CENTERS_T', 'out');
GRANT SELECT ON DM_PAL.PAL_KMEANS_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_KMEANS', 'AFLPAL', 'KMEANS', PAL_KMEANS_PDATA_TBL); DROP TABLE PAL_KMEANS_DATA_TBL; CREATE COLUMN TABLE PAL_KMEANS_DATA_TBL( "ID" INT, "AGE" DOUBLE, "INCOME" DOUBLE ); INSERT INTO PAL_KMEANS_DATA_TBL VALUES (0 INSERT INTO PAL_KMEANS_DATA_TBL VALUES (1 INSERT INTO PAL_KMEANS_DATA_TBL VALUES (2 INSERT INTO PAL_KMEANS_DATA_TBL VALUES (3 INSERT INTO PAL_KMEANS_DATA_TBL VALUES (4 INSERT INTO PAL_KMEANS_DATA_TBL VALUES (5 INSERT INTO PAL_KMEANS_DATA_TBL VALUES (6 INSERT INTO PAL_KMEANS_DATA_TBL VALUES (7 INSERT INTO PAL_KMEANS_DATA_TBL VALUES (8
, , , , , , , , ,
20, 21, 22, 30, 31, 32, 40, 41, 42,
100000); 101000); 102000); 200000); 201000); 202000); 400000); 401000); 402000);
DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100) ); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('GROUP_NUMBER',3,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('INIT_TYPE',4,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('DISTANCE_LEVEL',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('MAX_ITERATION',100,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('EXIT_THRESHOLD',null,0.000001,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('NORMALIZATION',0,null,null); --clean kmeans result DROP TABLE PAL_KMEANS_RESASSIGN_TBL ; CREATE COLUMN TABLE PAL_KMEANS_RESASSIGN_TBL ( "ID" INT, "CENTER_ASSIGN" INT, "DISTANCE" DOUBLE ); DROP TABLE PAL_KMEANS_CENTERS_TBL; CREATE COLUMN TABLE PAL_KMEANS_CENTERS_TBL( "CENTER_ID" INT, "V000" DOUBLE, "V001" DOUBLE); CALL _SYS_AFL.PAL_KMEANS(PAL_KMEANS_DATA_TBL, "#PAL_CONTROL_TBL", PAL_KMEANS_RESASSIGN_TBL , PAL_KMEANS_CENTERS_TBL) with overview; SELECT * FROM PAL_KMEANS_CENTERS_TBL; SELECT * FROM PAL_KMEANS_RESASSIGN_TBL ;
166
DROP TABLE PAL_KMEANS_RESULT_TBL; CREATE COLUMN TABLE PAL_KMEANS_RESULT_TBL( "AGE" DOUBLE, "INCOME" DOUBLE, "LEVEL" INT); TRUNCATE TABLE PAL_KMEANS_RESULT_TBL; INSERT INTO PAL_KMEANS_RESULT_TBL(SELECT PAL_KMEANS_DATA_TBL.AGE,PAL_KMEANS_DATA_TBL.INCOME,PAL_KMEANS_RESASSIGN_TBL.CENTER_A SSIGN FROM PAL_KMEANS_RESASSIGN_TBL INNER JOIN PAL_KMEANS_DATA_TBL ON PAL_KMEANS_RESASSIGN_TBL.ID = PAL_KMEANS_DATA_TBL.ID); SELECT * FROM PAL_KMEANS_RESULT_TBL; The result should show the following in PAL_KMEANS_RESULT_TBL.
Step 2 Use the above output as the training data of C4.5 Decision Tree. The C4.5 Decision Tree function will generate a tree model which maps the observations about an item to the conclusions about the item's target value. SET SCHEMA DM_PAL; DROP TYPE PAL_CDT_DATA_T; CREATE TYPE PAL_CDT_DATA_T AS TABLE( "AGE" DOUBLE, "INCOME" DOUBLE, "LEVEL" INT); DROP TYPE PAL_CDT_JSONMODEL_T; CREATE TYPE PAL_CDT_JSONMODEL_T AS TABLE( "ID" INT, "JSONMODEL" VARCHAR(5000)); DROP TYPE PAL_CDT_PMMLMODEL_T ; CREATE TYPE PAL_CDT_PMMLMODEL_T AS TABLE( "ID" INT, "PMMLMODEL" VARCHAR(5000) ); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR(100) ); --create procedure DROP TABLE PAL_CDT_PDATA_TBL; CREATE COLUMN TABLE PAL_CDT_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_CDT_PDATA_TBL VALUES (1, 'DM_PAL.PAL_CDT_DATA_T', 'in'); INSERT INTO PAL_CDT_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_CDT_PDATA_TBL VALUES (3, 'DM_PAL.PAL_CDT_JSONMODEL_T', 'out'); INSERT INTO PAL_CDT_PDATA_TBL VALUES (4, 'DM_PAL.PAL_CDT_PMMLMODEL_T', 'out'); GRANT SELECT ON DM_PAL.PAL_CDT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_CREATEDT', 'AFLPAL', 'CREATEDT', PAL_CDT_PDATA_TBL); DROP TABLE PAL_CDT_TRAINING_TBL ; CREATE COLUMN TABLE PAL_CDT_TRAINING_TBL ( "REGION" VARCHAR(50), "SALESPERIOD" VARCHAR(50), "REVENUE" Double, "CLASSLABEL" VARCHAR(50)
PUBLIC
167
); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100) ); INSERT INTO #PAL_CONTROL_TBL VALUES ('PERCENTAGE',null,1.0,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('IS_SPLIT_MODEL',1,null,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('PMML_EXPORT', 2, null, null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CONTINUOUS_COL',1,102001,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CONTINUOUS_COL',1,202001,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CONTINUOUS_COL',0,23,null); INSERT INTO #PAL_CONTROL_TBL VALUES ('CONTINUOUS_COL',0,12,null); DROP TABLE PAL_CDT_JSONMODEL_TBL ; CREATE COLUMN TABLE PAL_CDT_JSONMODEL_TBL ( "ID" INT, "JSONMODEL" VARCHAR(5000) ); DROP TABLE PAL_CDT_PMMLMODEL_TBL ; CREATE COLUMN TABLE PAL_CDT_PMMLMODEL_TBL ( "ID" INT, "PMMLMODEL" VARCHAR(5000) ); CALL _SYS_AFL.PAL_CREATEDT(PAL_KMEANS_RESULT_TBL, "#PAL_CONTROL_TBL", PAL_CDT_JSONMODEL_TBL, PAL_CDT_PMMLMODEL_TBL) with overview; SELECT * FROM PAL_CDT_JSONMODEL_TBL ; SELECT * FROM PAL_CDT_PMMLMODEL_TBL ; Step 3 Use the above tree model to map each new customer to the corresponding level he or she belongs to. SET SCHEMA DM_PAL; DROP TYPE PAL_PCDT_DATA_T; CREATE TYPE PAL_PCDT_DATA_T AS TABLE( "ID" INT, "AGE" DOUBLE, "INCOME" DOUBLE); DROP TYPE PAL_PCDT_JSONMODEL_T; CREATE TYPE PAL_PCDT_JSONMODEL_T AS TABLE( "ID" INT, "JSONMODEL" VARCHAR(5000)); DROP TYPE PAL_CONTROL_T; CREATE TYPE PAL_CONTROL_T AS TABLE( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); DROP TYPE PAL_PCDT_RESULT_T; CREATE TYPE PAL_PCDT_RESULT_T AS TABLE( "ID" INT, "CLASSLABEL" VARCHAR(50)); -- create procedure DROP TABLE PAL_PCDT_PDATA_TBL; CREATE COLUMN TABLE PAL_PCDT_PDATA_TBL( "ID" INT, "TYPENAME" VARCHAR(100), "DIRECTION" VARCHAR(100) ); INSERT INTO PAL_PCDT_PDATA_TBL VALUES (1, 'DM_PAL.PAL_PCDT_DATA_T', 'in'); INSERT INTO PAL_PCDT_PDATA_TBL VALUES (2, 'DM_PAL.PAL_CONTROL_T', 'in'); INSERT INTO PAL_PCDT_PDATA_TBL VALUES (3, 'DM_PAL.PAL_PCDT_JSONMODEL_T', 'in'); INSERT INTO PAL_PCDT_PDATA_TBL VALUES (4, 'DM_PAL.PAL_PCDT_RESULT_T', 'out'); GRANT SELECT ON DM_PAL.PAL_PCDT_PDATA_TBL to SYSTEM; call SYSTEM.afl_wrapper_generator('PAL_PREDICTWITHDT', 'AFLPAL', 'PREDICTWITHDT', PAL_PCDT_PDATA_TBL); DROP TABLE PAL_PCDT_DATA_TBL; CREATE COLUMN TABLE PAL_PCDT_DATA_TBL ( "ID" INT, "AGE" DOUBLE, "INCOME" DOUBLE); INSERT INTO PAL_PCDT_DATA_TBL VALUES (10 ,20, 100003); INSERT INTO PAL_PCDT_DATA_TBL VALUES (11 ,30, 200003); INSERT INTO PAL_PCDT_DATA_TBL VALUES (12 ,40, 400003); DROP TABLE #PAL_CONTROL_TBL; CREATE LOCAL TEMPORARY COLUMN TABLE #PAL_CONTROL_TBL ( "NAME" VARCHAR (50), "INTARGS" INTEGER, "DOUBLEARGS" DOUBLE, "STRINGARGS" VARCHAR (100)); INSERT INTO #PAL_CONTROL_TBL VALUES ('THREAD_NUMBER',2,null,null);
168
PUBLIC
DROP TABLE PAL_PCDT_RESULT_TBL; CREATE COLUMN TABLE PAL_PCDT_RESULT_TBL( "ID" INT, "CLASSLABEL" VARCHAR(50)); CALL _SYS_AFL.PAL_PREDICTWITHDT(PAL_PCDT_DATA_TBL, #PAL_CONTROL_TBL, PAL_CDT_JSONMODEL_TBL,PAL_PCDT_RESULT_TBL) with overview; SELECT * FROM PAL_PCDT_RESULT_TBL; The expected prediction result is as follows:
PUBLIC
169
Best Practices
Create an SQL view for the input table if the table structure does not meet what is specified in this guide. Avoid null values in the input data. You can replace the null values with the default values via an SQL statement (SQL view or SQL update) because PAL functions cannot infer the default values. Create the parameter table as a local temporary table to avoid table name conflicts. If you do not use PMML export, you do not need to create a PMML output table to store the result. Just set the PMML_EXPORT parameter to 0 and pass ? or null to the function. When using the KMEANS function, different INIT_TYPE and NORMALIZATION settings may produce different results. You may need to try a few combinations of these two parameters to get the best result. When using the APRIORIRULE function, in some circumstances the rules set can be huge. To avoid an extra long runtime, you can set the MAXITEMLENGTH parameter to a smaller number, such as 2 or 3.
170
PUBLIC
SAP HANA Predictive Analysis Library (PAL) Best Practices
www.sap.com/contactsap
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see http://www.sap.com/corporate-en/legal/copyright/ index.epx for additional trademark information and notices.

SAP HANA Predictive Analysis Library PAL en

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SAP HANA Predictive Analysis Library PAL en

Uploaded by

Copyright:

Available Formats

PUBLIC

SAP HANA Platform SPS 06 Document Version: 1.0 - 26-06-2013

SAP HANA Predictive Analysis Library (PAL)

Association Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105 Time Series Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Preprocessing Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125

PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.

SAP HANA Predictive Analysis Library (PAL) Table of Contents

3.5.6 3.6 3.7 3.6.1 3.7.1 3.7.2 4 5

Social Network Analysis Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Miscellaneous. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

End-to-End Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

SAP HANA Predictive Analysis Library (PAL) Table of Contents

SAP HANA Predictive Analysis Library (PAL) What is PAL?

Getting Started with PAL

Application Function Library (AFL)

User and Schema

Checking PAL Installation

PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.

How to Call PAL Functions

Step 1 Create the AFL_WRAPPER_GENERATOR and AFL_WRAPPER_ERASER Procedures

Step 2 Generate a PAL Procedure

Step 3 Call a PAL Procedure

PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.

Apriori [page 106]

Single Exponential Smoothing [page SINGLESMOOTH 113]

PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.

SAP HANA Predictive Analysis Library (PAL) PAL Functions

PAL Algorithm Double Exponential Smoothing [page 116]

Built-in Function Name DOUBLESMOOTH

SAP HANA Predictive Analysis Library (PAL) PAL Functions

SAP HANA Predictive Analysis Library (PAL) PAL Functions

Data Type Integer

Number of threads. Threshold (actual value) for exiting the iterations.

SAP HANA Predictive Analysis Library (PAL) PAL Functions

Column Other columns

Column Data Type Integer or double

Description Coordinates of outliers

PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.

SAP HANA Predictive Analysis Library (PAL) PAL Functions

SAP HANA Predictive Analysis Library (PAL) PAL Functions

PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.

SAP HANA Predictive Analysis Library (PAL) PAL Functions

Description 4: Chebyshev 5: Standardized Euclidean 6: Cosine

MINKOW_P Output Table Table Result Column

Specifies the Minkowski method.

Column Data Type Integer Integer

1st column 2nd column

SAP HANA Predictive Analysis Library (PAL) PAL Functions

SAP HANA Predictive Analysis Library (PAL) PAL Functions

SAP HANA Predictive Analysis Library (PAL) PAL Functions

Thus, the Euclidean distance between T1 and T2 is:

PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.

SAP HANA Predictive Analysis Library (PAL) PAL Functions

SAP HANA Predictive Analysis Library (PAL) PAL Functions

Data Type Integer

SAP HANA Predictive Analysis Library (PAL) PAL Functions

Data Type Double

1st column Other columns

Integer Double or string

SAP HANA Predictive Analysis Library (PAL) PAL Functions

'DM_PAL.PAL_KMEANS_DATA_T', 'in'); 'DM_PAL.PAL_CONTROL_T', 'in'); 'DM_PAL.PAL_KMEANS_ASSIGN_T', 'out'); 'DM_PAL.PAL_KMEANS_CENTERS_T',

PUBLIC 2013 SAP AG or an SAP affiliate company. All rights reserved.

SAP HANA Predictive Analysis Library (PAL) PAL Functions

SAP HANA Predictive Analysis Library (PAL) PAL Functions