Pig Script Results - Amanath

Amanath Mohammed
Page 1 of 31
PIG SCRIPTS
Script: loading and parsing data
A = load '/weatherPIG' using TextLoader as (data:chararray);
AF = foreach A generate TRIM(SUBSTRING(data, 6, 14)), TRIM(SUBSTRING(data,
46, 53)), TRIM(SUBSTRING(data, 38, 45));
store AF into '/data6' using PigStorage(',');
S = load '/data6/part-m-00000' using PigStorage(',') as (date:chararray,
min:double, max:double);
Execution Result:
Grunt> A = load '/pig/weatherPIG' using TextLoader as (data:chararray);
Grunt> dump A;
Success!
Job Stats (time in seconds):
JobId Maps Reduces
MaxMapTime MinMapTIme AvgMapTime MedianMapTime
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452816481079_0022 1
0
3
3
3
3
n/a
n/a
n/a
n/a
A
MAP_ONLY
hdfs://localhost:8020/tmp/temp800710239/tmp1385790899,
Input(s):
Successfully read 163 records (35887 bytes) from: "/pig/weatherPIG"
Output(s):
Successfully stored 163 records (36349 bytes) in:
"hdfs://localhost:8020/tmp/temp-800710239/tmp1385790899"
Counters:
Total records written : 163
Total bytes written : 36349
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1452816481079_0022
2016-01-15 09:45:19,581 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLaun
cher - Success!
2016-01-15 09:45:19,582 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2016-01-15 09:45:19,582 [main] INFO
org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not
set... will not generate code.
Amanath Mohammed
Page 2 of 31
2016-01-15 09:45:19,590 [main] INFO

org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2016-01-15 09:45:19,590 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1
(25380 20130101 2.514 -135.69
58.43
8.3
1.1
4.7
4.9
5.6
0.01 C
1.0
-0.1
0.4
97.3
36.0
69.4 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130102 2.514 -135.69
58.43
3.0
-0.3
1.4
1.2
0.0
0.35 C
1.3
-1.0
-0.1
100.0
89.5
98.2 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130103 2.514 -135.69
58.43
0.9
-0.5
0.2
0.0
2.8
0.43 C
0.1
-1.0
-0.2
99.9
96.5
99.5 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130104 2.514 -135.69
58.43
0.6
29.0
45.2
0.0
1.0
0.14 C
-0.1
-5.1
-1.1
99.9
97.1
99.0 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130105 2.514 -135.69
58.43
69.6
42.8
0.9
0.3
3.4
0.09 C
0.2
-2.1
-0.3
100.0
90.1
98.7 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130106 2.514 -135.69
58.43
4.2
-1.9
1.2
2.1
0.9
0.61 C
0.4
-5.1
-0.5
99.8
71.6
88.5 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130107 2.514 -135.69 -9999.0 -9999.0 -9999.0 -9999.0
0.8
15.2
0.12 C
0.2
-0.3
0.0
99.9
92.1
97.7 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
Grunt> AF = foreach A generate TRIM(SUBSTRING(data, 6, 14)),
TRIM(SUBSTRING(data, 46, 53)), TRIM(SUBSTRING(data, 38, 45));
Grunt> dump AF;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452816481079_0023 1
0
4
4
4
4
n/a
n/a
n/a
n/a
A,AF MAP_ONLY
hdfs://localhost:8020/tmp/temp-800710239/tmp1182280260,
Input(s):
Output(s):
"hdfs://localhost:8020/tmp/temp-800710239/tmp-1182280260"
Counters:
Amanath Mohammed
Page 3 of 31

Job DAG:
job_1452816481079_0023
2016-01-15 10:08:12,787 [main] INFO
cher - Success!
2016-01-15 10:08:12,792 [main] INFO
2016-01-15 10:08:12,795 [main] INFO
2016-01-15 10:08:12,801 [main] INFO
to process : 1
2016-01-15 10:08:12,801 [main] INFO
(20130101,1.1,8.3)
(20130102,-0.3,3.0)
(20130103,-0.5,0.9)
(20130104,29.0,0.6)
(20130105,42.8,69.6)
(20130106,-1.9,4.2)
(20130107,-9999.0,-9999.0)
(20130108,-1.2,2.4)
(20130109,-8.6,0.6)
(20130110,-2.8,-0.9)
(20130111,-1.2,0.1)
(20130112,0.0,0.3)
Grunt> store AF into '/pig/data6' using PigStorage(',');
Grunt> dump AF;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0003 1
0
2
2
2
2
n/a
n/a
n/a
n/a
A,AF MAP_ONLY
/data6,
Input(s):
Output(s):
Successfully stored 163 records (2930 bytes) in: "/data6"
Counters:
Amanath Mohammed
Page 4 of 31

Job DAG:
job_1452899978503_0003
2016-01-16 05:31:50,024 [main] INFO
cher - Success!
Grunt> S = load '/data6/part-m-00000' using PigStorage(',') as
(date:chararray, min:double, max:double);
Grunt> dump S;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0008 1
0
2
2
2
2
n/a
n/a
n/a
n/a
S
MAP_ONLY
Input(s):
Successfully read 163 records (3289 bytes) from: "/data6/part-m-00000"
Output(s):
"hdfs://localhost:8020/tmp/temp1501487145/tmp1581688072"
Counters:
Job DAG:
job_1452899978503_0008
2016-01-16 05:44:57,728 [main] INFO
cher - Success!
2016-01-16 05:44:57,729 [main] INFO
2016-01-16 05:44:57,729 [main] INFO
Amanath Mohammed
Page 5 of 31
2016-01-16 05:44:57,737 [main] INFO

to process : 1
2016-01-16 05:44:57,737 [main] INFO
(20130101,1.1,8.3)
(20130102,-0.3,3.0)
(20130103,-0.5,0.9)
(20130104,29.0,0.6)
(20130105,42.8,69.6)
(20130106,-1.9,4.2)
(20130107,-9999.0,-9999.0)
(20130108,-1.2,2.4)
(20130109,-8.6,0.6)
(20130110,-2.8,-0.9)
(20130111,-1.2,0.1)
(20130112,0.0,0.3)
Script: Hot Days
X = filter S by max > 25;
dump X;
Execution Result:
Grunt> X = filter S by max> 25;
Grunt> dump X;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0009 1
0
3
3
3
3
n/a
n/a
n/a
n/a
S,X
MAP_ONLY
Input(s):
Output(s):
Counters:
Job DAG:
job_1452899978503_0009
Amanath Mohammed
Page 6 of 31
2016-01-16 05:47:45,287 [main] INFO

cher - Success!
2016-01-16 05:47:45,287 [main] INFO
2016-01-16 05:47:45,288 [main] INFO
2016-01-16 05:47:45,293 [main] INFO
to process : 1
2016-01-16 05:47:45,293 [main] INFO
(20130105,42.8,69.6)
Script: Cold Days
X = filter S by min < 0;
dump X;
Execution Result:
Grunt> X = filter S by min < 0;
Grunt> dump X;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0010 1
0
2
2
2
2
n/a
n/a
n/a
n/a
S,X
MAP_ONLY
Input(s):
Output(s):
"hdfs://localhost:8020/tmp/temp1501487145/tmp-1563597824"
Counters:
Job DAG:
job_1452899978503_0010
Amanath Mohammed
Page 7 of 31
2016-01-16 05:52:58,221 [main] INFO

cher - Success!
2016-01-16 05:52:58,222 [main] INFO
2016-01-16 05:52:58,223 [main] INFO
2016-01-16 05:52:58,228 [main] INFO
to process : 1
2016-01-16 05:52:58,228 [main] INFO
(20130102,-0.3,3.0)
(20130103,-0.5,0.9)
(20130106,-1.9,4.2)
(20130107,-9999.0,-9999.0)
(20130108,-1.2,2.4)
(20130109,-8.6,0.6)
(20130110,-2.8,-0.9)
(20130111,-1.2,0.1)
(20130115,-0.1,5.0)
Script: Hottest Day
/* puts S's data in H1's Tuple */
H1 = group S all;
I = foreach H1 generate MAX(S.max) as maximum;
X = filter S by max == I.maximum;
Execution Result:
Grunt> H1 = group S all;
Grunt> dump H1;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0011 1
1
2
2
2
2
3
3
3
3
H1,S GROUP_BY
Input(s):
Output(s):
Amanath Mohammed
Page 8 of 31

Counters:
Job DAG:
job_1452899978503_0011
2016-01-16 05:59:06,274 [main] INFO
cher - Success!
2016-01-16 05:59:06,275 [main] INFO
2016-01-16 05:59:06,275 [main] INFO
2016-01-16 05:59:06,281 [main] INFO
to process : 1
2016-01-16 05:59:06,281 [main] INFO
(all,{(20130612,4.5,15.6),(20130611,1.8,24.2),(20130610,3.4,20.6),(2013060
9,0.6,18.3),(20130608,1.0,17.7),(20130607,4.9,16.3),(20130606,6.3,13.2),(2
0130605,6.7,9.5),(20130604,7.5,12.7),(20130603,6.7,13.2),(20130602,6.6,13.
4),(20130601,6.6,9.2),(20130531,7.1,9.4),(20130530,8.4,16.3),(20130529,4.9
,20.3),(20130528,2.9,23.8),(20130527,1.9,21.1),(20130526,1.3,18.3),(20130525,1.3,13.7),(20130524,-0.6,15.2),(20130523,2.5,15.6),(20130522,-1.3,17.1),(20130521,3.5,16.0),(20130520,2.2,11.2),(20130519,0.9,8.0),(20130518,2.0,9.3),(20130517,3.5,11.9),(20130516,2.0,12.6),(20130515,0.4,10.7),(20130514,6.8,10.4),(20130513,7.1,9.1),(20130512,5.6,13.3),(2013
0511,4.7,8.6),(20130510,-2.9,18.3),(20130509,-2.9,17.0),(20130508,3.8,15.2),(20130507,2.2,11.2),(20130506,1.5,8.7),(20130505,5.4,7.9),(20130504,6.5,10.3),(20130
503,5.7,9.4),(20130502,5.4,8.8),(20130501,3.2,7.2),(20130430,0.2,4.0),(20130429,-4.5,5.0),(20130428,2.5,3.3),(20130427,0.0,4.1),(20130426,0.1,1.5),(20130425,0.9,6.0),(2013042
4,2.1,5.6),(20130423,-0.4,7.0),(20130422,0.1,3.6),(20130421,10.1,9.0),(20130420,-7.1,8.6),(20130419,-9.6,6.9),(20130418,5.8,5.9),(20130417,2.3,8.3),(20130416,-1.5,4.4),(20130415,8.3,7.1),(20130414,-5.2,8.9),(20130413,-1.9,8.0),(20130412,0.2,7.6),(20130411,-3.3,3.0),(20130410,-1.7,2.1),(20130409,0.3,6.5),(20130408,-4.3,7.0),(20130407,-3.4,11.2),(20130406,2.6,4.4),(20130405,-7.2,3.7),(20130404,-8.7,5.8),(20130403,5.8,5.5),(20130402,-0.7,5.4),(20130401,0.0,4.2),(20130331,-
Amanath Mohammed
Page 9 of 31
1.0,6.3),(20130330,0.3,6.7),(20130329,-3.0,5.1),(20130328,6.7,4.2),(20130327,-5.7,4.0),(20130326,0.9,2.7),(20130325,0.9,7.5),(20130324,-0.5,4.2),(20130323,4.8,2.8),(20130322,-16.6,0.5),(20130321,-14.9,-0.6),(20130320,-15.5,0.5),(20130319,-8.2,0.3),(20130318,-18.2,-1.2),(20130317,-14.9,0.6),(20130316,-3.7,-0.5),(20130315,-5.2,-1.8),(20130314,15.1,1.3),(20130313,-12.3,-1.4),(20130312,-3.5,1.7),(20130311,0.6,4.2),(20130310,-0.6,5.0),(20130309,0.0,3.6),(20130308,5.8,4.8),(20130307,-11.5,4.5),(20130306,-11.7,4.2),(20130305,11.0,3.4),(20130304,-8.4,5.6),(20130303,-4.6,4.7),(20130302,4.3,6.3),(20130301,0.4,5.6),(20130228,2.3,6.9),(20130227,2.9,6.5),(2013022
6,2.6,5.5),(20130225,-2.5,4.2),(20130224,0.7,3.2),(20130223,0.3,2.8),(20130222,1.0,3.5),(20130221,2.3,5.2),(2013022
0,2.2,4.3),(20130219,0.4,5.6),(20130218,0.3,3.7),(20130217,0.5,4.5),(20130
216,0.7,3.7),(20130215,0.0,5.5),(20130214,2.5,6.2),(20130213,0.3,3.8),(20130212,-2.8,2.7),(20130211,0.7,4.5),(20130210,1.5,5.5),(20130209,2.2,6.6),(20130208,1.5,4.2),(2013020
7,-0.4,3.3),(20130206,-6.2,2.9),(20130205,-7.9,1.0),(20130204,3.7,4.0),(20130203,1.0,5.5),(20130202,1.2,4.6),(20130201,0.2,2.0),(2013013
1,-3.6,1.2),(20130130,-5.5,-1.3),(20130129,-12.9,-5.5),(20130128,-17.1,8.3),(20130127,-8.3,-0.4),(20130126,-1.4,3.2),(20130125,0.7,3.7),(20130124,-0.4,4.0),(20130123,0.4,4.5),(20130122,0.4,2.7),(20130121,0.5,2.1),(20130120,-1.3,0.6),(20130119,2.9,0.5),(20130118,2.1,2.1),(20130117,0.4,4.9),(20130116,0.0,2.9),(20130115,0.1,5.0),(20130114,4.3,5.4),(20130113,0.2,4.4),(20130112,0.0,0.3),(2013011
1,-1.2,0.1),(20130110,-2.8,-0.9),(20130109,-8.6,0.6),(20130108,1.2,2.4),(20130107,-9999.0,-9999.0),(20130106,1.9,4.2),(20130105,42.8,69.6),(20130104,29.0,0.6),(20130103,0.5,0.9),(20130102,-0.3,3.0),(20130101,1.1,8.3)})
Grunt> I = foreach H1 generate MAX(S.max) as maximum;
Grunt> dump I;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0012 1
1
2
2
2
2
2
2
2
2
H1,I,S
GROUP_BY,COMBINER
hdfs://localhost:8020/tmp/temp1501487145/tmp-446300512,
Input(s):
Output(s):
Counters:
Amanath Mohammed
Page 10 of 31

Job DAG:
job_1452899978503_0012
2016-01-16 06:02:19,421 [main] INFO
cher - Success!
2016-01-16 06:02:19,422 [main] INFO
2016-01-16 06:02:19,422 [main] INFO
2016-01-16 06:02:19,428 [main] INFO
to process : 1
2016-01-16 06:02:19,428 [main] INFO
(69.6)
Grunt> X = filter S by max == I.maximum;
Grunt> dump X;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0013 1
1
3
3
3
3
2
2
2
2
H1,I,S
MULTI_QUERY,COMBINER
job_1452899978503_0014 1
0
2
2
2
2
n/a
n/a
n/a
n/a
X
MAP_ONLY
Input(s):
Output(s):
Counters:
Amanath Mohammed
Job DAG:
job_1452899978503_0013 ->
job_1452899978503_0014
Page 11 of 31
job_1452899978503_0014,
2016-01-16 06:03:57,468 [main] INFO org.apache.hadoop.ipc.Client Retrying connect to server: localhost.localdomain/127.0.0.1:58853. Already
tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1 SECONDS)
2016-01-16 06:03:59,574 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
2016-01-16 06:03:59,896 [main] INFO
server
2016-01-16 06:04:00,034 [main] INFO
cher - Success!
2016-01-16 06:04:00,036 [main] INFO
2016-01-16 06:04:00,036 [main] INFO
2016-01-16 06:04:00,039 [main] INFO
to process : 1
2016-01-16 06:04:00,039 [main] INFO
(20130105,42.8,69.6)
Script: Coldest Day
H2 = group S all;
J = foreach H2 generate MIN(S.min) as minimum;
X = filter S by min == J.minimum;
Execution Result:
Grunt> H2 = group S all;
Grunt> dump H2;
Success!
Amanath Mohammed
Page 12 of 31

JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0015 1
1
2
2
2
2
2
2
2
2
H2,S GROUP_BY
Input(s):
Output(s):
Counters:
Job DAG:
job_1452899978503_0015
2016-01-16 06:08:33,532 [main] INFO
cher - Success!
2016-01-16 06:08:33,533 [main] INFO
2016-01-16 06:08:33,533 [main] INFO
2016-01-16 06:08:33,539 [main] INFO
to process : 1
2016-01-16 06:08:33,539 [main] INFO
(all,{(20130612,4.5,15.6),(20130611,1.8,24.2),(20130610,3.4,20.6),(2013060
9,0.6,18.3),(20130608,1.0,17.7),(20130607,4.9,16.3),(20130606,6.3,13.2),(2
0130605,6.7,9.5),(20130604,7.5,12.7),(20130603,6.7,13.2),(20130602,6.6,13.
4),(20130601,6.6,9.2),(20130531,7.1,9.4),(20130530,8.4,16.3),(20130529,4.9
,20.3),(20130528,2.9,23.8),(20130527,1.9,21.1),(20130526,1.3,18.3),(20130525,1.3,13.7),(20130524,-0.6,15.2),(20130523,2.5,15.6),(20130522,-1.3,17.1),(20130521,3.5,16.0),(20130520,2.2,11.2),(20130519,0.9,8.0),(20130518,2.0,9.3),(20130517,3.5,11.9),(20130516,2.0,12.6),(20130515,0.4,10.7),(20130514,6.8,10.4),(20130513,7.1,9.1),(20130512,5.6,13.3),(2013
0511,4.7,8.6),(20130510,-2.9,18.3),(20130509,-2.9,17.0),(20130508,3.8,15.2),(20130507,-
Amanath Mohammed
Page 13 of 31
2.2,11.2),(20130506,1.5,8.7),(20130505,5.4,7.9),(20130504,6.5,10.3),(20130
503,5.7,9.4),(20130502,5.4,8.8),(20130501,3.2,7.2),(20130430,0.2,4.0),(20130429,-4.5,5.0),(20130428,2.5,3.3),(20130427,0.0,4.1),(20130426,0.1,1.5),(20130425,0.9,6.0),(2013042
4,2.1,5.6),(20130423,-0.4,7.0),(20130422,0.1,3.6),(20130421,10.1,9.0),(20130420,-7.1,8.6),(20130419,-9.6,6.9),(20130418,5.8,5.9),(20130417,2.3,8.3),(20130416,-1.5,4.4),(20130415,8.3,7.1),(20130414,-5.2,8.9),(20130413,-1.9,8.0),(20130412,0.2,7.6),(20130411,-3.3,3.0),(20130410,-1.7,2.1),(20130409,0.3,6.5),(20130408,-4.3,7.0),(20130407,-3.4,11.2),(20130406,2.6,4.4),(20130405,-7.2,3.7),(20130404,-8.7,5.8),(20130403,5.8,5.5),(20130402,-0.7,5.4),(20130401,0.0,4.2),(20130331,1.0,6.3),(20130330,0.3,6.7),(20130329,-3.0,5.1),(20130328,6.7,4.2),(20130327,-5.7,4.0),(20130326,0.9,2.7),(20130325,0.9,7.5),(20130324,-0.5,4.2),(20130323,4.8,2.8),(20130322,-16.6,0.5),(20130321,-14.9,-0.6),(20130320,-15.5,0.5),(20130319,-8.2,0.3),(20130318,-18.2,-1.2),(20130317,-14.9,0.6),(20130316,-3.7,-0.5),(20130315,-5.2,-1.8),(20130314,15.1,1.3),(20130313,-12.3,-1.4),(20130312,-3.5,1.7),(20130311,0.6,4.2),(20130310,-0.6,5.0),(20130309,0.0,3.6),(20130308,5.8,4.8),(20130307,-11.5,4.5),(20130306,-11.7,4.2),(20130305,11.0,3.4),(20130304,-8.4,5.6),(20130303,-4.6,4.7),(20130302,4.3,6.3),(20130301,0.4,5.6),(20130228,2.3,6.9),(20130227,2.9,6.5),(2013022
6,2.6,5.5),(20130225,-2.5,4.2),(20130224,0.7,3.2),(20130223,0.3,2.8),(20130222,1.0,3.5),(20130221,2.3,5.2),(2013022
0,2.2,4.3),(20130219,0.4,5.6),(20130218,0.3,3.7),(20130217,0.5,4.5),(20130
216,0.7,3.7),(20130215,0.0,5.5),(20130214,2.5,6.2),(20130213,0.3,3.8),(20130212,-2.8,2.7),(20130211,0.7,4.5),(20130210,1.5,5.5),(20130209,2.2,6.6),(20130208,1.5,4.2),(2013020
7,-0.4,3.3),(20130206,-6.2,2.9),(20130205,-7.9,1.0),(20130204,3.7,4.0),(20130203,1.0,5.5),(20130202,1.2,4.6),(20130201,0.2,2.0),(2013013
1,-3.6,1.2),(20130130,-5.5,-1.3),(20130129,-12.9,-5.5),(20130128,-17.1,8.3),(20130127,-8.3,-0.4),(20130126,-1.4,3.2),(20130125,0.7,3.7),(20130124,-0.4,4.0),(20130123,0.4,4.5),(20130122,0.4,2.7),(20130121,0.5,2.1),(20130120,-1.3,0.6),(20130119,2.9,0.5),(20130118,2.1,2.1),(20130117,0.4,4.9),(20130116,0.0,2.9),(20130115,0.1,5.0),(20130114,4.3,5.4),(20130113,0.2,4.4),(20130112,0.0,0.3),(2013011
1,-1.2,0.1),(20130110,-2.8,-0.9),(20130109,-8.6,0.6),(20130108,1.2,2.4),(20130107,-9999.0,-9999.0),(20130106,1.9,4.2),(20130105,42.8,69.6),(20130104,29.0,0.6),(20130103,0.5,0.9),(20130102,-0.3,3.0),(20130101,1.1,8.3)})
Grunt> J = foreach H2 generate MIN(S.min) as minimum;
Grunt> dump J;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
Amanath Mohammed
job_1452899978503_0016 1
1
2
2
2
2
2
2
H2,J,S
GROUP_BY,COMBINER
Page 14 of 31
2
Input(s):
Output(s):
Counters:
Job DAG:
job_1452899978503_0016
2016-01-16 06:10:13,058 [main] INFO
cher - Success!
2016-01-16 06:10:13,059 [main] INFO
2016-01-16 06:10:13,060 [main] INFO
2016-01-16 06:10:13,065 [main] INFO
to process : 1
2016-01-16 06:10:13,066 [main] INFO
(-9999.0)
Grunt> X = filter S by min == J.minimum;
Grunt> dump X;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0017 1
1
3
3
3
3
5
5
5
5
H2,J,S
MULTI_QUERY,COMBINER
job_1452899978503_0018 1
0
2
2
2
2
n/a
n/a
n/a
n/a
X
MAP_ONLY
Amanath Mohammed
Page 15 of 31
Input(s):
Output(s):
Counters:
Job DAG:
job_1452899978503_0017 ->
job_1452899978503_0018
job_1452899978503_0018,
2016-01-16 06:14:28,146 [main] INFO
server
2016-01-16 06:14:28,393 [main] INFO
server
2016-01-16 06:14:28,509 [main] INFO
cher - Success!
2016-01-16 06:14:28,509 [main] INFO
2016-01-16 06:14:28,510 [main] INFO
2016-01-16 06:14:28,513 [main] INFO
to process : 1
2016-01-16 06:14:28,513 [main] INFO
Amanath Mohammed
Page 16 of 31
(20130107,-9999.0,-9999.0)
Script:
A = load 'pig/weatherPIG' using TextLoader as (data:chararray);
AF = foreach A generate TRIM(SUBSTRING(data, 6, 14)),
IfCorrupted(TRIM(SUBSTRING(data, 46, 53))),
IfCorrupted(TRIM(SUBSTRING(data, 38, 45)));
store AF into '/pig/data2' using PigStorage(',');
S = load '/pig/data2/part-m-00000' using PigStorage(',') as
(date:chararray, min:double, max:double);
Execution Result:
Grunt> A = load '/pig/weatherPIG' using TextLoader as (data:chararray);
Grunt> dump A;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0019 1
0
3
3
3
3
n/a
n/a
n/a
n/a
A
MAP_ONLY
Input(s):
Output(s):
Counters:
Job DAG:
job_1452899978503_0019
2016-01-16 06:23:03,254 [main] INFO
cher - Success!
2016-01-16 06:23:03,255 [main] INFO
2016-01-16 06:23:03,255 [main] INFO
Amanath Mohammed
Page 17 of 31
2016-01-16 06:23:03,263 [main] INFO

to process : 1
2016-01-16 06:23:03,263 [main] INFO
(25380 20130101 2.514 -135.69
58.43
8.3
1.1
4.7
4.9
5.6
0.01 C
1.0
-0.1
0.4
97.3
36.0
69.4 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130102 2.514 -135.69
58.43
3.0
-0.3
1.4
1.2
0.0
0.35 C
1.3
-1.0
-0.1
100.0
89.5
98.2 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130103 2.514 -135.69
58.43
0.9
-0.5
0.2
0.0
2.8
0.43 C
0.1
-1.0
-0.2
99.9
96.5
99.5 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
(25380 20130104 2.514 -135.69
58.43
0.6
29.0
45.2
0.0
1.0
0.14 C
-0.1
-5.1
-1.1
99.9
97.1
99.0 -99.000 99.000 -99.000 -99.000 -99.000 -9999.0 -9999.0 -9999.0 -9999.0 -9999.0)
Grunt> AF = foreach A generate TRIM(SUBSTRING(data, 6, 14)),
IfCorrupted(TRIM(SUBSTRING(data, 46, 53))),
IfCorrupted(TRIM(SUBSTRING(data, 38, 45)));
Grunt> dump AF;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0020 1
0
3
3
3
3
n/a
n/a
n/a
n/a
A,AF MAP_ONLY
Input(s):
Output(s):
Counters:
Job DAG:
job_1452899978503_0020
Amanath Mohammed
Page 18 of 31
2016-01-16 06:25:36,246 [main] INFO

cher - Success!
2016-01-16 06:25:36,247 [main] INFO
2016-01-16 06:25:36,247 [main] INFO
2016-01-16 06:25:36,253 [main] INFO
to process : 1
2016-01-16 06:25:36,253 [main] INFO
(20130101,1.1,8.3)
(20130102,-0.3,3.0)
(20130103,-0.5,0.9)
(20130104,29.0,0.6)
(20130105,42.8,69.6)
(20130106,-1.9,4.2)
(20130107,-9999.0,-9999.0)
(20130108,-1.2,2.4)
(20130109,-8.6,0.6)
(20130110,-2.8,-0.9)
(20130111,-1.2,0.1)
(20130112,0.0,0.3)
(20130113,0.2,4.4)
(20130114,4.3,5.4)
(20130115,-0.1,5.0)
(20130116,0.0,2.9)
Grunt> store AF into '/pig/data2' using PigStorage(',');
Grunt> dump AF;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452899978503_0021 1
0
3
3
3
3
n/a
n/a
n/a
n/a
A,AF MAP_ONLY
Input(s):
Output(s):
Counters:
Amanath Mohammed
Page 19 of 31

Job DAG:
job_1452899978503_0021
2016-01-16 06:44:33,805 [main] INFO
cher - Success!
2016-01-16 06:44:33,806 [main] INFO
2016-01-16 06:44:33,806 [main] INFO
2016-01-16 06:44:33,815 [main] INFO
to process : 1
2016-01-16 06:44:33,815 [main] INFO
(20130101,1.1,8.3)
(20130102,-0.3,3.0)
(20130103,-0.5,0.9)
(20130104,29.0,0.6)
(20130105,42.8,69.6)
(20130106,-1.9,4.2)
(20130107,-9999.0,-9999.0)
(20130108,-1.2,2.4)
(20130109,-8.6,0.6)
(20130110,-2.8,-0.9)
(20130111,-1.2,0.1)
(20130112,0.0,0.3)
(20130113,0.2,4.4)
(20130114,4.3,5.4)
(20130115,-0.1,5.0)
(20130116,0.0,2.9)
(20130117,0.4,4.9)
Grunt> S = load '/pig/data2/' using PigStorage(',') as (date:chararray,
min:double, max:double);
Grunt> dump S;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
Amanath Mohammed
job_1452899978503_0025 1
n/a
S
MAP_ONLY
48873957,
Page 20 of 31
0
3
3
3
3
n/a
n/a
n/a
hdfs://localhost:8020/tmp/temp1501487145/tmp-
Input(s):
Successfully read 7 records (385 bytes) from: "/pig/data2"
Output(s):
Counters:
Job DAG:
job_1452899978503_0025
2016-01-16 08:20:05,810 [main] WARN
cher - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 14 time(s).
2016-01-16 08:20:05,811 [main] INFO
cher - Success!
2016-01-16 08:20:05,811 [main] INFO
2016-01-16 08:20:05,812 [main] INFO
2016-01-16 08:20:05,817 [main] INFO
to process : 1
2016-01-16 08:20:05,817 [main] INFO
(2
4,,)
(8
9,,)
(1
3,,)
(2
7,,)
(2
9,,)
(4
6,,)
(4
9,,)
Amanath Mohammed
Page 21 of 31
Script:
A = load '/data1' as (a1:int, a2:int);
B = load '/data2' as (b1:int, b2:int);
X = UNION A, B;
dump X;
//onschema
Execution Result:
Grunt> A = load '/pig/data1' as (a1:int, a2:int);
Grunt> dump A;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452816481079_0018 1
0
3
3
3
3
n/a
n/a
n/a
n/a
A
MAP_ONLY
Input(s):
Output(s):
Counters:
Job DAG:
job_1452816481079_0018
2016-01-15 08:56:20,486 [main] INFO
cher - Success!
2016-01-15 08:56:20,487 [main] INFO
2016-01-15 08:56:20,487 [main] INFO
2016-01-15 08:56:20,492 [main] INFO
to process : 1
Amanath Mohammed
Page 22 of 31
2016-01-15 08:56:20,492 [main] INFO

(1,2)
(4,2)
(8,3)
(4,3)
(7,2)
(8,4)
Grunt> B = load '/pig/data2' as (b1:int, b2:int);
Grunt> dump = B;
Success!
JobId Maps Reduces
MaxReduceTime
Alias Feature

MinReduceTime
AvgReduceTime
MedianReducetime
Outputs
job_1452816481079_0019 1
n/a
B
MAP_ONLY
800710239/tmp1566287174,
0
4
4
4
4
n/a
hdfs://localhost:8020/tmp/temp-
Input(s):
Output(s):
Counters:
Job DAG:
job_1452816481079_0019
n/a
n/a
Amanath Mohammed
Page 23 of 31
2016-01-15 08:58:27,124 [main] INFO

cher - Success!
2016-01-15 08:58:27,126 [main] INFO
2016-01-15 08:58:27,126 [main] INFO
2016-01-15 08:58:27,137 [main] INFO
to process : 1
2016-01-15 08:58:27,137 [main] INFO
(2,4)
(8,9)
(1,3)
(2,7)
(2,9)
(4,6)
(4,9)
Grunt> X = UNION A, B;
Grunt> dump X;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452816481079_0020 2
0
6
6
6
6
n/a
n/a
n/a
n/a
A,B,X MAP_ONLY
Input(s):
Successfully read 6 records from: "/pig/data1"
Successfully read 7 records from: "/pig/data2"
Amanath Mohammed
Page 24 of 31
Output(s):
Counters:
Job DAG:
job_1452816481079_0020
2016-01-15 09:02:06,169 [main] INFO
cher - Success!
2016-01-15 09:02:06,169 [main] INFO
2016-01-15 09:02:06,169 [main] INFO
2016-01-15 09:02:06,176 [main] INFO
to process : 2
2016-01-15 09:02:06,176 [main] INFO
(2,4)
(8,9)
(1,3)
(2,7)
(2,9)
(4,6)
(4,9)
(1,2)
(4,2)
(8,3)
(4,3)
(7,2)
(8,4)
Amanath Mohammed
Page 25 of 31
Script:
A = LOAD '/pig/j1' as (a1:int, a2:int, a3:int);
B = LOAD '/pig/j2' as (b1:int, b2:int);
X = JOIN A BY a1, B BY b1;
dump X;
Execution Result:
Grunt> A = LOAD '/pig/j1' as (a1:int, a2:int, a3:int);
Grunt> dump A;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452816481079_0015 1
0
3
3
3
3
n/a
n/a
n/a
n/a
A
MAP_ONLY
Input(s):
Successfully read 6 records (383 bytes) from: "/pig/j1"
Output(s):
Counters:
Job DAG:
job_1452816481079_0015
2016-01-15 07:30:09,291 [main] INFO
cher - Success!
2016-01-15 07:30:09,292 [main] INFO
2016-01-15 07:30:09,293 [main] INFO
2016-01-15 07:30:09,300 [main] INFO
to process : 1
Amanath Mohammed
Page 26 of 31
2016-01-15 07:30:09,300 [main] INFO

(1,2,3)
(4,2,1)
(8,3,4)
(4,3,3)
(7,2,5)
(8,4,3)
Grunt> B = LOAD '/pig/j2' as (b1:int, b2:int);
Grunt> dump B;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452816481079_0016 1
0
3
3
3
3
n/a
n/a
n/a
n/a
B
MAP_ONLY
Input(s):
Successfully read 7 records (375 bytes) from: "/pig/j2"
Output(s):
Counters:
Job DAG:
job_1452816481079_0016
2016-01-15 07:33:32,965 [main] INFO
cher - Success!
2016-01-15 07:33:32,966 [main] INFO
2016-01-15 07:33:32,966 [main] INFO
Amanath Mohammed
Page 27 of 31
2016-01-15 07:33:32,973 [main] INFO

to process : 1
2016-01-15 07:33:32,974 [main] INFO
(2,4)
(8,9)
(1,3)
(2,7)
(2,9)
(4,6)
(4,9)
Grunt> X = JOIN A BY a1, B BY b1;
Grunt> dump X;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452816481079_0017 2
1
7
7
7
7
3
3
3
3
A,B,X HASH_JOIN hdfs://localhost:8020/tmp/temp-800710239/tmp1688811822,
Input(s):
Successfully read 7 records from: "/pig/j2"
Successfully read 6 records from: "/pig/j1"
Output(s):
Counters:
Job DAG:
job_1452816481079_0017
2016-01-15 07:38:16,624 [main] INFO
cher - Success!
2016-01-15 07:38:16,625 [main] INFO
Amanath Mohammed
Page 28 of 31
2016-01-15 07:38:16,625 [main] INFO

2016-01-15 07:38:16,631 [main] INFO
to process : 1
2016-01-15 07:38:16,631 [main] INFO
(1,2,3,1,3)
(4,3,3,4,9)
(4,3,3,4,6)
(4,2,1,4,9)
(4,2,1,4,6)
(8,4,3,8,9)
(8,3,4,8,9)
Script:
D = load '/pig/student' as (name:chararray, age:int, gpa:float);
E = load '/pig/studentRoll' as (name:chararray, rollno:int);
F = group D by name;
dump F;
F = cogroup D by name, E by name;
dump F;
Execution Result:
Grunt> D = load '/pig/student' as (name:chararray, age:int, gpa:float);
Grunt> Dump D;
Job DAG:
job_1452816481079_0007
2016-01-15 06:35:41,376 [main] INFO
cher - Success!
2016-01-15 06:35:41,377 [main] INFO
2016-01-15 06:35:41,377 [main] INFO
2016-01-15 06:35:41,384 [main] INFO
to process : 1
2016-01-15 06:35:41,384 [main] INFO
(joe,18,2.5)
(sam,,3.0)
(angel,21,7.9)
Amanath Mohammed
Page 29 of 31
(john,17,2.0)
(joe,19,2.9)
Grunt> E = load '/pig/studentRoll' as (name:chararray, rollno:int);
Grunt> Dump E;
Output(s):
Counters:
Job DAG:
job_1452816481079_0011
2016-01-15 07:07:05,318 [main] INFO
cher - Success!
2016-01-15 07:07:05,320 [main] INFO
2016-01-15 07:07:05,321 [main] INFO
2016-01-15 07:07:05,327 [main] INFO
to process : 1
2016-01-15 07:07:05,327 [main] INFO
(joe,13)
(sam,24)
(angel,21)
(john,17)
(joe,12)
Grunt> F = Group D by Name;
Grunt> dump F;
Output(s):
Counters:
Amanath Mohammed
Page 30 of 31

Job DAG:
job_1452816481079_0013
2016-01-15 07:20:07,652 [main] INFO
cher - Success!
2016-01-15 07:20:07,654 [main] INFO
2016-01-15 07:20:07,654 [main] INFO
2016-01-15 07:20:07,660 [main] INFO
to process : 1
2016-01-15 07:20:07,660 [main] INFO
(joe,{(joe,19,2.9),(joe,18,2.5)})
(sam,{(sam,,3.0)})
(john,{(john,17,2.0)})
(angel,{(angel,21,7.9)})
Grunt> F = cogroup D by name, E by name;
Grunt> dump F;
Success!
JobId Maps Reduces
MaxReduceTime
MinReduceTime
AvgReduceTime
MedianReducetime
Alias Feature
Outputs
job_1452816481079_0014 2
1
7
7
7
7
3
3
3
3
D,E,F COGROUP
Input(s):
Successfully read 5 records from: "/pig/student"
Successfully read 5 records from: "/pig/studentRoll"
Output(s):
Counters:
Amanath Mohammed
Page 31 of 31

Job DAG:
job_1452816481079_0014
2016-01-15 07:22:41,500 [main] INFO
cher - Success!
2016-01-15 07:22:41,501 [main] INFO
2016-01-15 07:22:41,501 [main] INFO
2016-01-15 07:22:41,507 [main] INFO
to process : 1
2016-01-15 07:22:41,507 [main] INFO
(joe,{(joe,19,2.9),(joe,18,2.5)},{(joe,12),(joe,13)})
(sam,{(sam,,3.0)},{(sam,24)})
(john,{(john,17,2.0)},{(john,17)})
(angel,{(angel,21,7.9)},{(angel,21)})
End of File.

Pig Script Results - Amanath

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pig Script Results - Amanath

Uploaded by

Copyright:

Available Formats

Amanath Mohammed

2016-01-15 09:45:19,590 [main] INFO

Total records proactively spilled: 0

Total bytes written : 2930

2016-01-16 05:44:57,737 [main] INFO

2016-01-16 05:47:45,287 [main] INFO

2016-01-16 05:52:58,221 [main] INFO

Successfully stored 1 records (4902 bytes) in:

Spillable Memory Manager spill count : 0

Job Stats (time in seconds):

2016-01-16 06:23:03,263 [main] INFO

2016-01-16 06:25:36,246 [main] INFO

Total bytes written : 4560

2016-01-15 08:56:20,492 [main] INFO

MaxMapTime MinMapTIme AvgMapTime MedianMapTime

2016-01-15 08:58:27,124 [main] INFO

2016-01-15 07:30:09,300 [main] INFO

2016-01-15 07:33:32,973 [main] INFO

2016-01-15 07:38:16,625 [main] INFO

Total bags proactively spilled: 0

Total records proactively spilled: 0

You might also like