SCOPE Basics

Introduction to SCOPE

What is Cosmos and SCOPE?

Cosmos is Microsoft's internal Big Data system and SCOPE is its query language.

Structured Computations Optimized for Parallel Execution

 A scripting language
 Easy to use: SQL-like syntax
 Build and submit the scripts on Cosmos
 U-SQL scripts are used (U-SQL is a language that combines declarative SQL with imperative
C# to let you process data at any scale)
 To compile, Right click on the script and select on “Build Script”
 To submit it on the cosmos cluster for execution(Run), Right click on the script and select
“Submit”

Dissimilarities with SQL

 Window function such as OVER is not supported in SCOPE

 Nested Queries such as SELECT query within a SELECT query is also not allowed
 In SCOPE, Output has to be assigned at a specified location and file. Otherwise the output is
not stored

Input, Output and Extract Syntax

IMPORT

IMPORT <Script File>

[PARAMS<par_name>=<value>]

EXTRACT

UETGoalMap =
EXTRACT CustomerId, AccountId
FROM @UETGoalMapFile
USING CSVFileExtractor("UETGoalMap_Stripe.dsv", "UETGoalMap_Stripe",
@CsvExtractorArgs);

Output To

JoinedData =
SELECT ClickData.*
FROM ClickData
LEFT JOIN ConversionData
ON ClickData.ClickId == ConversionData.ClickId;
OUTPUT TO SSTREAM@JoinedData WITH STREAMEXPIRY "60";
Basic Aggregations in SCOPE

 SCOPE supports basic aggregation functions like SUM, MAX, Min etc.
 But SCOPE does not support subqueries such as:

Equivalent Query in SCOPE:

Clusters Overview:

 We have been provided access to two clusters: VC1 and VC2

 Script File can be uploaded in either of the two Clusters
 But adRelevance Script files are only uploaded in VC2
 The status of file can be checked on the below link:
https://cosmos08.osdinfra.net/cosmos/bingads.market.VC1/_jobs/
https://cosmos08.osdinfra.net/cosmos/bingads.market.VC2/_jobs/
 To check the status of the specific file, replace ‘_jobs’ in link with the file location with name.

iSCOPE

 In iSCOPE, we just execute the queries which take less than 5 Min
 It is used as Quick Check (QC)
 To run the script for 5 Min following query is written: [SCRIPT: MAXDOP=0]
 If the above query is not written, the script here will run only for 30 Sec
 To Execute the script, Right click on the script and click on ‘Execute Script’
 Enter your credentials, select ‘Custom Runtime User’ and enter ‘iscope_beta’
 Ones the details are verified, it will execute the file and display the last query output which
can be saved by clicking on the “Save to File” which appears on the top right corner of the
output table
 C# functions does not work in iSCOPE
 Here Output to function is also not mandatory. Data is only used to display and only if
output to function is mentioned is then data is saved in specific file.