Experience Smart Multipurpose Shared Global Storage With ECS

EXPERIENCE SMART
MULTIPURPOSE SHARED
GLOBAL STORAGE WITH ECS
Experience Smart Multipurpose Shared Global Storage with ECS

Table of Contents
Introduction.................................................................................................................................................. 4
About this lab ..................................................................................................................................................5
Lab Overview ................................................................................................................................................ 6

Lab Environment .............................................................................................................................................7
Lab Credentials ................................................................................................................................................8
Lab Scenario ..................................................................................................................................................10
Labs...............................................................................................................................................................11
Lab 1 - Configuring A New Tenant In ECS ..................................................................................................... 13

Tenant Configuration .....................................................................................................................................14
Lab 2 - Ingesting Data Using An S3 Application ........................................................................................... 35

Data Ingest into the ECS Cloud Storage Platform ............................................................................................36
Lab 3 - Performing Metadata Search Using An S3 Client Application ........................................................... 40

ECS Metadata Search Media Reviewer............................................................................................................41
Lab 4 - Querying Object Metadata with Apache Spark SQL for use by HDFS based applications................... 51
Querying object metadata with Apache Spark SQL for analytics......................................................................52
Lab 5 - Accessing Data through NFS ........................................................................................................... 61

Working with ECS Data via NFS and S3 ...........................................................................................................62
Troubleshooting ......................................................................................................................................... 82
Troubleshooting & Tips ..................................................................................................................................83
Conclusion ................................................................................................................................................. 87
Conclusion.....................................................................................................................................................88
Additional Solution Information.................................................................................................................. 89

Business Challenge........................................................................................................................................90
Solution .........................................................................................................................................................91
ECS ................................................................................................................................................................92
Other Client Applications ...............................................................................................................................94

Introduction
Experience Smart Multipurpose Shared Global Storage with ECS Page 4

About this lab
Elastic Cloud Storage (ECS) is a modern, multi-purpose, shared global storage that scales into Exabytes to store both
small and large files with strong global consistency.
This lab offers first hand experience of how ECS smart storage simplifies multi-tenant data access and management
with instant meta-data search and multi-protocol access to the same data set using NFS or S3.

Lab Overview

Lab Environment
The environment for this lab includes the following:
• 1 x Windows 2012 Domain Controller. This server provides DNS and AD services to the environment.
• 1 x Windows 2012 Launchpad. This server is the host you are currently logged into and provides access into
the other components of this lab.
• Two sites with three ECS nodes each. Each node is a VM running ECS 2.2 software and has two disks: one for
OS install and another for object storage. Real world ECS installs require a minimum four-node setup; this
three-node install is for demonstration purposes only.
• 1 x Linux VM that hosts Spark Notebook
Lab Architecture

Lab Credentials
The following set of credentials are used in the labs:
ECS system administrator
• Username: ecsadmin01@vlab.local
• Password: Password123!
Namespace administrator for media unit
• Username: rainpoleadmin@vlab.local
Object User for media team
• Username: mediauser@vlab.local
Alternatively, if you chose to skip the configuration steps in Lab1 and Lab2, following set of credentials have been pre-
configured for use in Labs 3, 4, and 5:
Namespace administrator for media unit
• Username: rainpoleadmin2@vlab.local
Object User for media team
• Username: mediauser2@vlab.local

Environment Credentials
The following is a list of all lab servers and appliance and their corresponding IP addresses and credentials.
Note: This table is for reference purposes, you may not access everything when completing the use cases in the lab
guide.

Lab Scenario
In these labs, you will see how the storage administrator extends the ECS Cloud Storage platform to the new tenant,
which is the media division. Then you will see how each of the three teams in the division achieve the use cases listed
below:
The content generation team needs to ingest image content and its associated metadata.
Two different teams would use the metadata associated with the media images:
• The media review team has a custom application (Media Reviewer) that needs to search for images based on
the metadata to review the content.
• The analytics team uses a Zeppelin notebook to view object metadata and run Spark SQL queries to extract
data for analysis.
A remote advertising team needs to access archived media content from their windows platform as a mounted file
system, for reference and reuse.

Labs
In the exercises, you will experience how ECS smart storage simplifies data access and management with instant
meta-data search, NFS access, analytics enabled by HDFS and built-in optimizations for speed and storage efficiency,
and access data using multiple protocols simultaneously to appreciate how ECS is built from the ground up to support
multiple protocols for unstructured (Object and File) workloads on a single cloud-scale storage platform, natively
speeding up storage for traditional archive services as well as modern web, mobile, Internet of Things (IoT), and
Hadoop applications. You can store, access or consume data without any translation gateways.
You have the opportunity to perform tenant configuration, object data ingest using S3 protocol, metadata search using
SPARK SQL and S3 client applications, and NFS access from a Windows environment.
Note: You can optionally start with Lab 3, which allows you to skip the tenant configuration and object data ingest
modules. You will use a pre-configured tenant and pre-populated data for the metadata search and NFS access Labs.
Contents
Lab 1: Configuring A New Tenant In ECS
• Perform the role of a system administrator to create new namespace and configure a namespace admin
• Perform the role of a namespace admin to enable access for users and applications within the media unit
Lab 2: Ingesting Data Using An S3 Application
• Connect an S3 client application to a namespace and object user

• Observe ingest of image data
Lab 3: Performing Metadata Search Using an S3 Client Application
• Observe how S3 client applications can leverage the ECS 2.2 metadata search capability to efficiently
discover and access data sets
Lab 4: Querying Object Metadata with Apache Spark SQL
• Experience the ability to use an SQL interface to the metadata search capability in ECS
• Write SQL queries to access data based on metdata search criteria
Lab 5: Accessing Data through NFS and S3
• Walk through the use case of ingesting content via S3 and then accessing that content via NFS mount in
Windows.

• Configure an NFS export in ECS to allow the media unit to access bucket content via NFS from external sources
sharing data via S3. In other words, experience cross head access of a bucket in ECS using two different
protocols (NFS and S3).
• Access and view the shared content in Windows using NFS Client to mount the bucket and MS Paint to view
the object.

Lab 1 - Configuring A New Tenant In
ECS

Tenant Configuration
In this lab, you will experience how a storage administrator for RainPole can configure the media department as a
tenant in ECS. Prior to performing this task, an administration account was created that provides the storage
administrator with access to provision these tenants in ECS. ECS supports access by multiple tenants, where each
tenant is defined by a namespace and the namespace has a set of configured users who can store and access objects
within the namespace.
Note: You may skip this lab (Lab 1) and Lab 2 if you prefer to explore the features in ECS with preconfigured tenants
and data.
Open Site 1
The lab is backed by two virtual instances of ECS software, simulating Boston and New York data centers in a geo-
federated environment. The lab experience is identical to provisioning a true ECS appliance.
On the desktop, double click ECS Site1 (Boston)

Browser Exception
Click Advanced
Advanced.

Add Exception
Click Proceed to 192.168.1.11 (unsafe)

(unsafe).

Login to ECS Portal at Site 1 (Boston)
Login using the following credentials:
• User Name: ecsadmin01@vlab.local

Click Login
Open the Guide
To ensure all configuration steps are performed in the right order, the storage administrator can use the configuration
guide, which is displayed on their first login to ECS.
To display the guide on subsequent logins, click the Open Guide icon.

Open the Checklist
Click Browse the checklist to access the configuration checklist.
Pre-configuration
Before a new tenant/namespace can be provisioned, a storage pool, virtual data center (VDC) and replication group
need to be configured.

In the RainPole configuration there are two VDCs available, each representing a different site - Boston (BosVDC
BosVDC) and
New York (NycVDC
NycVDC). A replication group between these sites called BosNycRG has been created. The storage
administrator needs to start by creating a namespace for the tenant.
Note: All steps appear complete in green because a tenant has been preconfigured for lab users who prefer to skip
this lab (Lab 1) and Lab 2.
You can close the checklist.

Create a Namespace
To create a new namespace, click Manage, and then click Namespace

Namespace.
Click New Namespace

Namespace.
Enter Namespace Details
In the New Namespace area, specify the following:
• In the Name field, type media

• In the User Admin field, type rainpoleadmin@vlab.local
rainpoleadmin@vlab.local. This is the namespace administrator, a valid domain
user.
• In the Replication Group list box, accept the default, BosNycRG
Scroll down, and then click Save

Save.

Namespace Created
Your new Namespace has been created.
Logout
You will now log out of the ECS portal as the storage administrator and continue configuration of the tenant by acting
as the Namespace administrator, rainpoleadmin@vlab.local.
Click logout
logout.
Access the ECS Portal as Namespace Admin
You will now access the ECS portal using the credentials of the Namespace admin domain user. You will create an
object user and provision a bucket for object storage within the media Namespace.
• User Name: rainpoleadmin@vlab.local


Click Login
Error message
An error message will be displayed. This will not affect lab functionality and can be ignored.
Close dialog
Close the dialog.
New Object User
To provide access to the ECS object store, the Namespace administrator must first create an object user with the
appropriate credentials.
Click Manage, and then click Users

Users.

Click New Object User
User.
An object user account is utilized by a user or an application to consume the object storage platform via REST API like
S3, Swift or Centera CAS.
Specify a Username and Namespace
• In the Name field, type mediauser@vlab.local

• In the Namespace list box,, accept the default, media

Click Next to Add Passwords
Passwords.
Generate a S3 Secret Key (S3 Password)
Object users need a S3 Secret Key to be able to connect to the HTTP endpoint and read/write objects. You will now
create one for the user, mediauser@vlab.local. This S3 Secret Key is distributed out of band. You will save this access
key for use in later labs.
Click Generate and Add Password to create an S3 Secret Key for mediauser@vlab.local.

Note: Your access key will differ from the one in the screen shot.
Copy Access Key
Click in the key control field. A blue outline should appear around the field to signify it is active. Triple-click in the field
to select all of the key text. Right-click on the selection and select Copy to copy the access key to clipboard.
Create a Blank Text Document to Store the Access Key
Minimize the ECS Portal. Right-click on the desktop, and select New
New, Text Document
Document. Name the file
mediauser@vlab.local.txt
mediauser@vlab.local.txt.

Store the Access Key In the Text Document
Double click to open the text file. Right-click, and paste the access key into the file. Click File and select Save
Save. Finally,
close the text document.

Provision a Bucket
Now that you have provisioned an object user and secret key, you will now provision a bucket for the object user to
consume.
Click Buckets
Buckets, and then click New Bucket
Bucket.
Enter Bucket Details
In the New Bucket area, specify the following:

• In the Name field, type images
• In the Namespace list box, select media
• In the Replication Group list box, accept the default, BosNycRG
• In the Bucket Owner field, type mediauser@vlab.local. The owner will also act as the owner of the file system
when accessing the bucket via NFS export.
Proceed to the next step to enable metadata search on the bucket and configure indexes.
Enable Metadata Search
The media unit has both a media review team and an analytics team that needs to be able to search for objects within
the bucket using metadata search criteria.
The configuration to enable metadata search has to be done by the Namespace admin using the portal. This can also
be done by a bucket owner or an admin user using the ECS management API or the S3 API to configure metadata
search and indexes at the time of bucket creation.

Scroll down to Metadata Search
Search. Click Enabled
Enabled.

Configure Metdata Search Keys
The Namespace admin must know the metadata attributes that are required to be searchable. While system metadata
attributes are available to be selected, user metadata keys need to be manually entered.
The teams in the media unit need to be able to search for data based on the following user metadata attributes:
• image-width (integer)
• image-height (integer)
• image-viewcount (integer)
• gps-latitude (decimal)
• gps-longitude (decimal)
To configure these as search keys:
• Click Add
• In the Metadata Key Type list box, select User
• In the Key Name field, type x-amz-meta-image-width
x-amz-meta-image-width. The name is already prefixed, complete the rest.
• In the Data Type list box, select integer
• Click Add

Configure Additional Search keys
Repeat the previous step for the remaining four metadata search keys:
• image-height (integer)
• image-viewcount (integer)
• gps-latitude (decimal)
• gps-longitude (decimal)
When the five keys are complete, scroll down, and then click Save
Save.

Success!
You have successfully operated as a Namespace administrator to create an object user and provision a brand new
bucket for the teams in the media unit. They will now use the object user to ingest and access data.
Proceed to the next lab to start executing the media team use cases of ingest, metadata search and NFS based access.

Lab 2 - Ingesting Data Using An S3
Application

Data Ingest into the ECS Cloud Storage Platform
The ECS platform supports ingest of data via multiple protocols such as Object (S3, Swift, CAS, Atmos), File (NFS V3),
and HDFS. This lab will allow you to experience the simplicity and efficiency of ingesting data using an S3 client
application.
The content generation team in the media wing uses an S3-based client application called Media Loader to copy
images to the ECS platform. Its configuration process is covered in this lab and is similar to that of any S3 client.
You may skip Lab 1 and Lab 2 if you prefer to explore metadata search and NFS access capabilities with a pre-
configured tenant and pre-populated data.
Launch The Media Loader Client Application
Minimize the ECS portal window, and on the desktop double-click Media Loader
Loader.

Enter details to point the client to the Media tenant's bucket
In the Media Loader, specify the following:
• The IP and Port for the Boston site are pre-populated

• In the Username field, type mediauser@vlab.local
• In the Bucket Name field, type images
• Pictures
Pictures: The number of images to be loaded, accept the default, 100
For the secret key, follow the instructions detailed in the next steps.
Secret Key (S3 Password)
Object users need a secret key to be able to connect to the HTTP endpoint and read/write objects to this bucket. You
created this in the previous lab and copied into a file on the desktop called mediauser@vlab.local.txt
mediauser@vlab.local.txt.

Open the file, highlight the secret key text, right-click and select Copy
Copy.
Paste Key
Apply the copied key into the Secret Key area in the Media Loader application.
Right-click, and select Paste

Paste.

Upload Images into the ECS cloud storage platform
Click Upload to start loading the image files using the S3 client application.
Experience how quickly image contents are copied. When the process is complete you will see the success status and
duration of the copy.
Success!
You have successfully performed object I/O to ECS Cloud Storage.
Proceed to experience how the uploaded data can be accessed through S3 based metadata search and NFS protocols.

Lab 3 - Performing Metadata Search
Using An S3 Client Application

ECS Metadata Search Media Reviewer
After completing this lab, you will have a better understanding of how to leverage the ability to search for objects
residing in ECS using metadata search.
The media review team of the media unit has an S3 client application to review the content. This lab will take you
through the type of metadata queries that can be performed efficiently by using the ECS metadata search API.
Close Media Loader
Close Media Loader as it is no longer required.
Copy Secret Key
Open the mediauser2@vlab.local.txt file located on the desktop and copy the secret key.
Open Media Reviewer
On the desktop, there are two icons to open the Media Reviewer.

Double-click Media Reviewer - Media2.bat to use a preconfigured bucket that has already been loaded with many
images.
Log into Media Reviewer application
In the Media Reviewer, specify the following:
• In the Username field, type mediauser2@vlab.local

• In the Secret Key field, right-click, and select Paste
Click Login

Ensure the correct bucket is selected for meta data search
Ensure the images2 bucket is selected.

Search Example 1: Identify geographic of pictures with a high view count
In the Minimum number of views field, type 5000
Click Search
Search.
All images in the bucket with a meta data attribute named viewcount that is greater that 5000 will be returned.

View image
Click on any of the red pushpins on the map to view more details about the selected image. Observe Number of views
is above 5000
5000.
Click Close
Close.

Search Example 2: Search within a specific geographic boundary
Next, you will not be specifying a Minimum number of views but instead will search within a selected geographic
boundary.
Delete 5000 from the Minimum number of views field, and then select Use selected area
area.

Search
On the map, click and drag to move the gray marquee down to South America. You can use the circular touch points to
cover the entire landmass of South America.
Click Search
Search.
Based on the geographic boundaries you provided on the map, four meta data search parameters were passed in the
query to ECS. Two parameters indicated that the latitude has to be between the north and south boundaries. The
other two parameters indicated that the longitude had to be between the east and west boundaries.
Click Pictures to view thumbnails of the search results.

Show Picture
On any one of the pictures, click Show

Show.

Latitude and Longitude
At the bottom of the window, observe the latitude and longitude are within the boundaries of the search you
performed.
Click Close
Close.

Logout
Click Logout
Logout, and then close the Media Reviewer.
Conclusion
This lab demonstrates how an application can query objects using meta data indexes specified on the S3 bucket. The
application in this lab used different variations of meta data search queries to find and group results.
You began with a simple search that located all images with a meta data attribute viewcount that was larger than
5000. The second search was more complex, utilizing a geographic range in which longitude and latitude needed to
be within the plot you specified.

Lab 4 - Querying Object Metadata
with Apache Spark SQL for use by
HDFS based applications

Querying object metadata with Apache Spark SQL for analytics
Introduction
The analytics wing of the media unit uses Zeppelin Notebook with SPARK SQL libraries so that they can run SQL
queries on large data sets. Apache Spark is a fast general engine for large-scale data processing.
In this lab, SPARK SQL queries are translated to ECS metadata search queries so that data can be efficiently accessed
and queries can be serviced by use of ECS metadata indexes.
Launch Zeppelin
Apache Zeppelin is a web-based notebook for data analytics. It provides an easy front end for interacting with Apache
Spark data sets.
Minimize any open windows, and then double-click Zeppelin

Zeppelin.

Navigate to Notebook
Click Bucket Metadata Search Tutorial

Tutorial.
Notebook Overview
Inside the notebook, you will see a list of paragraphs. Each paragraph can be run individually by clicking run on the
step or by clicking the run all paragraphs at the top of the notebook. In this lab, you will be running them one at a
time.

Load the ECS Search Extension
First, you will need to load the ECS search extension into Zeppelin.
Click the run icon in the paragraph with the z.load() statement. This step may take up to a minute to complete the first
time it is run.
Result
When the extension is loaded, you will see a result object res0 or res1
res1.
Configure the Connection to ECS
Next, a connection to ECS is configured. There are four parameters already provided:
• bucketName = images2
• endpoint = http://192.168.1.11:9020/
• secretKey = the shared secret key for the user, mediauser2@vlab.local.
• user = mediauser2@vlab.local

Register bucket
The last two lines in this script will register the bucket as a temporary table for use with SparkSQL. This does not load
any ECS objects into memory; it simply queries ECS for the indexed attributes on the bucket for later querying.
Run
Click the run icon in the paragraph to connect to ECS.
When complete, you will now have a DataFrame object that has columns defined for the indexed attributes in ECS.
Use Spark SQL to Select Objects
Next, you can use Spark SQL to query the objects in your bucket. In the query, you will select all of the objects in the
bucket that have at least 5000 views and up to 10000 views.
Click the run icon in the paragraph to execute the query.

Results
The query will run and display the results. Feel free to modify the SQL and try your own examples. You can quickly
execute the SQL statement by pressing Shift-Enter
Shift-Enter.
While it is outside the scope of this lab, such query results can easily be saved on ECS as an HDFS file and made
available to analytics applications.
Compute the Average View Count
The next paragraph will compute the average number of views for the images in the bucket. It uses the avg function to
compute the averages of all the results in the column. Click the run icon to execute the query.

Results
The notebook will be populated with the results of your query:
View Count Histogram
Next, you will create a histogram of the view counts. This query uses Hive operators and grouping to break the objects
down into buckets of 1000 views. Click the run icon to execute the query.

Graph the Results
A table is displayed with the number of images per each bucket of thousands of views. To view this visually, click the
bar graph icon:

Bar Graph
You can now see the same data set rendered as a bar graph. Clicking on the other icons can change the view style of
the result set.
Explore - Optional
SparkSQL is a powerful engine that can be used to analyze your data in many ways. Feel free to experiment. To create
a new paragraph, hover under the previous paragraph and click +
The results of such analysis can also be made available to Hadoop applications directly from the Zeppelin notebook or
by persisting the results as an HDFS file on ECS

%sql
To use SparkSQL in your paragraph, start the first line with %sql then begin your SELECT statement on the following
line. Press Shift-Enter to execute your query.
Close Zeppelin
When ready, close Zeppelin

Zeppelin, and then proceed to the next lab.
For More Information
• You can get more information about SparkSQL on the Apache Spark website.
• The Zeppelin notebook application is available from Apache.
• The ECS driver for Spark is available from GitHub under emcvipr/spark-ecs-s3.
• More information on HDFS capabilities in ECS is available in http://www.emc.com/campaign/ecs-hadoop/
index.htm
• This ECS demo is also available as a Docker container from DockerHub under emccorp/spark-ecs-s3.

Lab 5 - Accessing Data through NFS

Working with ECS Data via NFS and S3
After completing these steps, you will have a better understanding of how to work with data using multiple protocols
(S3 and NFS). You will first upload data via S3 browser into a bucket and then retrieve that data via NFS mount in
Windows Explorer.
This lab details the steps for how an outside marketing firm could sharing advertising media via S3 to RainPole's
advertising team whom them will access this data via NFS mount in Windows.
Logout
If you are still logged in from a previous step, click logout

logout.

Log into ECS portal as the Namespace admin
• User Name: rainpoleadmin2@vlab.local

Click Login
Error message
An error message will be displayed. This will not affect lab functionality and can be ignored.
Close dialog
Close the dialog.

Create file system enabled bucket
1. Click Manage
2. Click Buckets
3. In the Namespace list box, ensure media2 is selected
4. Click New Bucket
Bucket.
Bucket details
1. In the Name field, type rainpole-media

2. In the Namespace list box, select media2
3. In the Replication Group list box, ensure BosNycRG is selected
4. In the Bucket Owner field, type mediauser2@vlab.local
Scroll down to File System

System.

Bucket details contd.
1. Under File System

System, click Enabled
2. In the Default Bucket Group field, type DefaultBucketGroup1
3. Under Group FIle Permissions
Permissions, click Read and Write
4. Under Group Directory Permissions
Permissions, click Read and Write
Scroll down to the end of the page.
Save
Click Save to create the bucket.

Confirm creation of new bucket
Your new bucket has been successfully created.

Configure the NFS export
1. Click File
3. Click New Export
Export.
Export details

2. In the Bucket list box, select rainpole-media

3. Click Add
Add Export Host
1. In the Export Host field, type * (this will allow access from all NFS clients)

2. Under Permissions
Permissions, click Read/Write
3. Under Authentication
Authentication, click Sys
4. In the AnonUser field, type mediauser2@vlab.local
5. In the AnonGroup field, type DefaultBucketGroup1
6. In the RootSquash field, type mediauser2@vlab.local
7. Click Add

Confirm export
The new export host is displayed in the grid.
Click Save to complete the export configuration.
Verify configuration
Verify your NFS export configuration was added for the rainpole-media bucket.

Open S3 Browser
You will now configure access the the file system enabled bucket via S3 Browser. This will allow you to upload your
source image that you want to access and read back via NFS.
On the desktop, double-click S3 Browser

Browser.

Configure S3 browser to view the edited image
You are now prompted to Add New Account. Specify the following:
1. In the Account Name field, type RainPole Media

2. In the Storage Type list box, select S3 Compatible Storage
3. In the REST Endpoint field, type 192.168.1.11:9020
4. In the Access KEY ID field, type mediauser2@vlab.local
5. In the Secret Access Key field, open the file mediauser2@vlab.local.txt on the desktop. Right-click, and then
copy/paste the Secret Access Key.
6. Click Add new account

Preview the edited image in S3 browser
After the account is added, select rainpole-media

rainpole-media. There are currently no objects in the bucket.
Upload
Click Upload
Upload, and then select Upload file(s)

Select file
In the left navigation pane, click Desktop

Desktop.
In the right navigation pane, select ECS.jpg and then click Open
Open.

Confirm upload
Confirm ECS.jpg has been successfully uploaded into the rainpole-media bucket.
Mount the NFS export in Windows
You will now use a Windows client for NFS command to mount the export. This command will allow access to the
export as if it were another drive in the Windows environment.
On the taskbar, click Command Prompt

Prompt.

Mount command
Type the following command:
mount -o mtype=soft -o nolock -o sec=sys -o fileaccess=777 192.168.1.11:/media2/rainpole-media e:
Press Enter.
Type:
mount
Press Enter
Enter.
Verify the mount is listed and ready.
Open and edit a file via Windows Explorer and MS Paint
On the taskbar, click Windows Explorer

Explorer.

Select E:
Using Windows Explorer, you gain a more visual representation of the export you mounted in ECS.
In the left navigation pane, the NFS drive is mounted and named rainpole-media (\\192.168.1.11\media2) (E:)
(E:).
Click to display the contents.

Select image
You can think of this as a listing of the objects in the rainpole-media bucket.
Right-click ECS.jpg
ECS.jpg, and then select Edit
Edit.

View image
Verify the image displays correctly.
Close Paint
Paint.

Conclusion
You have now experienced the capabilities of using NFS within ECS. Furthermore, you have seen first-hand how cross-
head access can be leveraged.
You first created a new file system enabled bucket in ECS which was then referenced via NFS export configuration.
Then, you uploaded content to the bucket via the AWS S3 API using S3 Browser. To experience cross-head access, you
viewed the uploaded image by creating a mount in Windows (using the Windows NFS Client) and opening/previewing
the image.

Troubleshooting

Troubleshooting & Tips
If you encounter issues during your vLab experience, there are a couple of things to check.
Check ECS Health
If you have trouble logging in to the ECS GUI (hangs, reports bad password), you should make sure that ECS has fully
initialized. Sometimes if the vLab is busy, this can take a little longer than normal.
Launch DT Status Checker
On your desktop, double-click the icon that says ECS DT Status

Status:
Wait for ECS to Become 100% Ready
Inside the GUI, you'll see the initialization status for each Virtual Data Center (VDC). If both VDCs do not indicate
100% readiness, wait for initialization to complete and then return to your task.

Once all the DTs (directory tables) are initialized, you should see 100% in both VDCs. Note that it's normal for the
VDCs to have a different number of total DTs.
Restart Zeppelin Interpreter
If you get any strange errors with Zeppelin, you can try restarting its interpreter. First, click on the Interpreter tab at the
top:
Then click the restart button to restart the interpreter. You can then go back to your notebook and restart your
execution from the top.

Recreate NFS mount
If you experience any issues (like the message seem below) attempting to access/modify images via NFS, you can try
to recreate the NFS mount by following the steps below.

From the command prompt, unmount the NFS export by typing:
umount e:
Enter. If you're warned and asked to continue this operation, type y and press Enter
Press Enter Enter.
Type the following command to recreate the mount:
mount -o mtype=soft -o nolock -o sec=sys -o fileaccess=777 192.168.1.11:/media2/images2 e:
Press Enter
Enter. You will see a success message.
Type the following command and press Enter to verify your mount properties.
mount
Verify the mount is listed and ready. Note that the drive letter is e:

Conclusion

Conclusion
Congratulations!
RainPole's IT team has successfully extended the ECS cloud storage platform to its media and communications wing.
All of the following needs have been successfully met and the IT team is convinced that ECS is a smart and flexible
multipurpose cloud storage platform that addresses varied needs:
Ingest of image content and associated metadata by the content generation team.
Use of the metadata associated with the media images by two different teams:
• The media review team used a custom application (Media Reviewer) to search for images based on the
metadata to review the content
• The analytics team used a Zeppelin notebook to view object metadata and run Spark SQL queries to extract
data for analysis
A remote advertising team could access archived media content from their windows platform as a mounted file
system, for reference and reuse.

Additional Solution Information

Business Challenge
RainPole
RainPole’s enterprise IT offers a competitive cloud storage platform to all business units using EMC Elastic Cloud
Storage. They have achieved Terabyte to Exabyte multisite scale-out architecture, giving customers flexibility and
control to scale out according to business needs.
RainPole's Media Business
RainPole now wants to extend their cloud infrastructure to the media and communications wing. The media wing is
faced with an explosive growth of unstructured data and the need to serve modern cloud applications.
The following use cases are to be supported for the media wing:
The content generation team needs to ingest image content and its associated metadata.
Two different teams would use the metadata associated with the media images:
• The media review team has a custom application (Media Reviewer) that needs to search for images based on
the metadata to review the content.
• The analytics team wants to use Zeppelin notebook to view object metadata and run Spark SQL queries to
extract data for analysis.
A remote advertising team needs to access archived media content from their Windows platform as a mounted file
system for reference and reuse.

Solution
RainPole’s IT team have evaluated EMC ECS 2.2 and determined that its metadata search functionality and NFS V3
support can help address the media unit’s needs. Metadata search can assist in the discovery of objects based on
metadata, as ECS 2.2 has the ability to index object metadata and make it available for search. Support for NFS
enables access to object data as a file system by legacy applications requiring file access.
The ECS infrastructure gives RainPole’s IT team the ability to offer a wide range of data ingest and access options such
as Object protocols (S3, Swift), NFS, and Hadoop file system access to achieve a TCO that is 30% lower than that of
public cloud storage. ECS also provides the flexibility of being deployed as software on commodity hardware or in a
turnkey appliance form factor.

ECS
Elastic Cloud Storage (ECS) is a multi-purpose shared global storage that scales to store exabytes of both small and
large files with strong global consistency.
ECS is built from the ground up to support multiple protocols for unstructured (Object and File) workloads on a single
cloud-scale storage platform, natively speeding up storage for traditional archive services as well as modern web,
mobile, Internet of Things (IoT), and Hadoop applications. You can store, access or consume data without any
translation gateways.

Elastic Cloud Storage Benefits
ECS delivers simple storage management of globally-distributed infrastructure under a single global namespace with
anywhere access to content. It simplifies data governance and management with instant metadata search, analytics
enabled by HDFS, and built-in optimizations for speed and storage efficiency.
Building on the EMC experience for enterprise-grade features for protection, availability, encryption, authentication,
and access controls, ECS delivers a long list of ISV application and technology support. ECS provides you more control
of your data assets with enterprise-class object, file, and HDFS storage in a secure and compliant system. With the
new management, monitoring, and scripting capabilities, you can offer storage as a service within your enterprises.
ECS features a flexible software-defined architecture that is layered in such a way to promote limitless scalability. Each
layer is completely abstracted and independently scalable with high availability and no single point of failure.
ECS brings public cloud capabilities within your own datacenter and behind the corporate firewall. Leveraging COTS
(commodity off the shelf) hardware, high storage efficiency, smaller datacenter footprint, and simple management,
enterprises can benefit from public-cloud economics within your own datacenter.

Other Client Applications
Media Reviewer
A sample custom client application that runs on Windows platforms and uses the S3 API interface to extract data from
ECS.
Media Loader
A sample s3 client application that ingests image data into the ECS cloud storage platform using the S3 object
protocol.
Apache Zeppelin Spark Notepad
A sample general-purpose client application. It is a web-based notebook that enables interactive data analytics and
use of SPARK SQL on top of ECS metadata search APIs.

Experience Smart Multipurpose Shared Global Storage With ECS

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Experience Smart Multipurpose Shared Global Storage With ECS

Uploaded by

Copyright:

Available Formats

EXPERIENCE SMART

Experience Smart Multipurpose Shared Global Storage with ECS

Lab Overview ................................................................................................................................................ 6

Lab Credentials ................................................................................................................................................8

Lab Scenario ..................................................................................................................................................10

Lab 1 - Configuring A New Tenant In ECS ..................................................................................................... 13

Lab 2 - Ingesting Data Using An S3 Application ........................................................................................... 35

Lab 3 - Performing Metadata Search Using An S3 Client Application ........................................................... 40

Lab 5 - Accessing Data through NFS ........................................................................................................... 61

Additional Solution Information.................................................................................................................. 89

Other Client Applications ...............................................................................................................................94

Experience Smart Multipurpose Shared Global Storage with ECS Page 4

Experience Smart Multipurpose Shared Global Storage with ECS Page 5

Experience Smart Multipurpose Shared Global Storage with ECS Page 6

Experience Smart Multipurpose Shared Global Storage with ECS Page 7

ECS system administrator

Namespace administrator for media unit

Object User for media team

Namespace administrator for media unit

Object User for media team

Experience Smart Multipurpose Shared Global Storage with ECS Page 8

Experience Smart Multipurpose Shared Global Storage with ECS Page 9

Experience Smart Multipurpose Shared Global Storage with ECS Page 10

Lab 1: Configuring A New Tenant In ECS

Lab 2: Ingesting Data Using An S3 Application

• Connect an S3 client application to a namespace and object user

Lab 3: Performing Metadata Search Using an S3 Client Application

Lab 4: Querying Object Metadata with Apache Spark SQL

Lab 5: Accessing Data through NFS and S3

Experience Smart Multipurpose Shared Global Storage with ECS Page 11

Experience Smart Multipurpose Shared Global Storage with ECS Page 12

Experience Smart Multipurpose Shared Global Storage with ECS Page 13

On the desktop, double click ECS Site1 (Boston)

Experience Smart Multipurpose Shared Global Storage with ECS Page 14

Experience Smart Multipurpose Shared Global Storage with ECS Page 15

Click Proceed to 192.168.1.11 (unsafe)

Experience Smart Multipurpose Shared Global Storage with ECS Page 16

Login using the following credentials:

• User Name: ecsadmin01@vlab.local

Open the Guide

Experience Smart Multipurpose Shared Global Storage with ECS Page 17

Click Browse the checklist to access the configuration checklist.

Experience Smart Multipurpose Shared Global Storage with ECS Page 18

You can close the checklist.

Experience Smart Multipurpose Shared Global Storage with ECS Page 19

To create a new namespace, click Manage, and then click Namespace

Click New Namespace

Enter Namespace Details

In the New Namespace area, specify the following:

• In the Name field, type media

Experience Smart Multipurpose Shared Global Storage with ECS Page 20

Scroll down, and then click Save

Experience Smart Multipurpose Shared Global Storage with ECS Page 21

Your new Namespace has been created.

Access the ECS Portal as Namespace Admin

Login using the following credentials:

• User Name: rainpoleadmin@vlab.local

Experience Smart Multipurpose Shared Global Storage with ECS Page 22

Close the dialog.

New Object User

Click Manage, and then click Users

Experience Smart Multipurpose Shared Global Storage with ECS Page 23

Specify a Username and Namespace

• In the Name field, type mediauser@vlab.local