TSUBAME Users Guide

Global Scientific Information and Computing Center (GSIC) In Tokyo Institute of Technology 2009/05 English Ver. 4.4

Contents
Introduction........................................................................................................................................... 1. System summary ................................................................................................................................2 1.1. System image..............................................................................................................................2 1.2. On Hardware ..............................................................................................................................4 1.2.1. 1.2.2. 1.2.3. 1.2.4. 1.2.5. 1.2.6. 1.3.1. 1.3.2. Computing Servers: Sun Fire X4600 ................................................................................4 Accelerators made by ClearSpeed Technology, Inc. .........................................................5 GPU accelerators by NVIDIA Corporate ..........................................................................5 100 Gigabit Network .........................................................................................................5 Storage with a large amount of capacity ..........................................................................5 Storage with a high availability .......................................................................................5 Computing servers ............................................................................................................6 Storage with a large amount capacity ..............................................................................6

1.3. On software.................................................................................................................................6

1.4. Hardware list..............................................................................................................................7 1.5. Software list................................................................................................................................8 1.5.1. 1.5.2. 2. ISV applications ................................................................................................................8 Free applications ...............................................................................................................9

Usage environment........................................................................................................................10 2.1. Available Resources..................................................................................................................10 2.1.1. 2.1.2. 2.1.3. 2.1.4. 2.1.5. 2.1.6. 2.1.7. 2.2.1. 2.2.2. 2.2.3. Available systems for general purpose ...........................................................................10 Resource limit ..................................................................................................................11 User account ....................................................................................................................11 Disk space ........................................................................................................................12 Terminal...........................................................................................................................15 Printers ............................................................................................................................15 Backup .............................................................................................................................16 Computing nodes (Sun Fire X4600)................................................................................17 Media workstations .........................................................................................................18 Proxy server .....................................................................................................................18

2.2. Environment for each purposes ...............................................................................................17

3.

Use of the system...........................................................................................................................19 3.1. Login .........................................................................................................................................19 3.1.1. 3.1.2. 3.2.1. 3.3.1. Remote-Login...................................................................................................................19 Local log in to media workstation and log out ...............................................................20 Change password.............................................................................................................20 Environment setup script ...............................................................................................21 -i-

3.2. Password management ............................................................................................................20 3.3. Environment settings...............................................................................................................21

3.3.2. 3.3.3. 3.3.4. 3.3.5. 3.4.1. 3.4.2. 3.4.3. 3.5.1. 3.5.2. 3.5.3. 3.5.4. 3.5.5. 3.5.6. 3.5.7. 3.5.8. 3.5.9. 3.5.11. 3.6.1. 3.6.2.

Settings for the pre-installed applications .....................................................................21 Available shells ................................................................................................................21 Language Environment...................................................................................................22 Adjust Lustre setting of a directory................................................................................22 Execution of a program on an interactive node .............................................................24 X Window System............................................................................................................24 Limitations on interactive nodes ....................................................................................24 Make a job script .............................................................................................................26 Available queues..............................................................................................................27 Confirm available queues................................................................................................29 Submit a job .....................................................................................................................31 Using n1ge command ......................................................................................................31 Parallel jobs .....................................................................................................................34 Memory restriction on batch jobs ...................................................................................35 Confirm the job status.....................................................................................................39 Delete a job ......................................................................................................................40 GUI...................................................................................................................................41 Printing from a local PC..................................................................................................42 Use the printers on the media workstations, or on the interactive nodes ....................42

3.4. Using interactive nodes............................................................................................................24

3.5. Batch jobs..................................................................................................................................26

3.5.10. Confirm the results .........................................................................................................41 3.6. Printing.....................................................................................................................................42

3.7. Usage of LTO3 ..........................................................................................................................45 3.8. Proxy server..............................................................................................................................45 4. Program Languages and development environment ...................................................................46 4.1. Compilers..................................................................................................................................46 4.1.1. 4.1.2. 4.1.3. 4.1.4. 4.1.5. 4.2.1. 4.2.2. 4.2.3. 4.3.1. 4.3.2. 5. Compile and link program ..............................................................................................46 Compiler version..............................................................................................................46 Other compiler.................................................................................................................47 File input and output ......................................................................................................47 Using library....................................................................................................................48 MPI...................................................................................................................................48 OpenMP ...........................................................................................................................50 pthread.............................................................................................................................51 PGPROF...........................................................................................................................53 PGDBG.............................................................................................................................54 - ii -

4.2. Parallelization ..........................................................................................................................48

4.3. Performance measurement tools .............................................................................................53

Applications ...................................................................................................................................55

5.1. ISV applications .......................................................................................................................55 5.1.1. 5.1.2. 5.1.3. 5.1.4. 5.1.5. 5.1.6. 5.1.7. 5.1.8. 5.1.9. 5.1.11. AVS/Express PCE ............................................................................................................55 ABAQUS ..........................................................................................................................57 ABAQUS CAE .................................................................................................................58 AMBER ............................................................................................................................59 AVS Express Developer ...................................................................................................61 EnSight ............................................................................................................................62 Gaussian ..........................................................................................................................63 Gauss View ......................................................................................................................64 MATLAB ..........................................................................................................................65 Mathematica....................................................................................................................68

5.1.10. MOPAC ............................................................................................................................67 5.1.12. Molpro ..............................................................................................................................70 5.1.13. MD NASTRAN ................................................................................................................71 5.1.14. MD PATRAN....................................................................................................................72 5.1.15. SAS...................................................................................................................................72 5.1.16. Maple ...............................................................................................................................73 5.1.17. LS-DYNA .........................................................................................................................74 5.1.18. FLUENT ..........................................................................................................................75 5.2. ISV applications (On Windows terminal)................................................................................76 5.2.1. 5.2.2. 5.2.3. 5.3.1. 5.3.2. 5.3.3. 5.3.4. 5.3.5. 5.4.1. 5.4.2. 5.4.3. 5.4.4. 5.4.5. 5.4.6. 5.5.1. 5.5.2. 5.5.3. Materials Explorer ..........................................................................................................76 Materials Studio ..............................................................................................................76 Discovery Studio Modeling..............................................................................................77 To confirm usage of FLEXlm...........................................................................................78 To confirm the usage of LmServer ..................................................................................81 To confirm the usage of slim ...........................................................................................81 To confirm the usage of MathLM....................................................................................82 Usage of Lsdyna ..............................................................................................................82 GROMACS .......................................................................................................................82 POV-Ray...........................................................................................................................82 Tinker...............................................................................................................................83 UTChem...........................................................................................................................83 NWChem..........................................................................................................................83 Gamess.............................................................................................................................84 The usage with program sources ....................................................................................86 The usage without program sources...............................................................................86 Making Clear Speed execution module by SDK.............................................................87 - iii-

5.3. Confirm the status of license of ISV Applications...................................................................78

5.4. Free applications ......................................................................................................................82

5.5. ClearSpeed................................................................................................................................86

5.5.4. 5.5.5. 6. 7. 8. 9.

Use ClearSeed on batch nodes ........................................................................................87 isv application with ClearSpeed .....................................................................................88

5.6. TESLA.......................................................................................................................................89 MAIL ..............................................................................................................................................90 Application for the system.............................................................................................................91 Question and answer ....................................................................................................................92 English Version History.................................................................................................................93

- iv -

Introduction
This document is a users guide to TSUBAME() that the Global Scientific Information and Computing Center (GSIC) in Tokyo Institute of Technology (Titech) introduced. The TSUBAME system is a super grid computing system mainly on the Sun Fire X4600 Clusters made by Sun Microsystems inc., with (1) the total peak performance : 85TFLOPS (2) the storage with large amount of capacity (3) the network with high bandwidth OS: Linux Designed for the users of the TSUBAME system, the current guide requires some basic knowledge on Unix/Linux. Use of Character-based User Interface (CUI) is essential. Main contents as follows. An introduction / summary of TSUBAME Use environment of the system (resources) Usage of the system (resources) Through TSUBAME, we are convinced that we can contribute to research activities of users more. 2006.3.31 NEC

() About the name of the TSUBAME system The name of the system came from a public subscription on introducing the system, and the name TSUBAME presented by Mr. Satoi Ogawa was accepted. technology. The naming origin is as follows, 1. The new grid system can achieve its highest performance by putting the power of all the nodes together, and a higher level may be achieved so as to some new values may be created while the students and researches locating in the campus discuss it and communicate each other. By the way, a bird has two wings, and it can not fly only by using one wing. In this way, the cooperation is like as two wings of a kind of bird swallow (TSUBAME). 2. 3. 4. The system has the highest performance in this spring, which is similar to that the swallow (TSUBAME) is a kind of smart bird flying in early spring. With a dream to teach youths, the young swallows (TSUBAME) grow up to leaders in the field of numerical computation from nest of the present super computer. The TSUBAME is also designed as a symbol mark of Tokyo Institute of Technology. 2006.4 GSIC - Mr. Satoi Ogawa is a master graduate student majored in the nuclear engineering, in the graduate school of science and

1. System summary
1.1. System image The TSUBAME system consists of computing servers, storage servers and some other support devices. One Core is regarded as one CPU in the computing servers SUN X4600.

- 2-

- 3-

1.2. On Hardware 1.2.1. Computing Servers: Sun Fire X4600 There are 8 AMD dual core Opteron processors (total 16 CPU cores) in a Sun Fire X4600 serverHereafter, a CPU core is regarded as a CPU.

The 657 Computing Servers consist of three types of machines Type A B C Number of Nodes 639 16 2 CPU Clock 2.4GHz 2.6GHz 2.6GHz Memory/node 32GB 64GB 128GB

Batch nodes Interactive nodes Type A6 nodes Type A633 nodesType B16 nodes, Type C2 nodes

InfiniBand

Gigabit Ethernet
Storage Thumper Total capacityabout 1PB /work iStorage Total capacityabout 96TB /home

All the computing servers are connected with InfiniBand Interface, and the Storage devices are connected to all the operation servers with InfiniBand and Gigabit Ethernet

-4-

1.2.2.

Accelerators made by ClearSpeed Technology, Inc.

Each of all the 657 computing nodes is equipped with a Dual CSX600 PCI-X Board in which one chip has a 50 GFLOPS capacity.

1.2.3.

GPU accelerators by NVIDIA Corporate

The total 170 operation accelerator racks TESLA are arranged to the clusters on TSUBAME to process matrix operation, in which each rack is equipped with four GPUs made by NVIDIA corporate. 1.2.4. 100 Gigabit Network The Network with a total bandwidth 1400Gbps(double-sided) is constructed with Voltaire InfiniBand Interfaces The SuperTitanet Network within the Titech Campus, SuperSINET, and JGN are connected with 10GigabitEthernet Interfaces.

1.2.5.

Storage with a large amount of capacity

The storage is composed of 42 Thumpers, which has a total capacity of about 1PB

1.2.6.

Storage with a high availability

The storage is composed of 2 Express5800/120Rh-2 servers and one iStorage S1800AT storage device. The total capacity is about 96 TB

-5-

1.3. On software 1.3.1. Computing servers (1) Operating system SUSE LINUX Enterprise Server 10 SP 2 The Operating system of the computing servers is SUSE LINUX Enterprise Sever 10 which is one of Linux Distributions widely used in the all worldThe Linux kernel 2.6 is adopted in the OS SUSE LINUX Enterprise Server 10 is a system with correspondence to high performance computing and high security (2) Compilers, Debugger, and Development Tool PGI Cluster Development KitPGI CDK The PGI CDK is installed on the computing servers. PGI CDK includes the following main functions. PGI compiler (Fortran, C, C++) with OpenMPMPI parallel library A debugger and PROF for PC clusters ScaLAPACK library BLASLAPACK library Large tutorials and many sample programs on HPF OpenMPMPI (3) Voltaire MPI library The Voltaire MPI library can be used as an interface library where it controls Infiniband between nodes made by the Voltaire Inc. And the MPI library supports MPI 1.2. (4) Applications Many ISV applications and some free ones are available on the computing servers. Please refer to the section 1.5 and the chapter 5 for details. 1.3.2. Storage with a large amount capacity

Lustre parallel file system is constructed with several Thumper servers. Lustre is software that demonstrates an excellent scalability on both sides of capacity and the access performance, and can construct a large-scale, parallel file system easily. A Virtual file system with a large amount of capacity constructed with several Thumper nodes by using Lustre is mounted to the directory /work.

-6-

1.4. Hardware list

Machine Sun Fire X4600 Type A Num. Hardware components Dual Core AMD Opteron 880 2.4GHz x 8 Memory32GB HDD built-in36GB2RAID1 Dual Core AMD Opteron 885 2.6GHz x 8 Memory64GB HDD built-in36GB2RAID1 Dual Core AMD Opteron 8218 2.6GHz x 8 Memory128GB HDD built-in72GB2RAID1 Dual CSX600PCI-X Board 50GFLOPS NVIDIA TESLA GPU X4 / 16GB DDR3 48 SATA disk drivers with 500GB Volume:24TB [Servers] 64bit Intel Xeon 3.80GHz 2 Memory:6GB HDD built-in 36GB2RAID1 [Storage devices] 240 SATA disk drivers with 400GB Memory:16GB Total volume:96TB Intel XeonTM 3.20GHz 1 Memory:1GB HDD built-in:300GB2RAID1 3.5 inch Floppy disk drivers CD-ROM LTO3 device (backup) A3 printer 1067mm(B0) Intel XeonTM 3.80GHz 2 Memory:1GB HDD built-in:146.5GB2RAID1 Intel XeonTM 3.80GHz 1 Memory:1GB HDD built-in146.5GB2RAID1 Intel PentiumD 3.20GHz 1 Memory:512MB HDD built-in73.2GB2RAID1 Intel XeonTM 3.20DGHz 1 Memory:1GB HDD built-in300GB2RAID1 Dual Core AMD Opteron 275 2.2GHz x 2 Memory:16GB HDD built-in:36GB2RAID1

639

Computing servers

Type B

16

Type C All the nodes TESLA S1070 Storage Sun Thumper NEC Express5800/120Rh-2 Storage with availability high NEC iStorage S1800AT

170 42

Media workstations

NEC Express5800/120Lh

Color PS printer B0 color PS printer WWW server

NEC Color MultiWriter 9800C HP DesignJet 5500ps NEC Express5800/120Rf-1 NEC Express5800/120Rf-1 NEC Express5800/110Rg-1 NEC Express5800/120Lh

2 2 2

DNS server Authority system CARALDAP server Management WS Login node N1GE master LDAP,License,Backup, NFS/DB server Computing server for special applications

Sun Fire X4100

11

-7-

1.5. Software list 1.5.1. ISV applications Application

PGI CDK Intel Compiler IMSLFortran & C AVS/Express PCE ABAQUS Standard Version 6.1-2/7.2-4 10.1.011/11.0.074 6.0/7.0 7.0.1/7.1.1/7.2 6.5.6/6.7.1/6.8.4 Introduction Compilers, Debugger, and Development Kit Intel compiler(32/64bit) Math Library for Fortran & C A parallel large-scale visual software General-purpose nonlinear finite element analysis (FEA) programs for structural analysis General-purpose nonlinear finite element analysis (FEA) programs for structural analysis ABAQUS modeling/Visualization environment Finite element analysis (FEA) software for engineering structural analysis Pre-Post tools for integration of CAD and CAE General-purpose electronic structure programs Graphical interface of Gaussian 03 Parallel programming tool Ab-initio programs for molecular electronic structure calculations Programs for quantum Chemistry calculations A package of programs for molecular dynamics simulations of proteins and nucleic acids A program for molecular dynamics calculations Modeling and simulation product suite for materials A protein modeling and simulations environment General-purpose visualization environment Visualization environment for scientific/engineering data A analytics program for Statistics A general computing system for numerical calculation and equation operation An integrated technical computing environment that combines numeric computation and graphics Mathematical and analytical software Structural analysis Hydrodynamics computing

ABAQUS Explicit ABAQUS/CAE MD NASTRAN MD PATRAN Gaussian GaussView Linda(for Gaussian) Molpro MOPAC AMBER Materials Explorer Materials Studio Discovery Studio AVS/Express Developer EnSight SAS Mathematica

6.5.6/6.7.1/6.8.4 6.5.6/6.7.1/6.8.4 R2.1 R2.1 03 Rev.D02/C02/E01 3.0/4.1 7.1/7.2 2002.6 2006 V1 8.0/9.0/10/10CS 4.0 4.0/4.1/4.3/4.4 1.5/1.6/1.7/2.0/2.1 7.0.1/7.1.1/7.2 8.0.7(k)/8.2.6(c) 9.1.3 5.2/6.0/7.0 7.1(R14SP3)/7.2(R2006a) 7.5(R2007b)/7.8(R2009a) 11/12 971R3 6.3.26

MATLAB Maple LS-DYNA FLUENT

-8-

1.5.2.

Free applications Application NWChem UTChem Tinker GROMACS POV-Ray GAMESS ImageMagick acml fftw j2sdk mpich2 Openmpi Version 4.7 2004-a 4.2 3.3/3.3.1 3.6.1/3.5C 22 FEB 2006 (R5) 24 MAR 2007 (R3) 11 Apr 2008 (R1) 6.2.7 3.0.2/4.0.11 3.1.2 1.5.0 1.4.2 1.0.4 1.1.a2 Introduction Programs for computing Chemistry, analysis of crystal structure A package of programs for Quantum Chemistry A molecular modeling package for molecular mechanics and dynamics A versatile package to perform molecular dynamics 3DCG software An ab-initio quantum chemistry package A software suite creating, editing, and composing bitmap images. AMD Core Math Library FFT subroutine in C Java Development Kit MPI2 driver MPI2 driver

-9-

2. Usage environment
2.1. Available Resources 2.1.1. Available systems for general purpose The following servers and storage devices are available. (1) Servers
Login/no Hostname login (login1,login2 ) tgg075017 tgg075022 tgg072220-221 tgg075053-055 tgg072001-118 tgg072121-238 tgg073001-118 tgg073121-238 tgg074001-116 tgg074161-162 tgg075001-016 tgg075042-055 tgg074117-118 service Login nodes Interactive computing nodes/AP CS debug nodes Tesla debug nodes OS SLES9 Num. 2 CPU (Core) 8 Comment IP address 172.17.75.161 172.17.75.162 Automatically log in from login nodes Automatically log in from login nodes Automatically log in from login nodes

SLES10 SLES10 SLES10

6 2 3

96 32 48

Batch nodes/AP

SLES10

620

9920

Used by N1 Grid Engine

sles9 nodes Computing server for special applications Media Proxy Machine

SLES9

32

tgg075181

RHEL4

Type(use) home area

media-o media-s proxy-o

RHEL4 RHEL4

2 1 Volume 96TB 1PB

4 1

auto log in from login Computing server for Materials Studio 4.0,4.1, Discovery Studio 1.5/1.6/1.7/2.0 LTO3 Print out

(2) StorageOthers Comment Use it as home space on all the nodes Use it as working space on all the computing nodes Print from media-o Print from media-s Print from media-o Print from media-s

NEC/iStorage S1800AT Sun/Thumper NEC/ ColorMultiWriter9800C

working area

Color PS Printers

B0 color PS Printers

HP/Designjet5500ps

Ookayama: prn1-o Suzukakedai: prn1-s Ookayama: prn0-o Suzukakedai: prn0-s

(1) Please use an appropriate system according to your purpose. (2) The change in IP address and service might be executed. So please choose the use of each system not IP address but the host name. - 10 -

2.1.2.

Resource limit The resource limitation of an interactive node includes the limitation in cpu time, size of using memory and the number of processes of MPI jobs now. The batch nodes are limited depending on queue type. Please refer to the chapter 3 for details.

(1) Computing nodes at present (2009/4/8)

(2) Storage The resource to each storage area is limited on April, 2009. Home space is limited within 300 GB and work space is limited within 1TB for one user. Please refer to the chapter 3 for details. 2.1.3. User account The information on user ID and password is managed in a unitary mode by using LDAP. All the machines in the TSUBAME system share the identical user ID, password, home directory, and login shell. Any change on a machine will take effect in all the machines. Main user information is listed as follows, User ID password login shell home directory not to be changed to be changed to be changed not to be changed

- 11 -

2.1.4.

Disk space

(1) Available disk space The available disk space is listed as follows at present (2008/11). Note that some changes can occur according to utilization condition. directory /home0 /home1 /home2 /home3 /home4 /home5 /home6 /home7 /home_bk0 /work /work2 /work_2008 /scrbes1 /scrbes2 /scrsla1 /scrsla2 /scrsla3 /depot1 /depot2 /archive volume 7.4TB 7.4TB 7.4TB 7.4TB 7.4TB 4.4TB 5.6TB 2.4TB 1.9TB 73TB 73TB 59.5TB 11.2TB 11.2TB 11.2TB 11.2TB 11.2TB 20TB 59.5TB Comment

user home directory

directory for backup directory for working space directory for working space directory for working(2008) space(read only) Gaussian Scratch directory (for the queues bes1 and novice) Gaussian Scratch directory (for the queue bes2) Gaussian Scratch directory (for the queue sla1,inno1,pinno2) Gaussian Scratch directory (for the queue sla2,inno2,pinno2) Gaussian Scratch directory (for the queue sla3,inno3,pinno3, RAM64GB,RAM128GB) For temporary data data archive space

(2) Home directories /home0/home7 The home directories are shared by all the machines though NFS. Files created by a machine are available on all the machines without transferring them. The home directory of a user has the following structure. /home/usr?/userID (3) Working directory /work, /work2 The working area composed of Lustre file system and is mounted to the directory /work and shared by all the computing nodes. Files created on a machine are also available on all the computing nodes without transferring them. You may create your directory under the /work by typing the following command to utilize the work space. To utilize the space under the directory /work/ $ mkdir /work/userID - 12 -

Note This directory is NOT the one for keeping your files for a long time. You should make use of this directory with the purpose of outputting the results of MPI programs or programs which output large data The directory is NOT backed up, please back up your important files in your chargeThe /work space is limited to 1TB per user. (4) Backup directory /home_bk0 The directory /home_bk0 are backup one. You may create your directory under them by typing the following command to utilize the backup space To utilize the space under the directory /home_bk0 $ mkdir /home_bk0/userID

(5) Scratch directories /scr* The directories /scr* are used to store scratch files created by Gaussian jobs. Different scratch directories will be used according to the execution nodes. An appropriate scratch directory is set by the batch system while a Gaussian batch job is submitted with n1ge command. It is not necessary to set GAUSS_SCRDIR variables for an n1ge Gaussian job. The directories /scrsla1, /scrsla2, /scrbes1, /scrbes2 are accessible from the interactive nodes tgg075017~tgg075022. The environmental variable TMPSCR_DIR and the corresponding directory will be automatically set up while a job begins to run. You can write tempronly/scratch files to the directory if you appoint scratch directory to $TMPSCR_DIR. And the directory name of the scratch space has the following form, /scrXXX/$JOB_ID.$TASK_ID.$N1GE_CELL.$QUEUE Where, the XXX is one of {bes1|bes2|sla1|sla2|sla3} The scratch file will be saved one week after your job is completed as like the Gaussian scratch files from the H21 fysical year. You can copy/move the scratch file made by your program to /work or the other space whithin one week if it is necessary. An example is shown as follows, [the contents of a sample script]
$ cat chk_scr.sh #!/bin/sh echo "$TMPSCR_DIR[$TMPSCR_DIR]" echo "ls -l $TMPSCR_DIR" ls -l $TMPSCR_DIR

[submit it with n1ge]

$ n1ge -g 4S080138 -q RAM64GB ./chk_scr.sh *n1ge> Checking n1ge options.... *n1ge> Checking about 4S080138 group you specified.... *n1ge> < Premium Level 1 > *n1ge> RUNTIME of this job will be set to 30 minutes. *n1ge> You should specify it by yourself when you submit jobs as SLA, INNO, PINNO type group.

- 13 -

*n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> Creating qsub options.... *n1ge> Submitting Job to Cluster...... Your job 228850 ("OTHERS") has been submitted

[Result file]
$ cat OTHERS.o228850 [Starting Job 228850 on tgg075008] $TMPSCR_DIR[/gau/228850.undefined.TITECH_SLA.RAM64GB] ls -l $TMPSCR_DIR total 0 [Ending Job 228850] EXIT_CODE:0

Note The directories are labeled /scr* on system, so please use /scr* rather than /gau* if possible, although the /scr* and /gau* are linked to the same physical space and the old name "/gau*" is often shown when run "df -k" or the other commands. (6) Data library /archive The data library area composed of Lustre file system and is mounted to the directory /archive and shared by all the TSUBAME nodes. Files created on a machine are also available on all the computing nodes without transferring them. You may create your directory under the /archive like as under the /work. The /archive space is limited to 2TB per user. This directory is NOT the one for keeping your files for a long time. And the directory is NOT backed up, please back up your important files in your charge (7) Limit The resource to each storage area is limited on April 8, 2009 as follows. And, the users who are rearranging their data have not been limited. We will limit storage usage to all the users this August. Please discuss it with staffs in the section of computing system, if you want to use more disk space. space home work work2 archive Limitation of storage space soft limits hard limits 300GB 303GB 1TB 1TB 1TB 1TB 2TB 2TB time limits 1 week 1 week 1 week 1 week

(For archive, the limit is going to be relaxed with introduction tape library) Files can not be created or written to disk if the hard limits are exceeded, so that your jobs will be stopped sometimes. In addition, if the soft limits are exceeded and time limit is also exceeded, files can not yet be created or written as like the hard limits are exceeded. Please - 14 -

reduce your disk usage if you cant create files due to exceeding quota limits. Usage confirmation for home space $ quota -v

for work space $ lfs quota /work

2.1.5.

Terminal

The media-o or media-s is ready for a terminal in the Ookayama campus or in the Suzukakedai campus, respectively. The media workstations share the same account information as the TSUBAME system. LTO3 is available on the media workstations for backup. In addition, the printers are available on the media workstations. Hostname media-o media-s Location Ookayama Suzukakedai OS RHEL4 RHEL4 Use LTO3 print to the printers at Ookayama LTO3 print to the printers at Suzukakedai

2.1.6.

Printers Color PS printers Machine model NEC ColorMultiWriter9800C Printer prn1-o prn1-s Location Ookayama Suzukakedai available on the following nodes media-o tgg075017 tgg075022 media-s tgg075017 tgg075022 available media-o media-s

The following printers are available.

B0 color PS printers Machine model HP Designjet5500ps Printer prn0-o prn0-s Location Ookayama Suzukakedai

The main characters of the printers are listed as follows (1) Postscript color printers Colored Japanese PS printers Paper size:A3, B4, A4, B5, A5, letter, any size File format: Print speed: single-sided color A4 31 P/min double-sided A4 17 P/min : single-sided monochrome A4 40 P/min double-sided morocco A4 22 P/min Resolution:9600dpi600dpi 1200dpi1200dpi - 15 -

(2) B0 color PS printers dye Ink coated paper Highest speed normal quality high quality highest quality 2.1.7. Backup 52.86 /hr 17.56 /hr 9.29 /hr 9.29 /hr Photo paper2 13.66 /hr 9.29 /hr 7.06 /hr 5.57 /hr UV ink thick coated paper Photo Paper2 26.76 /hr 9.29 /hr 5.57 /hr 13.66 /hr 9.29 /hr 7.06 /hr 5.57 /hr

Users would back up their data by themselves with the following methods, (1) Using the LTO3 on the media workstations The commands tar and cpio are available having logged in to the media workstations. (2) The files are transferred by using scp, sftp, or ftp. In the login nodes, mount does each directory (/home, /work). And, because scp, sftp, and the ftp service operate in this machine, the file forwarding can be done to own PC and storage. You can't access the login nodes using ftp from outside of TITECH, so you have to use sftp/scp to access media-o, or login-gw to transfer your files). (3) Copy it onto other areas on the disk. /home_bk0 is used as system backup space on the TSUBAME system.

- 16 -

2.2. Environment for each purposes The environments for use of machines with different purposes are described as follows. 2.2.1. Computing nodes (Sun Fire X4600) The nodes are divided to two parts according their uses, (1) Interactive jobssmall scale jobs Test jobs(total processes are less than 4, memory/process is less tan 4GB, and execution time is shorter than 30 minutes. Compile Use of floating acceleration (ClearSpeed) MPI parallel jobs Use of floating acceleration (ClearSpeed) Execution of applications The available applications are listed in the following tables. ISV applications PGI CDK IMSL (Fortran & C) AVS/Express PCE ABAQUS Standard ABAQUS Explicit ABAQUS/CAE MD NASTRAN MD PATRAN Gaussian Gauss View Linda Molpro MOPAC AMBER Materials Explorer Materials Studio Discovery Studio Modeling AVS/Express Developer EnSight SAS Mathematica MATLAB Maple LS-DYBA FLUENT Free applications NWChem UTChem Tinker GROMACS POV-Ray Gamess - 17 -

(2) Batch jobs (large scale jobs)

(3) Execution of ISV applications and free ones

2.2.2.

Media workstations

The following tasks can be done (1) To use LTO device. (2) To print out to the printers of GSIC. (3) To convert Image files 2.2.3. Proxy server

A webpage of the Internet can be referred to by setting the following proxy. Hostname IP address Port proxy-o.cc.titech.ac.jp 131.112.14.2 8080

- 18 -

3. Use of the system

3.1. Login 3.1.1. Remote-Login Use ssh command to log in TSUBAME system, and you can log in the following machines. (1) login node (login.cc.titech.ac.jp) (2) media workstations (media-o, media-s) (3) login node from outside of Titech (login-gw.cc.titech.ac.jp) The notice regarding password You would be required to change your password when you log in to the TSUBAME system for the first time. You have to change password when the password expires in 24 weeks (168 days). The login process would be automatically forwarded to an interactive node when logging in to Login node. You would have to log in with additional options if you want to execute SAS, MOPAC, Clear Speed debugging on an exclusive interactive node. Example 1: Remote-login without X11 forwarding $ ssh userID@login.cc.titech.ac.jp $ ssh -l userID login.cc.titech.ac.jp Example 2: Remote-login with X11 forwarding -Y: You should use the Y(-X) option if you want to execute X11 applications. $ ssh -Y userID@login.cc.titech.ac.jp Example 3: log in a special interactive node (sles9), $ ssh -t -Y userID@login.cc.titech.ac.jp sles9 Example 4: log in a special interactive node to execute SAS, $ ssh -t -Y userID@login.cc.titech.ac.jp sas Example 5: log in a special interactive node to execute MOPAC, $ ssh -t userID@login.cc.titech.ac.jp mopac Example 6: log in a special interactive node to execute Clear Speed debugging, $ ssh -t userID@login.cc.titech.ac.jp csdebug Example 7: log in the media workstation in Suzukakedai $ ssh userID@media-s.cc.titech.ac.jp Example 8: log in a tesladebug node $ ssh t userID@login.cc.titech.ac.jp tesladebug

- 19 -

3.1.2.

Local log in to media workstation and log out Please enter your userID and password.

(1) Login

(2) Log out Press the [Log out] submenu in the [Action/Session] menu on the bottom/top of the screen.

3.2. Password management 3.2.1. Change password You may change your password having logged in an interactive node or media workstations, $ passwd Changing password for user user0001. Enter login(LDAP) password: old password New UNIX password: new password Retype new UNIX password: new password LDAP password information changed for user0001 passwd: all authentication tokens updated successfully.

The password had been changed successfully if the message all authentication tokens updated successfully are displayed.

- 20 -

3.3. Environment settings 3.3.1. Environment setup script It is not necessary to set your environment by yourself, generally. You may edit the following startup file according to your login shell if you need a customized environment. login shell bash cshtcsh sh script on environment setting ~/.bashrc ~/.cshrc ~/.profile (1) environment setting

You have to set your environment carefully in order not to remove some necessary settings for pre-installed applications when you edit your file. (2) display your settings You may confirm your environment settings by typing the following commands.

$ printenv $ echo $variable

display all the defined environment variables display a defined environment variable

3.3.2.

Settings for the pre-installed applications

It is not necessary to set them by yourself, generally. 3.3.3. Available shells

The default shell is bash on the TSUBAME system. You may confirm your shell variables with the echo $SHELL command. In the following example, BASH is being used. $ echo $SHELL /bin/bash You may change your default shell to shtcshor csh if you want. The available shells and their full paths are listed in the following table. Shell sh tcsh bash csh PATH /bin/sh /bin/tcsh /bin/bash /bin/csh

- 21 -

3.3.4.

Language Environment

The default LOCALE is C. No need to set LANG variable if you use English as your default language. You may set your LOCALE by setting LANG variable if you want to use other language on the TSUBAME system. For example, the available Japanese Locales are listed as, LOCALE ja_JP.eucjp ja_JP.sjis ja_JP.utf8 Code set Japanese EUC Japanese Shift-JIS Japanese UTF-8

You can display the current settings on LOCALE by typing $ locale LANG=C

You may get all the LOCALE availables on the TSUBAME system by, $ locale -a C POSIX For example, you can set the LOCALE to UTF-8 Japanese by the following command if you want to display file in Japanese, $ export LANG=ja_JP.utf8

You may set LOCALE to C if you want use English environment, $ export LANG=C

3.3.5.

Adjust Lustre setting of a directory

The /work that is constructed with lustre servers consists of a group of OSTs with capacity of 1.7TB, and it offers total about 52.6TB available space now. A file larger than 1.7TB can not be created because one file can use only one OST under the present setting. However, you can change the stripe setting of a directory with the following method. [The setting method] You can set and show the stripe status of a directory by using the subcommand "setstripe" and "getstripe" of the command "/usr/bin/lfs" respectively. - 22 -

$ lfs help setstripe $ lfs help getstripe [Setting Example] The stripe count of the directory /work/testuser/stripe is set to 3, so the directory can use 3 OSTs simultaneously. As a result, a file with the maximum size 5.1TB(1.7TB*3) can be created on it. (1) make a directory on Lustre FS $ mkdir -p /work/testuser/stripe (2) set stripe of the directory $ lfs setstripe /work/testuser/stripe 0 -1 3 => size of stripe size => stripe start => stripe count 3 (the default size is1MB the start OST is decided by Lustre server default (the count is 3)

(3) show stripe status of the directory $ lfs getstripe /work/testuser/stripe OBDS: 0: tgt075121-ost1_UUID ACTIVE 1: tgt075122-ost1_UUID ACTIVE : --omit-: 29: tgt075125-ost7_UUID ACTIVE /work/testuser/stripe default stripe_count: 3 stripe_size: 0 stripe_offset: -1 (4) make a test file on the directory which is set stripe $ dd if=/dev/zero of=/work/testuser/stripe/zerofile bs=1024k count=10 10+0 records in 10+0 records out (5) show which OSTs the file is stored $ lfs getstripe /work/testuser/stripe/zerofile OBDS: 0: tgt075121-ost1_UUID ACTIVE 1: tgt075122-ost1_UUID ACTIVE : --omit-: 29: tgt075125-ost7_UUID ACTIVE /work/testuser/stripe/zerofile obdidx objid objid 15 2294545 0x230311 16 2272520 0x22ad08 17 2307219 0x233493 => the data is stored in the 3 OSTs (15,16,17)

group 0 0 0

- 23 -

3.4.

Using interactive nodes

It is same as the usage of a general Unix machine. The login process would be automatically forwarded to an interactive node. Please refer to the section 3.1 about login process. 3.4.1. Execution of a program on an interactive node

You may compile, link programs, and run a test job on an interactive node. A factual computation should be executed on batch nodes. Please refer to the section 3.5 for details. $ pgcc myprog.c $ ./a.out

3.4.2.

X Window System

X Window application may be executed if you log in the system with X11 forwarding. For example, $ /usr/X11R6/bin/xclock & An xclock window will be displayed in the X11 terminal. * When gnome-terminal is launched on TSUBAME, its helper process gnome-pty-helper remains with locking a file wmp after logoff of the user. As it affects other processes using wmp, gnome-pty-helper in zombie state is to be deleted by GSIC when found. 3.4.3. Limitations on interactive nodes Some restrictions on interactive nodes for resource usage are adopted. The limitation of cpu time is 30 minutes per process. The limitation of memory is 4 GB per process. The command mpirun can start a job within 4 CPUS. It is possible to modify the limitations according to the use state. For details, please refer to

http://www.gsic.titech.ac.jp/%7eccwww/tgc/tgcreminder_e.pdf

- 24 -

- 25 -

3.5. Batch jobs You may execute your application either on batch nodes or on interactive nodes. A factual computation costing many system resources (memory, CPU time, etc.) should be executed on batch nodes to make use of the TSUBAME system efficiently and decrease the load of the interactive nodes. The usage of the batch job system is described as following steps, (1) Make a job script (2) Submit a job (3) Confirm the job status (4) Confirm the result 3.5.1. Make a job script #!/bin/sh # # sample jobscript # cd $HOME/test ./myprog < input_data Then change its permission to make it executable. $ chmod a+x myjob.sh

For example, contents of myjob.sh script

- 26 -

3.5.2.

Available queues Now available queues are following.

2009/4/8

Queue novice sles9 sas mopac bes1 bes2 cs1 cs2 bes1tes2 bes2tes2 tsubasa cotsubame sla1 sla2 sla3 sla1tes2 sla2tes2 sla3tes3 RAM64GB RAM128GB hpc1/hpc2 hpc4 inno1 inno2 inno3 pinno1 pinno2 pinno3

Number of nodes 11 2 1 3 234 234 87 88 14 266

Comment 2.4GHz, 32GB/node, 32 CPUs/user old settings with SLES9 For SAS app. (share with sles9 nodes) For MOPAC app. (share with bes2 nodes) 2.4GHz, 32GB/node Maximum 1888 CPUs share with bes, limited to 32 nodes/job now share with bes adjusting for general users 2.66GHz Xeon 8cores, 24GB/node Maximum 112 CPUs 2.4GHz, 32GB/node Maximum 1888 or 960 CPUs share with sla nodes 2.6GHz, 64GB/node Maximum 256 CPUs 2.6GHz, 128GB/node Maximum 32 CPUs Maximum 1040 CPUs. (pre-order system) adjusting 2.4GHz, 32GB/node Maximum 1888 or 960 CPUs 2.4GHz, 32GB/node Maximum 1888 or 960 CPUs

Service Free-of-charge serivces for porting

Best-effort services (fixed charge)

132 16 2 65 33 266

Service Level Agreement services (charge for time)

266

Service Level Agreement services for external users (charge for time)

Allocation of nodes for best-effort services and serivce level agreement services may change according to the service utilization. <=32CPUs/user for all the free batch queues, i.e. novice queue (sas queue excluded) Please send mail to soudan@cc.titech.ac.jp, if you hope to change the cs1 and cs2 limit. Please refer to TSUBASA users guide to use the tsubasa queue. sles9 nodes are limited to 7 days and maximum of 4GB memory (as interactive nodes) for a job. Please refer to coTSUBAME users guide to use the cotsubame queue. hpc1 and hpc2 queues may be devided for pre-order. hpc4 queue is going to be available in this summer. Batch job assignment and charge - 27 -

- Free-of-Charge services (mandatory service for all the TSUBAME users, no charge, = 32CPUs/user, and 1hours/job for novice queues) - Best-effort services (charge a fee, a job must complete within 7 days, and the parallel number specified by you must be less than the available CPUs of the BES group) - Service Level Agree services (charge a fee based on execution time*nodes, and limits in available time must be noticed) You must be a member of a corresponding group to use a charge queue. For details, please refer to http://www.gsic.titech.ac.jp/%7Eccwww/index.html Premium service The practice frame, priority and waiting schedule of jobs can be adjusted by using the Premium service. For details, please refer to the webpage on research computing system of GSIC. http://www.gsic.titech.ac.jp/%7Eccwww/index.html Three N1GE groups manage these queues as folows. free,besabbreviation : free novice, bes1/cs1, bes2/cs2, mopac int interactive, sas, csdebug, tesladebug sla sla1, sla2, sla3, RAM64GB, RAM128GB, inno1, inno2, inno3, pinno1, pinno2, pinno3 You may show all the available queues by typing the qlist command. Adding the command -a can display the N1GE attribute information of each queue. $ qlist a RAM128GB RAM64GB bes1 bes1tes2 bes2 bes2tes2 cs1 cs2 csdebug hpc1 hpc2 inno1 inno2 inno3 interactive interactive2 mopac novice pinno1
sla sla free,bes free,bes free,bes free,bes free,bes free,bes int sla sla sla sla sla int int free,bes free,bes sla

- 28 -

pinno2 pinno3 sas sla1 sla1tes2 sla2 sla2tes2 sla3 sla3tes2 sles9 tesladebug tsubasa tsubasa_int

sla sla int sla sla sla sla sla sla int int gcoe int

Note that the interactive, sas, csdebug, sunstudio, and tesladebug queues are for interactive usage so that they are not able to be submitted as batch jobs by n1ge. 3.5.3. Confirm available queues Please use charged queues to submit jobs not being small scale. And, you can use the following command to confirm whether charged queues are available.
$ qgroup -a
GROUP_NAME TYPE STATUS MAX NJOBS PEND RUN CONSUME AVAILABLE -------------------------------------------------------------------------------------------4B080137 BES OK 320(cpu) 6( 56) 0( 0) 6( 56) 160 160(cpu) 4S080138 SLA OK 28800(h) 0( 0) 0( 0) 0( 0) 27845(h)

--------------------------------------------------------------------------------------BES GROUP 4 5 6 7 8 9 10 11 12 1 2 3 TOTAL --------------------------------------------------------------------------------------4B0801xx 320 320 320 320 320 320 320 320 320 320 320 320 3840
-[ Running jobs ] job-ID pl ppri prior et rt(min) name user group state start at running queue slots ---------------------------------------------------------------------------------------------------------------1410425 1 -700 0.26320 2w 10081 SMP_8_1 suntest 4B080xxx r 10/14 12:54 bes2@tgg072164 8(16) 1410426 1 -700 0.21320 1w 10080 SMP_4_1 suntest 4B080xxx r 10/14 12:55 bes2@tgg072151 4 1410427 1 -700 0.21320 1w 10080 SMP_4_2 suntest 4B080xxx r 10/14 12:55 bes2@tgg072163 4 1410428 1 -700 0.21320 1w 10080 SMP_4_3 suntest 4B080xxx r 10/14 12:55 bes2@tgg072155 4 1410429 1 -700 0.21320 1w 10080 SMP_4_4 suntest 4B080xxx r 10/14 12:55 bes2@tgg072157 4 1410430 3 -100 0.45666 1w 10080 MPI_3_32_1 suntest 4B080xxx r 10/14 12:57 bes2@tgg072165 32(128) [ Pending jobs ] job-ID pl ppri prior et rt(min) name user group state submit at requested queue slots ---------------------------------------------------------------------------------------------------------------1410431 1 -700 0.00000 1w 10080 SMP_16_16 suntest 4B080137 qw 10/14 13:07 bes2 16

The bes queue and sla queue is available in the above example. You can specify a queue (bes1/bes2/cs1/cs2, or sla1/sla2/sla3) based on the batch queue predictor.

The status of each queue can be checked using the command qstatus. $ qstatus [option] [queue_name]
[option] -node display node list that you specified queuename (default)

- 29 -

-stat -free -sum

display statistics information display only free node and some summarized information display only summarized information

[queue_name]

-nocharge -bes -sla -inno -pinno -sla1all -sla2all -sla3all -slaall -hpc -ram
-<queuname> None(*)

novice bes1,bes2,cs1,cs2,bes1tes2,bes2tes2 sla1,sla2,sla3,sla1tes2,sla2tes2,sla3tes2 inno1,inno2,inno3 pinno1,pinno2,pinno3 sla1,inno1,pinno1 sla2,inno2,pinno2 sla3,inno3,pinno3 sla1,sla2,sla3,inno1,inno2,inno3,pinno1,pinno2,pinno3 hpc1,hpc2 RAM64GB,RAM128GB
display host information about <queuname> that you specified. (e.g.: -novice -bes) novice,bes1,bes2,sla1,sla2,sla3,RAM64GB,RAM128GB,sas

(*)If you don't specify queue_name, you have to specify an option.

The status of each queue is displayed. Please refer to the queue status before submitting your jobs. (Example) Display the node list of bes $ qstatus -node -bes
-HOSTNAME STATUS REST CPUs REST MEM ENABLE-QUEUES --------------------------------------------------------------tgg073121 OK 4/16CPU 20/32 GB bes2 tgg073122 FULL 0/16CPU 16/32 GB tgg073123 OK 8/16CPU 24/32 GB bes2 tgg073124 OK 8/16CPU 24/32 GB bes2 tgg073125 OK 16/16CPU 32/32 GB tgg073126 FULL 0/16CPU 16/32 GB tgg073127 DISABLE 16/16CPU 28/32 GB tgg073128 ERROR 16/16CPU 32/32 GB tgg073129 OK 8/16CPU 16/32 GB bes2 : (skip) tgg074001 OK 2/16CPU 7/32 GB bes1 tgg074002 FULL 0/16CPU 8/32 GB : (skip) Note: "ENABLE-QUEUES" means that the displayed queue names can use the host.

(e.g.) Display only summarized information of sla, inno and pinno queues $ qstatus -sum -slaall
-QUEUE FREE NODE FREE CPU FREE MEMORY USED NODE DOWN NODE DISABLE NODE TOTAL NODE ---------------------------------------------------------------------------------------------TOTAL 149 2384 CPU 4768 GB 539 3 197 - inno1 15 240 CPU 480 GB 99 0 4 118 - inno2 35 560 CPU 1120 GB 81 0 2 118

- 30 -

inno3 pinno1 pinno2 pinno3 sla1 sla2 sla3

0 15 35 0 14 35 0

0 240 560 0 224 560 0

CPU CPU CPU CPU CPU CPU CPU

0 480 1120 0 448 1120 0

GB GB GB GB GB GB GB

0 99 81 0 98 81 0

1 0 0 1 0 0 1

59 4 2 59 6 2 59

60 118 118 60 118 118 60

Note: The values of plural queues on the same node equalize. For the above, "sla1/inno1/pinno1", "sla2/inno2/pinno2" and "sla3/inno3/pinno3".

3.5.4.

Submit a job

To use the batch processing system, the job script is submitted to the queue as a job. n1ge command (N1 Grid Engine) is used for submitting. The jobs submitted by n1ge are divided to two types: background and foreground jobs. [Foreground] The standard output and error message are displayed in the terminal in which the n1ge command is executed, and the prompt will not appear until the job is finished. $ n1ge -fore [Background] The standard output and error messages are outputted to files and the prompt appears on submitting the job. $ n1ge myjob.sh Your job 753 (jobname) has been submitted. myjob.sh

3.5.5.

Using n1ge command

The n1ge command has the option to control the job. The specification of the foreground explains by paragraph 3.5.4 is one of the options. The option often used is specification of the job name and the queue, and specified options of the account group. In the TSUBAME system, n1ge is used to submit a batch job as follows. $ n1ge -N jobname -q sla2 -g 4S0801xx myjob.sh Your job 753 (jobname) has been submitted. -N option specifies a job name -q option specifies a QUEUE You may get the other options by typing the command n1ge help. 753 is the job ID assigned by the system uniquely. Not specifying the queue - 31 -

A job is allotted to the nodes with novice queue if its parallel number is less than 32, or it can not be submitted. If -g (group) is specified, the job will be allotted to the queue with respect to the group. You may get the help list of n1ge by tying the n1ge help command.
$ n1ge -help n1ge [n1ge_options.]<program>[options] n1ge_options: -help -verify -smp <cpus> :: This message. :: Do not submit just verify. :: Request SMP machines. <cpus> Specify the number of cpus to run on. -mpi <cpus> :: Request using message passing interface. <cpus> Specify the number of cpus to run on. -linda <cpus> :: Request using linda for Gaussian 03 rev E.01. <cpus> Specify the number of cpus to run on. -ddi <cpus> :: Request using DDI for GAMESS. <cpus> Specify the number of cpus to run on. -proc <cpus> :: Request SMP machines. Use this option only for your original jobs. this option limits the memory of not the job but each process. <cpus> Specify the number of cpus(processes) to run on. -mem <memory size> :: [MPI Jobs] <memory size> Specify the size of memory for each process of job.(GBytes) With -mpi, -openmpi, -ddi options. [NonMPI Jobs] <memory size> Specify the size of memory for the job not processes. ONLY with -smp option. ex1) "-smp 8 -mem 16" means this job will use 16GB and limit 16GB/job. ex2) "-mpi 4 -mem 2" means this job will use 2GB*4=8GB memory and limit 2GB/proc. -apps_list :: Show Application Name you can specify with -apps or -apps_help option. -apps <APPS Name> :: Specify Application Name <APPS Name> apps_name[,apps_name,...] -apps_help <APPS Name> :: Show applicatoin specific help message of n1ge command. <APPS Name> apps_name -q <Queue Name> :: Bind job to the specified queue(s). <Queue Name> queue[,queue,....] -mail <Mail Address> :: Request mail notification when begging, ending of the job and aborted or suspended the jobs. <Mail Address> Specify the E-mail Address. -env <variable_list> :: Export these environment variables on Execution Hosts. <variable_list> variable[=value][,variable[=value],....] -fore :: Executing this job as foreground job. -noreq :: Define job as NOT Re-queueing. -paramfile <filename> :: Specify parameter file of Voltaire MPI. You can get the sample paramfile as /opt/vltmpi/OPENIB/mpi.pgcc.rsh/doc/param.sample You need to use -mpi option together. -N <JOB NAME> :: Specify Job Name -hold_jid <Job-ID list> :: Define jobnet interdependencies <Job-ID list> {job_id|job_name|reg_exp}[,{job_id|job_name|reg_exp},...] -divout :: Divide standard output and err stream of N1GE [default stdout filename]: jobname.oJOB_ID [default stderr filename]: jobname.eJOB_ID If you want to specify each filename please use -sgeout,-sgeerr options -sgeout <filename> :: Specify standard output stream path(s) of N1GE -sgeerr <filename> :: Specify standard err stream path(s) of N1GE -g <Group Name>

- 32 -

:: Specify the Group Name. -rt <runtime(min)> :: Specify the runtime of the job when you belong to SLA group. <runtime(min)> In minutes. SLA: Default value is 30 minitus now. BES: Default value depends on the specified value with -et. -rank0mem <memory size> :: Specify the memory size of Rank0 processes for MPI jobs. (GBytes) -stripe <count> :: Specify the number of stripes for Lustre FS /gau. This option is available with only G -pl <Premium Level> :: Specify The TSUBAME Premium Level. The value that can be selected is 1, 2 and 3. Hierarchical relationship of job priority is shown as follows. Priority | default high grade highest grade ------------+------------------------------------------n1ge option | -pl 1 < -pl 2 < -pl 3 Please refer to the GSIC Web site for a detailed usage of this function. Default level is 1. -et <Executable Period> :: About only BES type job, the execution time limit will be extended. The value that can be selected is 1w, 2w, and 4w. (weeks) 1w = 7 days 2w = 14 days 4w = 28 days Please refer to the GSIC Web site for a detailed usage of this function. Default level is 1w. option -help -verify -smp -mpi -linda -ddi -proc -mem -apps_list -apps -apps_help -q -mail -env -fore -noreq -paramfile -N -hold_jid -divout -sgeout -sgeerr -g -rt -ran0mem -stripe -pl -et option parameter Introduction n1ge help not submit a job, and display the queue command and job script created by the n1ge system processors in an SMP job processors in an MPI job processors in a Gaussian_linda job processors in a Gamess job processes in a parallel job (intra-node) Specify memory size for a process Show application list ready for N1GE Specify a application display submitting methods with n1ge for all the installed applications designed for n1ge batch system specify a queue name A message will be send to the Mail_Address when the job starts running, terminates, is deleted. specify environment variables for the job The job will be executed in foreground, or it will be executed in background. Not re-queue after a job failed Specify parameter file of Voltaire MPI Specify a jobname The submitted job is not executed until the jobs <Job-ID list> terminate. Divide standard output and err stream of N1GE N1GE will output to the <filename> instead of jobname.ojobid. Specify standard err stream path(s) of N1GE Specify a N1GE_GROUP to utilize the BES or SLA queue Specify the runtime for a BES or SLA job Specify the memory size of Rank0 processor for MPI jobs (GBytes) Specify the number of stripes for Lustre FS /gau Specify the Premium level Specify execution time unit

<cpus> <cpus> <cpus> <cpus> <cpus> <memory size> <APPS Name> <APPS Name> <Queue Name> <Mail_Address> <variable_list>

<paramfile> <JOB NAME> <Job-ID list> <filename> <filrname> <Group Name> <runtime(min)> <memory size> <count> <Premium Level> <Execution period>

- 33 -

Note that a SLA job would be terminated in 30 minutes without rt option. We show some examples on how to use the options. (1) I want sequentially to execute two or more jobs. (-hold_jid) By using the "-hold_jid" option of n1ge, the following job runs following a job specified. $ n1ge -hold_jid Job-ID PROGRAM Job ID can be confirmed by the qjobs command (2) The job doesn't run again while it is terminated abnormally. (-noreq) When a job is abnormally terminated due to system troubles, the job is automatically submitted again by the system by default. However, in many cases the job can not be completed normally even though re-running. A job is not submitted again if the -noreq option is specified. $ n1ge -noreq PROGRAM (3) I want to make some special processing only on the rank0. use the -rank0mem option. $ n1ge -mpi 9 -mem 2 -rank0mem 28 PROGRAM In this case, 28GByte memory is used with rank0, and 2GByte memory/process is used in the other processes. (-rank0mem)

If you want to make the rank0 use more memory than the other ranks, it is effective to

3.5.6. Parallel jobs The TUBAME system supports SMP parallel and MPI parallel (1) SMP parallel: OpenMPpthread Submit an OpenMP job with 4CPUs. $ chmod a+x parallel.sh $ n1ge -smp 4 parallel.sh (2) MPI Submit an MPI job with 4 CPUs. $ n1ge -mpi 4 MPI-PROGRAM Submit an MPI job by specifying the total CPU number CPUS-TOTAL and CPU number/node CPUS-NODE. $ n1ge -mpi CPUS-TOTAL:CPUS-NODE MPI-PROGRAM For example, submit an MPI job by requesting total 32 CPUs and using 8 CPUs in each node. - 34 -

$ n1ge -mpi 32:8 MPI-PROGRAM You also can specify a different memory size for Rank=0 with the other Rank. For example, a job with 9 CPU in which the Rank=0 need 10GB memory and the others use 2GB memory/rank may be submitted as follows. n1ge mpi 9 rank0mem 10 -mem 2 MPI-PROGRAMM

Where, the MPI-PROGRAMM is either an executable program compiled by using the command mpicc, mpif77, or mpif90 in the directory /opt/vltmpi/OPENIB/mpi.pgcc.rsh/bin. You may also use a Shell script instead of an MPI-program, but dont use the mpirun command in the shell script. (3) MPI + OpenMP, pthread A job may be submitted as follows, in which MPI is used among nodes and OpenMP or pthread is used within a node. $ n1ge mpi NODES -smp CPUS/NODE

PROGRAM

Example, submit a job in which MPI is used among nodes, and SMP parallel is used within each node, where we request 4 nodes, total 32 CPUs and use 8 CPUs in each node to submit the batch job. $ n1ge -mpi 4 -smp 8 PROGRAM The PROGRAM is an MPI program compiled by the command mpicc, mpif77, or mpif90 locating in the directory /opt/vltmpi/OPENIB/mpi.pgcc.rsh/bin, and it is also an OpenMP /pthread SMP parallel program. (4) Simultaneous processing of single job When the sla queue is used, the single job might have room in the resource. In this case, the work efficiency can be improved by doing the simultaneous processing. To use each 3Gbyte, eight single jobs are specified as follows. $ n1ge -proc 8 memory size. -mem 3 parallel.sh though the mem specification of -smp is the entire

It becomes about one job in -proc

Only one node (16CPU) can be specified, and it is not possible to use it

together with other parallel options. It doesn't correspond to the application in Chapter 5. Please refer to FAQ for how to make the shell. 3.5.7. Memory restriction on batch jobs

Users can specify memory usage when submitting a job. For example, specify 4GB memory for an OpenMP/pthread job.

$ n1ge -smp 4 -mem 4 parallel.sh

- 35 -

Note: The method to specify memory size for an SMP parallel job is different the one for an MPI parallel job. (1) Specify the maximum memory usage of all the threads for an SMP job (2) Specify the maximum memory usage of 1 process for an MPI job (3) Specify the maximum memory usage of all SMP threads in 1 Linda process for a Linda job (4) Specify the maximum memory usage of all the threads in an MPI process for MPI+SMP job If the job request more memory than that specified by system, the job will be terminated immediately. The system allocates 1GB memory for one process without -mem option. The maximum size of available memory is different between different queues, so please confirm the available memory size when submitting a batch job. (1) Specify memory for a single job Please specify memory size for a single job based on the memory size needed by the program which can be shown in the examples as follows. (a) Static array allocation is adopted in the program. $ pgsize a.out
:

3221238694 Total Where, the memory size needed by program is 3.22GB. So please specify 3.3GB to submit the job.

1161

.strtab

$ n1ge -mem 3.3 ./a.out

(b) Dynamic array allocation is adopted in the program. The memory size needed by the program can not be shown by the method in (a). Please run it on an interactive node, and then confirm the memory size using top command. (You can refer to the column VIRT or RES). Please specify the memory size same as the method (a). (2) Specify memory for an MPI job Voltaire communication uses memory as its communication buffer. Some additional buffer memory is necessary for each process in an intra-node communication. The memory size changes by the intra-node communications, and it can easily be shown as follows, if we assume as follows, The memory size is 1.4GB if it is executed in single mode MPI parallel number: 128 Some additional memory must be allocated as communication buffer when we estimate the memory size for an MPI program. For example, about additional 500MB memory/proc is - 36 -

needed by the intra-node communication if there are 16 processes per node, and about 200MB memory/proc is needed if there are 8 processes per node. Thus you can submit a job (16 processes per node) with memory 1.4GB+500MB, or submit a job (8 processes per node) with memory size 1.4GB+200MB, as follows. $ n1ge -mpi 128:8 -mem 1.6 ./a.out $ n1ge -mpi 128:16 -mem 1.9 ./a.out A job would be submitted as mpi 128:8 if no intra-node parallel number (-mpi 128) is specified, and it be submitted as -mpi 128:1 (1 process per node) if smp option is also specified. (3) Notices In the above example, 1.9GB memory is needed by one process when the job is submitted with the command n1ge mpi 128:16 , and some additional system memory must be used as communication buffer between nodes. The size of the additional buffer memory used by inter-node communication is about 200MB/proc while the parallel number is 128 as shown in Fig. 1. In this case where the mem 2 is used, no enough memory would be used as communication buffer between nodes, thus the program would have to use swap space, which will result communication delay. So you have to decrease the CPU number per node, or you must use a parameter which cost less memory than default one. In addition, when there is large amount of communication data or there is large number of communication times in the program, the more swap will be used. So please specify total memory size less than 30GB for an MPI job if possible.

- 37 -

Fig 1. memory size used by per process with communication 1:1(sendrecv) 1200 1000 Memory siez(MB) 800 600 400 200 0 0 50 100 150 200 Paralle number(core) 250 300
within one node no paramfile(:8) lowmem(:8) Debug(:8) combine(:8) no paramfile(:16)

On the Figure The memory used by the program without MPI communication is about 100MB, and the factual memory size is shown for different MPI parallel number. The memory size used by MPI communication is determined by the parallel number and independent of communication type. Several simple samples are shown on how to use MPI parameters the parallel number < = 64 It is not necessary to specify a parameter file if the job executes normally. You can specify the lowmem parameter file as follows if an error occurs without parameter file. n1ge .... -paramfile lowmem the parallel number >64 You need to specify the parameter lowmem corresponding to massive parallel job. (Sometimes a program work without parameter file) n1ge .... -paramfile lownoSHD -mpi 512 ... You have to specify the parameter lownoSHD exclusive for massive parallel job if errors occur.. On use other parameter files(for development/debug, or the above parameter files - 38 -mpi 64 ...

dont work) Costing less memory, and longer execution time(about 30%) n1ge .... paramfile debug -mpi 512 .... ) Costing little more memory, and the performace keeps with parallel number increasing( limited to *:8 or less processes per node n1ge .... paramfile combine 3.5.8. Confirm the job status -mpi 512:8 ...

You can confirm the status of the job submitted using qjobs or qstat, where -u option is used to specify a userID.
$ qstat -u user001 job-ID prior name user state submit/start at queue slots ja-task-ID -------------------------------------------------------------------------------------------------------753 0.50511 jobname user001 r 08/01/2007 15:53:13 novice@tgg075049 8 job-ID prior name user state submit/start at queue slots ja-task-ID -------------------------------------------------------------------------------------------------------4364 0.55500 LOGIN user001 r 07/20/2007 11:20:11 interactive@tgg075018 1 job-ID prior name user state submit/start at queue slots ja-task-ID -------------------------------------------------------------------------------------------------------1455 0.55500 TEST user001 r 07/30/2007 12:45:21 sla3@tgg072123 1

Two sections are shown, the second one shows the queue information about avs, sas,

interactive. And the job 753 with a queue novice in running status is shown in the first section. And there is a job with the queue interactive, which is a job produced automatically login process in the second section. For more detailed information,you can read online manual by typing man qstat. The finished or being finishing jobs may be confirmed by using qjobs f.
$ qjobs -f 2 Following job use "novice bes1 bes2 mopac" queue. [ Finished Jobs (2 hours ago -> 07/30/2007 16:34:12)] job-ID type name user state submit time submit host executed queue slots cpu memory -----------------------------------------------------------------------------------------------------------1105507 BACK AMBER user001 DONE 07/25/2007 15:53:07 tgg075017 bes1@tgg074049 8 0.0 60.72 Following job use "avs gridMathematica" queue. [ Finished Jobs (2 hours ago -> 07/30/2007 16:34:12)] job-ID type name user state submit time submit host executed queue slots cpu memory -----------------------------------------------------------------------------------------------------------4736 FORE LOGIN user001 DONE 07/29/2007 09:29:48 login2 interact@tgg075001 1 1.0 82.67 Following job use "RAM128GB inno2 RAM64GB pinno2 sla1 pinno3 pinno1 sla2 inno1 sla3 inno3" queue. [ Finished Jobs (2 hours ago -> 07/30/2007 16:34:12)] job-ID type name user state submit time submit host executed queue slots cpu memory -----------------------------------------------------------------------------------------------------------1239 BACK ****** ******* DONE 07/30/2007 20:16:00 tgg075020 pinno1@tgg072150 4 0.0 41.23 1237 BACK OTHERS user001 DONE 07/30/2007 20:11:24 tgg075017 inno1@tgg072154 4 0.0 61.55

by a

The output is divided into three groups, based on the N1GE attribute. The commands qstat and qjobs dont display the names of user and job of the charge group which you dont belong to. - 39 -

The cpu is the CPU time in second, and the memory is the maximum memory size during execution in Mbytes. The jobs finished for less than two hours would be displayed without any option.

3.5.9. to.

Delete a job

Jobs may be deleted by using the following commands as to N1GE group which the jobs belong freeqdel (For novice, bes[1,2], mopac) int qdel_int (For interactive, sas, csdebug, sunstudio) sla qdel_sla (For sla[1-3], RAM??GB, inno[1-3], pinno[1-3]) You can delete job specifying the job id. $ qdel 312 user001 has deleted job 312 There is yet another command to delete jobs. It is qdelete.
$ qdelete -help usage: qdelete [-q queue|-c n1ge_group] [options] job_task_list [-f] [-help] [-q queue] [-c n1ge_group] [-u user_list] job_task_list force action print this help specify queue specify n1ge_group delete all jobs of users specified in list delete all jobs given in list

job_task_list job_tasks task_id_range user_list queue n1ge_group

job_tasks[,job_tasks,...] [job_id['.'task_id_range]|job_name|pattern][' -t 'task_id_range] task_id['-'task_id[':'step]] user|pattern[,user|pattern,...] queue free|bes|int|sla

When you specified "-q" option, system determine "n1ge_group" from queue you specified. When you specify neither "-q" nor "-c", system will specify "-c free" automatically. You can refer to "n1ge_group" by executing "qlist -a".

Example: Delete the job(Job-ID 1350) running on the inno1 queue. - 40 -

$ qdelete -c sla 1350 or $ qdelete -q inno1 1350

When you specify the queue with -q option to delete the jobs, the objects of deletion are the ones in the N1GE group which the specified queue belongs to.

3.5.10. Confirm the results An execution log file is created in the execution directory when the job completes. The file with a name JOBNAME.oJOBID contains the messages from standard output and error. $ ls jobname.o753

3.5.11. GUI A GUI tool for the queue system is also available to confirm the status of jobs (only in English environment) You can execute qmon with X Window environment as follows. $ LANG=C $ export LANG $ qmon & The qmon display the queues information of only an N1GE group. According to your subject, please run the following environment-set-script before running qmon to refer the queues information. 1) free: novice, bes1, bes2, mopac $ . /n1ge/TITECH_INT/common/set_free.sh for sh/bash for csh/tcsh) for sh/bash ($ source /n1ge/TITECH_INT/common/set_free.csh $ . /n1ge/TITECH_INT/common/set_int.sh

2) int: interactive, sas, csdebug, sunstudio,tesladebug ($ source /n1ge/TITECH_INT/common/set_int.csh for csh/tcsh) 3) sla: sla1, sla2, sla3, inno1, inno2, inno3, pinno1, pinno2, pinno3, RAM64GB, RAM128GB,hpc $ . /n1ge/TITECH_INT/common/set_sla.sh for sh/bash ($ source /n1ge/TITECH_INT/common/set_sla.csh for csh/tcsh)

- 41 -

Please press the [Help] button to find its usage. Note: Use n1ge command to submit a batch job. Not guarantee the result or execution without using n1ge command. 3.6. Printing All the printers in GSIC support PostScript(hereafter PS) file format only A text file should be converted to PS format by using a2ps before it is sent to a printer. All the available printers in GSIC are described in the subsection 2.1.6 .

3.6.1.

Printing from a local PC

Printing out from a local PC is not available. 3.6.2. Use the printers on the media workstations, or on the interactive nodes

The printers are available on the media-o, media-s, and tgg075017~tgg075032. However, the prn0-o or prn0-s only receives print jobs from media-o or media-s respectively, and the print method is described as follows, (1) Available printers (2) Print by using lpr (3) Confirm print jobs using lpq (4) Terminate a print job using lprm (5) Set a default printer (1) Available printers location printer prn0-o_a0 prn0-o_b0 prn1-o_a4 (*) prn1-o_a4r prn1-o_a3 prn1-o_a3r comment A0 printer B0 A4 A4 double-sided A3 A3 double-sided - 42 -

Ookayama

Suzukakedai

prn0-s_a0 prn0-s_b0 prn1-s_a4 (*) prn1-s_a4r prn1-s_a3 prn1-s_a3r nodes

A0 B0 A4 A4 double-sided A3 A3 double-sided only the printers with

*Default printer On the interactive tgg075017 tgg075032, names prn1-* are available. (2) Print by using lpr The lpr command is used to submit a file to a printer. The printer supports PS file format only. Please refer to example 2 to print a text file. Print PS files with lpr $ lpr [-P<Pinter>] -P <Printer> file1 file2 ... The files will be printed to a printer specified by <Printer>, unless the files will be printed to the default printer. Example 1. Print sample.ps on the colored PS printer in Ookayama. $ lpr -Pprn1-o_a4 sample.ps

Example 2. Print a Japanese text file sample.txt to the color PS printer in Suzukakedai. It is not necessary to set LANG variable if you print an English text file. for sh/bash $ LANG=ja_JP.eucjp;export LANG $ nkf -e sample1.txt | a2ps -Pprn1-s_a4 for csh/tcsh $ setenv LANG ja_JP.eucjp $ nkf -e sample1.txt | a2ps -Pprn1-s_a4 Example 3. Print a PS file with A3 size on the color PS printer in Ookayama $ lpr -Pprn1-o_a3 sample.ps Example 4. double-sided print on the color PS printer in Ookayama $ lpr -Pprn1-o_a4r sample.ps Example 5. print to the B0 printer in Ookayama $ psresize -pB0 -PA4 sample.ps | lpr (3) Confirm print jobs using lpq - 43 -Pprn0-o_b0

Use lpq command as follows, $ lpq [-P<Printer>] -P option: specifies a printer, or the default printer would be specified. Example, Confirm the print jobs queued on the default printer. $ lpq prn1-o_a4 is ready and Rank Owner Job active titech 9 1st titech 10 2nd titech 11 3rd titech 12 printing File(s) (stdin) sample.ps (stdin) test.ps

Total Size 19456 bytes 20855 bytes 1024 bytes 21569 bytes

(4) Terminate a print job using lprm You may list the print jobs by using lpq, and then delete a print job by using lprm. Usage of the command lprm $ lprm [-P<Printer>] iobID ... Options -P Printer: specifies a printer, or the default printer is used. For example, delete the jobs 9 10 11 listed in the step (3) $ lprm 9 10 11 (5) Set a default printer You may set a default printer as follows, or the prn1-o_a4 media-o /prn1-s_a4 (media-s) is the default printer. If the PRINTER variable is set, the printer is used as default printer. For example, set the prn1-s_a4 as your default printer as follows. for sh/bash $ PRINTER=prn1-s_a4;export PRINTER for csh/tcsh $ setenv PRINTER prn1-s_a4

- 44 -

3.7. Usage of LTO3 The LTO3 tape device connected to the media-o or the media-s is available to backup your data as follows, (1) Insert a media Insert a media to LTO3. (2) Read/Write data to the media You can copy your backup data by using the command tar or cpio, having logged in media-o or media-s. The device file name of the LTO3 is /dev/st0 in the media workstations. For example, backup the data in the directory ./save to the LTO3 media. $ tar cvf /dev/st0 ./save

(3) Eject the media Press the button on the front panel of LTO3 to eject the media on finishing the backup.

Eject button

3.8. Proxy server Hostname IP address Port proxy-o.cc.titech.ac.jp 131.112.14.2 8080

- 45 -

4. Program Languages and development environment

The PGI CDK(Cluster Development Kit) is installed in the TSUBAME system. The PGI CDK contains all the software development and cluster management tools you need to get a Linux cluster running production applications. 4.1. Compilers 4.1.1. Compile and link program Use pgf95 command to compile Fortran program. The pgf95 compiler is upper compatible with Fortran 77, so you can compile Fortran77, Fortran90, and Fortran 95 programs using pgf95. And, the usage of the other compilers is similar. Example 1, Compile Fortran program $ pgf95 sample.f Example 2, Compile C program $ pgcc sample.c Example 3, Specify output file with -o option. $ pgf95 -o sample sample.f The file a.out is produced without o option in the Example 1 and 2. Caution on executing object file The message FORTARN STOP will be outputed as a standard error after the execution is completed. You can remove the message by setting the varaible NO_STOP_MESSAGE to any value. 4.1.2. Compiler version

The PGI 7.2-4 is available on TSUBAME without additional settings. You can also use the some other versions as follows. The following two versions and their locations are listed. /usr/apps/isv/pre_release/pgi6.2.5 /usr/apps/isv/pre_release/pgi6.1-2 (Default compiler for sles9) You can choose PGI compiler version by setting the following enviroment variables, and you have to confirm the directory name and specify the corresponding name instead of the part of "**".
$ export PGI=/usr/apps/isv/pre_release/pgi** $ export PATH=/usr/apps/isv/pre_release/pgi**/linux86-64/**/bin:$PATH $ export LD_LIBRARY_PATH=/usr/apps/isv/pre_release/pgi**/linux86-64/**/libso:$LD_LIBRARY_PATH

You can check the PGI version as follows,

$ pgf95 -V

pgf95 7.2-4 64-bit target on x86-64 Linux - 46 -

Copyright 1989-2000, The Portland Group, Inc. All Rights Reserved. The version of PGI compiler is shown, and the MPI command (such as mpif90) calls the shown version If you want to use the versions with MPI libarary, please confrim their movement state, because it is possible that the current MPI libarary doesn't adapt to them. 4.1.3. Other compiler The Intel compilers are also available on TSUBAME, which are introduced on the main purpose of compiling user subroutine of some ISV applications such as MSC. Nastran, and ABAQUS. Youd better to use PGI CDK to compile program. Please set compile environment as follows to use INTEL EMT64bit Fortran/C++ compiler. Fortran compiler Version 11 $ . /usr/apps/isv10/intel/11/bin/ifortvars.sh intel64 sh/bash (source /usr/apps/isv10/intel/11/bin/ifortvars.csh intel64) csh/tcsh Version 10 $ . /usr/apps/isv10/intel/fce/10.1.011/bin/ifortvars.sh sh/bash (source /usr/apps/isv10/intel/10.1.011/bin/ifortvars.csh) csh/tcsh C/C++ compiler Version 11 $ . /usr/apps/isv10/intel/11/bin/iccvars.sh intel64 sh/bash (source /usr/apps/isv10/intel/11/bin/iccvars.csh intel64) csh/tcsh Version 10 $ . /usr/apps/isv10/intel/cce/10.1.011/bin/iccvars.sh sh/bash (source /usr/apps/isv10/intel/cce/10.1.011/bin/iccvars.csh) csh/tcsh

(1)About Intel compilers

(2) SUN compiler Fortran $ /usr/apps/isv/pre_release/sunstudio/sunstudio12/bin/f95 -fast -m64 -o sample2 sample2.f C $ /usr/apps/isv/pre_release/sunstudio/sunstudio12/bin/cc -o sample2 sample2.c

4.1.4.

File input and output

Compile and link is successfully finished, then execute it. Input data are necessary to execute the program if there are READ statements in the program. For Fortran, the IO files have the following names if you specify the unit number in Read/write statement, write(10,*) read (20,*) fort.10 - 47 fort.20

However, the units 0, 5, and 6 are pre-defined: unit 0(standard error), unit 5 (standard input), and unit 6(standard output). devices. For example, you may redirect standard input to the input file sample.inp, and redirect the standard output to the file sample.out when executing a program. $ ./a.out < sample.inp > sample.out The standard input and output units can be redirected to files or

4.1.5.

Using library

Specify the library to link program with -l option. Example 1, link libblas.a in Fortran program. $ pgf95 sample.f -lblas Example 2, link libblas.a in C program. $ pgcc sample.c -lblas Example 3, compile Fortran program with libprivate.a in the directory /home/hoge/private. $ pgf95 sample.f -L/home/hoge/private -lprivate Example 4, compile Fortran program with ACML installed on TSUBAME $ pgf95 sample.f -lacml Please refer to http://www.gsic.titech.ac.jp/~ccwww/tebiki/FORTRAN/fortran4.html#41

4.2. Parallelization MPI: distributed parallel SMP: OpenMP, pthread 4.2.1. MPI MPI is a library specification for message-passing, proposed as a standard by a broadly based committee of vendors, implementers, and users. The mpif90 (mpicc) is used to compile MPI program on the TSUBAME system. Example 1, compile and link MPI programs. $ mpif90 -o sample sample.f Example 2, execute a MPI program sample with 2CPUs. Execute it on the interactive nodes $ mpirun -np 2 ./sample Execute it in foreground on the batch nodes with 2 CPUs. $ n1ge -fore -mpi 2 ./sample

- 48 -

Sample MPI program calculates .

program compute_pi implicit none include 'mpif.h' integer n,i,myid,numprocs,rc,ierr double precision w,x,sum,pi,f,a,mypi

! function to integrate f(a)=4.d0/(1.d0+a*a) call mpi_init(ierr) call mpi_comm_rank(mpi_comm_world,myid, ierr) call mpi_comm_size(mpi_comm_world,numprocs,ierr) if(myid.eq.0) then ! print *,'Enter number of intervals: ' ! read *,n n=500000000 end if call mpi_bcast(n,1,mpi_integer,0,mpi_comm_world,ierr) ! calculate the interval size w=1.0d0/n sum=0.0d0 do i=myid+1,n,numprocs x=w*(i-0.5d0) sum=sum+f(x) end do mypi=w*sum ! collect all the partial sums call mpi_reduce(mypi,pi,1,mpi_double_precision, &mpi_sum,0,mpi_comm_world,ierr) ! node 0 prints the answer. if(myid.eq.0) then print *,'computed pi=',pi end if call mpi_finalize(rc) end program compute_pi

Compile and link the program $ mpif90 -o compute_pi compute_pi.f - 49 -

Execute it on the interactive node $ mpirun -np 2 ./compute_pi Execute it on batch nodes $ n1ge -fore -mpi 2 ./compute_pi Result $ computed pi= Note on MPI (1) Using fork function Please use vfork function instead of fork function because fork function may cause error. (2) Using mmap and munmap function Please avoid repeating uses of mmap function and munmap function in case of an inconsistence of the data during communication. 4.2.2. OpenMP 3.141592653589621

OpenMP is an API for multi-platform shared-memory parallel programming in C and Fortran languages. Use pgf95/pgcc with -mp option to compile and link OpenMP parallel programs. Example 1, compile and link an OpenMP Fortran program. $ pgf95 -mp -o sample sample.f Example 2, execute an OpenMP program sample with 2 CPUs. Execute it on an interactive node $ export OMP_NUM_THREADS=2 for sh/bash (setenv OMP_NUM_THREADS 2) for csh/tcsh $ ./sample Execute it on the batch nodes with. $ n1ge -fore -smp 2 ./sample Sample OpenMP program calculates
program compute_pi implicit none integer n,i double precision w,x,sum,pi,f,a ! function to integrate f(a)=4.d0/(1.d0+a*a) n=500000000 ! calculate the interval size

- 50 -

w=1.0d0/n sum=0.0d0 !$omp parallel do private(x),shared(w),reduction(+:sum) do i=1,n x=w*(i-0.5d0) sum=sum+f(x) end do pi=w*sum print *,'computed pi=',pi stop end program compute_pi

Compile and link the program $ pgf95 -mp -o compute_pi compute_pi.f Execute it on the interactive nodes with 2 CPUs $ export OMP_NUM_THREADS=2 for sh/bash (setenv OMP_NUM_THREADS 2) for csh/tcsh $ ./compute_pi Or execute it on the batch nodes with 2 CPUs. $ n1ge -force -smp 2 compute_pi Result $ computed pi= 3.141592653589814

4.2.3.

pthread

PGI compilers also support pthread and pthread is only available in C program. The pthread program creates or deletes threads by using C functions. Threads share memory space, so CPUs execute one process. Example 1, compile and link a pthread parallel program. $ pgcc -o sample sample.c -lpthread Example 2, execute a pthread program with 2 CPUs. Execute it on the interactive node $ ./sample 2 Execute it on batch nodes $ n1ge -fore -smp 2 ./sample Sample pthread program calculates - 51 -

#include <stdio.h> #include <stdlib.h> #include "pthread.h" volatile double pi = 0.0; /* Approximation to pi (shared) */ pthread_mutex_t pi_lock; /* Lock for above */ volatile double intervals; /* How many intervals? */ void * process(void *arg) { register double width, localsum; register int i; register int iproc = (*((char *) arg) - '0'); /* Set width */ width = 1.0 / intervals; /* Do the local computations */ localsum = 0; for (i=iproc; i<intervals; i+=2) { register double x = (i + 0.5) * width; localsum += 4.0 / (1.0 + x * x); } localsum *= width; /* Lock pi for update, update it, and unlock */ pthread_mutex_lock(&pi_lock); pi += localsum; pthread_mutex_unlock(&pi_lock); return(NULL); } int main(int argc, char **argv) { pthread_t thread0, thread1; void * retval; /* Check command line */ if (argc != 2) { printf("Usage : %s Num-intervalsn",argv[0]); exit(0); } /* Get the number of intervals */ intervals = atoi(argv[1]); /* Initialize the lock on pi */ pthread_mutex_init(&pi_lock, NULL); /* Make the two threads */ if (pthread_create(&thread0, NULL, process, "0") || pthread_create(&thread1, NULL, process, "1")) { fprintf(stderr, "%s: cannot make threadn", argv[0]); exit(1); } /* Join (collapse) the two threads */ if (pthread_join(thread0, &retval) || pthread_join(thread1, &retval)) { fprintf(stderr, "%s: thread join failedn", argv[0]); exit(1); }

- 52 -

/* Print the result */ printf("computed pi = %.16fn", pi); /* Check-out */ exit(0); }

Compile and link the program $ pgcc -o compute_pi compute_pi.c -lpthread Execute it on the interactive node $ ./compute_pi Or, Execute it on batch nodes $ n1ge -fore -smp 2 ./compute_pi Result $ computed pi= 3.1415926535896208

4.3. Performance measurement tools As a part of PGI CDK, PGPROF, the performance measurement tool of parallel program is available on the TSUBAME system, which supports OpenMP and MPI. In addition, a debug tools PGDBG is also available on the TSUBAME system. 4.3.1. PGPROF (1) Compile and link programs ready for PGDBG $ pgf95 -Mprof=[func|lines] -o compute_pi compute_pi.f Options -Mprof=[func|lines] : -Mprof=func or -Mprof=lines option can be specified, and the default setting func is used if neither func nor lines option is specified. (2) Execute the program $ ./compute_pi A file named pgprof.out will be created on having executed the program (3) Start up PGPROF $ pgprof pgprof.out * Note: X windows system is necessary to startup PGPROF or PGDBG.

- 53 -

4.3.2.

PGDBG $ pgf95 -g -o compute_pi compute_pi.f Options -g generate executable program with symbolic debug information

(1) Compile and link programs to use PGDBG

(2) Start up PGDBG $ pgdbg compute_pi Debugging information is displayed.

- 54 -

5.

Applications
The execution method of an application is different from anothers due to different environment variables and license type. A CUI computation can be executed in background or foreground, but a GUI application can only be executed in foreground. You can get the newest help on submitting method with n1ge for all the ISV and free applications designed for n1ge batch system by typing the following command.

$ n1ge -apps_help [Application] The help on Application is displayed if Application is one of pre-registered applications. You can get the list of pre-registered applications by, $ n1ge -apps_list

5.1. 5.1.1.

ISV applications AVS/Express PCE AVS/Express PCE 7.0.1, 7.1.1 and 7.2 are installed in the TSUBAME system, in which

you have to log in a sles9 node described in (3) if you want to use old version 7.0.1. Please log in to an interactive node by using ssh X userID@login.cc.titech.ac.jp to execute a GUI application, AVS/Express PCE. (1) [Set X access limit on local X-terminal] Before accessing to TSUBAME, you should execute the command below. $ xhost + (2) [An execution example of AVS/Express] Get ready for a directory for test AVS/Express. tgg075017 > mkdir avspce_test Copy sample input files from /work/samples/avs_pce. tgg075017 > cd avspce_test/ tgg075017 > cp /work/samples/avs_pce/* . tgg075017 > ls app_extedges.v app_iso.v app_orth.v app_val_loop.v app_volrend.v app_extface.v app_isoline.v app_orth_grey.v app_vector.v app_volrend_grey.v Startup AVS/Express PCE [Version 7.1.1] The execution method is /usr/apps/isv10/avs/para_express_7.1.1/utl/para_start
$PARA_INI

INI_file Display_HOSTNAME userID

tgg075017 > /usr/apps/isv10/avs/para_express_7.1.1/utl/para_start 172.17.74.223 sun

[Version 7.2] The execution method is para_start INI_file - 55 -

Display_HOSTNAME userID

tgg075017 > para_start $PARA_INI 172.17.74.223 sun

Click Select Filethen select app_extedges.vfile, press OK button

Open the para app window, then click the button Calc Start

- 56 -

Quit the application by click the [File Exit] (3) Execute V7.01 on SLES9 nodes Log in a nodes with SLES9 $ ssh -Y login.cc.titech.ac.jp sles9 Startup AVS/Express PCE 7.0.1 $ /usr/apps/isv/avs/para_express/utl/para_start $PARA_INI 172.17.74.223 sun (4) About Display error Sometimes, the program can not be started up, with the error message: Error: Can't open display IP address:0.0. Please confirm the following points to find out the reason. Do you carry it out following the steps in the TSUBAME guide? AVS/Express PCE uses port 6000, and do you make sure if the port is available in the terminal used by you?

5.1.2. ABAQUS ABAQUS 6.5.6 ,6.7.1 and 6.8.4 are available on the TSUBAME system. You have to use the nodes with old settings (SLES9) if you want to use ABAQUS (6.5.6). Without setting the environment, ABAQUS 6.7.1 is used. (1) How to run ABAQUS 6.7.1 or 6.8.4 Before using ABAQUS 6.8.4, you have to set the environment variable ABAQUS_VERSION [Version 6.8.4] $ export ABAQUS_VERSION=6.8 or $ setenv ABAQUS_VERSION 6.8 [Foreground] For multi-CPU () The CPUS following the mpi should be equal the one which follows the cpus= option For 1CPU $ n1ge -fore abaqus job=INPUTFILE [Background] For multi-CPU () The CPUS following the mpi should be equal the one which follows the cpus= option - 57 $ n1ge -mpi CPUS abaqus job=INPUFILE cpus=CPUS $ n1ge -fore mpi CPUS abaqus job=INPUTFILE cpus=CPUS bash/ksh/sh csh/tcsh

For 1CPU

$ n1ge abaqus job=INPUTFILE (2) [An execution example] Make a directory for test jobs tgg075017 admin/sun> mkdir abaqus_test Copy a sample input file using abaqus fetch tgg075017 admin/sun> cd abaqus_test/ tgg075017 sun/abaqus_test> abaqus fetch job=damage_combine_deg Abaqus FETCH job damage_combine_deg Archive: /usr/apps/isv/abaqus/6.7-1/samples/job_archive/samples.zip inflating: damage_combine_deg.inp tgg075017 sun/abaqus_test> ls damage_combine_deg.inp Execute ABAQUS with the input file on 4CPUs tgg075017 sun/abaqus_test> n1ge -fore -mpi 4 abaqus job=damage_combine_deg cpus=4 *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of ABAQUS you specified is 6.7. *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3065681 on tgg075033] Abaqus JOB damage_combine_deg Abaqus Version 6.7-1 Begin Analysis Input File Processor Wed 07 May 2008 10:06:50 AM JST Run pre.exe Abaqus JOB damage_combine_deg COMPLETED [Ending Job 3065681] Connection to tgg075033 closed. Please refer to sles9queue. 5.1.3. ABAQUS CAE ABAQUS 6.5.6, 6.7.1 and 6.8.4 are available on the TSUBAME system. The ver. 6.7.1 will be used without setting execution environment. ABAQUS 6.5.6 is available only with SLES9. ABAQUS CAE can only be executed in foreground because it is a GUI application (1) Run ABAQUS CAE 6.7.1 or 6.8.4 Before using ABAQUS 6.8.4, you have to set the environment variable ABAQUS_VERSION [Version 6.8.4] $ export ABAQUS_VERSION=6.8 or $ setenv ABAQUS_VERSION 6.8 - 58 bash/ksh/sh csh/tcsh

(3) Run ABAQUS 6.5.6 on a node with the old settings

$ n1ge -fore abaqus cae [An execution example] Start up ABAQUS CAE tgg075017 sun/abaqus_test> n1ge -fore abaqus cae *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of ABAQUS you specified is 6.7. *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3065684 on tgg075034] Abaqus License Manager checked out the following license(s): "cae" version 6.7 from tggls <3 out of 5 licenses remain available>.

Click [File Exit] to quit (2) Run ABAQUS CAE 6.5.6 Please refer to sles9queue. 5.1.4. AMBER The AMBER 8 ,9 and AMBER 10 are available on the TSUBAME system. AMBER 10 is executed if without setting environment variables. (1) Run AMBER 9 or 10 Before using AMBER 9, you have to set the environment variable AMBER_VERSION - 59 -

[Version 9] $ export AMBER_VERSION=9 or $ setenv AMBER_VERSION 9

bash/ksh/sh csh/tcsh

The Amber 8 is available only on the nodes with old settings (SLES9). [Foreground] Execute a job with CPUS CPUs (including 1CPU) $ n1ge -fore -mpi CPUS sander -O -i INPUTFILE -o OUTPUTFILE [Background] Execute a job with CPUS CPUs (including 1CPU) $ n1ge -mpi CPUS sander -O -i INPUTFILE -o OUTPUTFILE For AMBER version 10, ClearSpeed(CS) version is also available . [Foreground] $ n1ge -fore -g BESgroup -q cs2 sander.CS -i INPUTFILE -o OUTPUTFILE O [Background] $ n1ge -g BESgroup -q cs2 sander.CS -i INPUTFILE -o OUTPUTFILE O -mpi option in the n1ge command cannot be specified. Moreover, the -smp option can be specified only with "-smp 1". (2) [An execution example with sample data] Make a directory for test jobs
tgg075017 admin/sun> mkdir amber_test

Copy sample files from /work/samples/amber

tgg075017 admin/sun> cd amber_test/ tgg075017 sun/amber_test> cp /work/samples/amber/* . tgg075017 sun/amber_test> ls inpcrd mdin prmtop

Execute an AMBER job using the sample data copied in

- 60 -

tgg075017 sun/amber_test> n1ge -fore -mpi 8 sander -O -i mdin -o test *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of AMBER you specified is 10. *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3560921 on tgg075037] [mpirund@tgg075037] The maximum size of Virtual Memory => 1048576 kb. Cleaning up all processes ... done. [Ending Job 3560921] Connection to tgg075037 closed. tgg075017 sun/amber_test> ls inpcrd mdin mdinfo prmtop profile_mpi restrt test tgg075017 sun/amber_test>

(3) AMBER 8 Please refer to sles9queue.

5.1.5. AVS Express Developer AVS/Express Developer 7.0.1, 7.1.1 and 7.2 are both installed in the TSUBAME system. The version 7.0.1 is available only on the nodes with old settings (SLES9). Before using AVS/Express 7.1.1, you have to set the environment variable AVS_VERSION Without setting the environment, AVS/Express Developer 7.2 is used. (1) Run AVS/Express Developer 7.11 or 7.2 Set the environment variable AVS_VERSION for 7.11, please skip this step for 7.2 $ export AVS_VERSION=7.1.1 or $ setenv AVS_VERSION 7.1.1 bash/ksh/sh csh/tcsh

AVS/Express Developer can only be executed in foreground because it is a GUI application $ n1ge -fore run_express [An example]

- 61 -

Start up Avs Express Developertgg075017 admin/sun> n1ge -fore run_express *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of AVS/Express you specified is 7.1.1. *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3065695 on tgg075033]
AVS/Express Developer Edition Version: 7.1.1 fcs linux_64 Project: /usr/apps/isv/avs/express_7.1.1 (read-only) Attempting to connect to license server (172.17.75.167:33333) Received license from server Checking hostid 0

Quit the application by clicking the [File Exit] button. (2) Run AVS/Express Developer Please refer to sles9queue. 5.1.6. EnSight EnSight can only be executed in foreground because it is a GUI application EnSight 8.0.7_k and 8.2.6_c are both installed in the TSUBAME system. Before using EnSight 8.0.7_k, you have to set the environment variable

ENSIGHT_VERSION to 8.0 $ export ENSIGHT_VERSION=8.0 or $ setenvENSIGHT VERSION 8.0 [Foreground] $ n1ge -fore ensight8 X () X option must be specified if the X11 server does not support OpenGL () If you use EnSight 8.2, you dont need to specify "-X" option. - 62 bash/ksh/sh csh/tcsh

[An example] Start up ensight8 tgg075017 admin/sun> n1ge -fore ensight8 -X *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of EnSight you specified is 8.2.6_c *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3065698 on tgg075033] Using CEI_HOME = /usr/apps/isv/ensight/ensight_8.2.6_c/CEI Using CEI_ARCH = linux_2.6_64 DISPLAY = localhost:11.0 hostname = tgg075033 <Starting with gldriver Mesa64 (Don't use -X for hardware rendering)>

Click the buttonFile Quit to 5.1.7. Gaussian

quit

The Gaussian03 Rev E01 is available on TSUBAME system. The Gaussian 03 Rev C02/D02 are available on the nodes with old settings (SLES9), please refer to the part (3) in this section.. (1) Run Gaussian 03 Rev. E01 [Foreground] For multi-CPU, single node (without Linda) For 1CPU $ n1ge -fore -smp CPUS g03 INPUTFILE $ n1ge -fore g03 INPUTFILE [Background] For multi-CPU, single node (without Linda) - 63 $ n1ge -smp CPUS g03 INPUTFILE

For 1CPU

$ n1ge g03 INPUTFILE For multi-CPU among nodes (with Linda)

$ n1ge -linda CPUS g03 INPUTFILE <--the CPUS must be times of 8 (16,32,) $ n1ge -linda CPUS-total:CPUS-node g03 INPUFILE --the CPUS-node may be 1-8,10,12,14, or 16, and the CPUS-total must be times of CPUS-node * A Gaussian job using two or more nodes must be submitted with -linda option.
* If the CPUS-node is 1-8, there is a Linda process in one node and SMPS equals to CPUS-node. There are two Linda processes in one node and SMPS equals to half of the CPUS-node if the

CPUS-node is 10,12,14, or 16.

[Execute a Gaussian job with a sample input file] Make a directory for test Gaussian jobs tgg075017 admin/sun> mkdir Gaussian_test Copy a sample file from the /usr/apps/isv/gaussian/g03/tests/com directory. tgg075017 sun/gaussian_test> cd gaussian_test tgg075017 sun/gaussian_test> cp $g03root/g03/tests/com/test000.com . tgg075017 sun/gaussian_test> ls test000.com Execute a sample job in foreground with 8 CPUs and 2GB memory. tgg075017 sun/gaussian_test> n1ge fore mem 2 -smp 8 g03 test000.com *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> Analyzing Input file for Gaussian.... *n1ge> The revision of Gaussian you specified is E.01. *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3065705 on tgg075033] Saving original input file as test000.com.3065705
real 0m39.745s user 0m23.492s sys 0m28.660s Restoring original input file from test000.com.3065705 [Ending Job 3065705] Connection to tgg075033 closed.

(2) Execute Gaussian 03 Rev C02 or D02. Please refer to sles9queue. 5.1.8. Gauss View Gauss View can only be executed in foreground because it is a GUI application. Gauss View 4.1.2 is available in the TSUBAME system. Gauss View 3.0.9 is available with old settings (SLES9). - 64 -

(1) Run Gauss View 4.1.2 Startup Gauss View 4.12. $ n1ge -fore gview [An Example] Start up Gauss View tgg075017 sun/ gaussian_test > n1ge -fore gview *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of GaussView you specified is 4.1 *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3065706 on tgg075034] [Ending Job 3065706] Connection to tgg075034 closed.

Click File Exitto quit. (2) [Execute GaussView with old settins] Please refer to sles9queue. 5.1.9. MATLAB MATLAB 7.1(R14SP3), 7.2(R2006a), 7.5(R2007b) and 7.8(2009a) are available on TSUBAME system. Of them, MATLAB 7.1 is available only on the nodes with the old settings. (1) Run MATLAB 7.2 ,7.5 or 7.8 Please set environment variables before executing MATLAB 7.2. $ export MATLAB_VERSION=7.2 bash/ksh/sh or $ setenv MATLAB_VERSION 7.2 $ export MATLAB_VERSION=7.8 or $ setenv MATLAB_VERSION 7.8 - 65 csh/tcsh bash/ksh/sh csh/tcsh

Without setting the environment, MATLAB 7.2 is used. [Foreground] CLI GUI $ n1ge -fore matlab nodisplay $ n1ge -fore matlab [Background]

In addtion, the ClearSpeed

version is available for Matlab7.2. Please refer to the Section 5.5 (ClearSpeed).

Execute a MATLAB batch job in background using a MATLAB script file samp.m. $ n1ge matlab -nodisplay -r samp [Examples] CLI Execute MATLAB in CLI mode
tgg075017 admin/sun> n1ge -fore matlab nodisplay *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of MATLAB you specified is 7.2(R2006a). *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3065712 on tgg075033] Warning: MATLAB is starting without a display, using internal event queue. You will not be able to display graphics on the screen. < M A T L A B > Copyright 1984-2005 The MathWorks, Inc. Version 7.2.0.283(R2006a) January 27,2006 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com. >> >> exit (to quit)

GUI Start up MATLAB in GUI mode. tgg075017 admin/sun> n1ge -fore matlab *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of MATLAB you specified is 7.2(R2006a). *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3065715 on tgg075034]

- 66 -

Execute a MATLAB program in batch mode as following

tgg075017 admin/sun> n1ge matlab -nodisplay -r samp *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of MATLAB you specified is 7.2(R2006a). *n1ge> Creating qsub options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... Your job 3065732 ("MATLAB") has been submitted

(2)

Run MATLAB 7.1 on the nodes with the old settings. Please refer to sles9queue.

5.1.10. MOPAC [Foreground] $ n1ge -fore -g BESGROUP mopac2006 INPUTFILE [Background] $ n1ge -g BESGROUP mopac2006 INPUTFILE [Execute a job with a sample input file] Make a directory for test MOPAC jobs
tgg075017 admin/sun> mkdir mopac_test

Copy a sample file from the /usr/apps/isv10/mopac/x86-64_LINUX/input/mopac. - 67 -

tgg075017 sun/mopac_test> cd mopac_test tgg075017 sun/mopac_test> cp $MOPAC_HOME/x86-64_LINUX/input/mopac/Test411.dat . tgg075017 sun/mopac_test> ls Test411.dat

Execute a MOPAC job with the sample data copied in the step .
tgg075017 sun/mopac_test> n1ge -g 4B***** -fore mopac2006 Test411.dat *n1ge> Checking about 4B***** group you specified.... *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> Creating qrsh options.... *n1ge> Submitting Job to Cluster...... [Starting Job 1679 on tgg073121] [Ending Job 1679] Connection to tgg073121 closed. tgg075017 sun/mopac_test> ls Test411.arc Test411.dat Test411.log Test411.out

[Note] You cant execute MOPAC jobs in batch mode, if you dont apply for the best effort services. 5.1.11. Mathematica The Mathematica 6.0, 7.0 are available on the TSUBAME system. Mathematica 5.2 can only run on the nodes with SLES9. (1) Run Mathematica 6.0 or 7.0 The command Mathematica can only be executed in foreground because it is a GUI interface command. And, the math or MathKernel can either be executed in interactive mode in foreground, or be executed in program mode in background. You have to set MATH_VERSION to use Mathematica 6.0 $ export MATH_VERSION=6.0.2 or $ setenv MATH_VERSION 6.0.2 [Foreground] CLI GUI $ n1ge -fore math $ n1ge -fore Mathematica [Background] $ n1ge math \< [Examples] CLI Execute Mathematica in CLI mode. - 68 sample.m \> sample.out for csh/tcsh for bash/ksh/sh

tgg075017 admin/sun> n1ge -fore math *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> Creating qrsh options.... *n1ge> Submitting Job to Cluster...... [Starting Job 3082982 on tgg074055] Mathematica 6.0 for Linux x86 (64-bit) Copyright 1988-2008 Wolfram Research, Inc. In[1]:= In[1]:= Quit

(type Quit to quit)

GUI Start up Mathematica in GUI mode

tgg075017 admin/sun> n1ge -fore Mathematica *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> Creating qrsh options.... *n1ge> Submitting Job to Cluster...... [Starting Job 1687 on tgg075033]

ClickFile Exit to quit Mathematica. Background Submit a background job by using Mathematica program.

- 69 -

Tgg075017 admin/sun> n1ge math \<samp.m \> samp.out *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> Creating qsub options.... *n1ge> Submitting Job to Cluster...... Your job 3082984 ("Mathematica") has been submitted

(2) Run Mathematica 5.2 Please refer to sles9queue. 5.1.12. Molpro [Foreground] For multi-CPU For 1CPU $ n1ge -fore -mpi CPUS molpro \< INPUTFILE $ n1ge -fore molpro \< INPUTFILE [Background] For multi-CPU For 1CPU $ n1ge -mpi CPUS molpro \< INPUTFILE $ n1ge molpro \< INPUTFILE [Execute a job with sample data] Make a directory for test Molpro jobs. tgg075017 admin/sun> mkdir molpro_test

(1) Copy a sample data from /work/samples/molpro. tgg075017 sun/molpro_test> cd molpro_test tgg075017 sun/molpro_test> cp /work/samples/molpro/molpro.in . tgg075017 sun/molpro_test> ls molpro.in (2) Execute a job with the sample data copied in the step with 2 CPUs.
tgg075017 sun/molpro_test> n1ge -fore -mpi 2 molpro \< molpro.in *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> Creating qrsh options.... *n1ge> Submitting Job to Cluster...... [Starting Job 1691 on tgg075033] MPP nodes nproc tgg075033 2 [Ending Job 1691] Connection to tgg075033 closed.

- 70 -

5.1.13. MD NASTRAN [Foreground] For multi-CPU For 1CPU $ n1ge -fore -mpi CPUS nastran INPUTFILE $ n1ge -fore nastran INPUTFILE [Background] For multi-CPU For 1CPU $ n1ge -mpi CPUS nastran INPUTFILE $ n1ge nastran INPUTFILE [Execute a sample job with sample data] Make a directory for test jobs. tgg075017 admin/sun> mkdir nastran_test

(1) Copy sample data from /work/samples/nastran. tgg075017 sun/nastran_test> cd nastran_test tgg075017 sun/nastran_test> cp /work/samples/nastran/bcell9a.dat . tgg075017 sun/nastran_test> ls bcell9a.dat (2) Execute a test job with the sample data copied in the step .
tgg075017 sun/nastran_test> n1ge -fore nastran bcell9a.dat *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of NASTRAN you specified is R2.1. *n1ge> Creating qrsh options.... *n1ge> Submitting Job to Cluster...... [Starting Job 3083003 on tgg074002] MD Nastran V2007.1 (AMD Linux 2.6.5-7.282-smp) Thu May 8 11:17:16 2008 *** SYSTEM INFORMATION MESSAGE (pgm: nastran, fn: estimate_job_requirements) Starting ESTIMATE, please wait... *** USER INFORMATION MESSAGE (pgm: nastran, fn: estimate_job_requirements) Estimated DOF=5308 Estimated memory=32.0MB Estimated disk=35.9MB Note: ESTIMATE had some difficulty analyzing this job. MD Nastran beginning job bcell9a. [Ending Job 3083003] Connection to tgg074002 closed. tgg075017 sun/nastran_test> ls bcell9a.DBALL bcell9a.MASTER bcell9a.dat bcell9a.f04 bcell9a.f06 bcell9a.log

- 71 -

5.1.14. MD PATRAN PATRAN can only be executed in foreground because its a GUI application. $ n1ge -fore patran [An example] Startup PATRAN tgg075017 sun/nastran_test> n1ge -fore patran *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> The version of PATRAN you specified is R2.1. *n1ge> Creating qrsh options.... *n1ge> Submitting Job to Cluster...... [Starting Job 3083004 on tgg075034]

Click File Quitto quit it.

5.1.15. SAS Please log in a interactive node to with the command ssh Y userID@login.cc.titech.ac.jp t sas to execute SAS. () SAS can only runs on the special interactive node, so it can not run on the batch nodes. tgg074117 admin/sun> sas

Click File Exitto quit it. - 72 -

5.1.16. Maple Maple 11 and 12 are available, and the Maple 11 is executed without version settings, so you have to set MAPLE_VERSION if you want to run Maple 11 as follows. [Set version for Maple 11] $ export MAPLE_VERSION=11 or $ setenv MAPLE_VERSION 11 for csh/tcsh for bash/ksh/sh

[Foreground] CLI GUI $ n1ge -fore maple $ n1ge -fore xmatlab [Background] $ n1ge maple -s -t \< <INPUTFILE> [Examples] CLI Execute Maple in CLI mode. You can exit it by type quit. tgg075017 admin/sun> n1ge -fore maple *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> MAPLE needs arguments when you execute batch mode. *n1ge> So MAPLE will run as foreground job. *n1ge> The version of MAPLE you specified is v12. *n1ge> Creating qrsh options.... *n1ge> Submitting Job to Cluster...... [Starting Job 3147857 on tgg075034] |\^/| Maple 12 (X86 64 LINUX) ._|\| |/|_. Copyright (c) Maplesoft, a division of Waterloo Maple Inc. 2007 \ MAPLE / All rights reserved. Maple is a trademark of <____ ____> Waterloo Maple Inc. | Type ? for help. > GUI Start up Maple in GUI mode

- 73 -

tgg075017 admin/sun> n1ge -fore xmaple *n1ge> Checking which tool have the command you specified.... *n1ge> Reading specific configuration file for each tools.... *n1ge> xmaple demands more than 1.5GB memory to start up. *n1ge> The memory is not specified. 2GB memory (-mem 2) is set automatically. *n1ge> The version of MAPLE you specified is v12. *n1ge> Creating qrsh options.... *n1ge> Warning: The runtime limit for novice queue is 1 hours. *n1ge> Submitting Job to Cluster...... [Starting Job 3147861 on tgg075034]

5.1.17. LS-DYNA The following four commands is adopted to the n1ge system. mppdynas mppdynad mppdynas_dfail mppdynad_dfail Execution Standard LS-DYNAusing mppdynas $ n1ge -mpi 4:1 mppdynas i=<INPUTFILE> LS-DYNA with ductile fracture model (using mppdynas_dfail ) $ n1ge -mpi 4:1 mppdynas_dfail i=<INPUTFILE> - 74 LS-DYNA with single precision LS-DYNA with double precision LS-DYNA with single precision (ductile fracture model LS-DYNA with double precisionductile fracture model

Post MPP program l2ausing l2as as a example $ /usr/apps/isv10/lsdyna/971R3/mod/mpp971_s_R3.44113_PGI_linux86-64_voltaire355.l2a binout0000 2Notices LS-DYNA uses its own queue system. The n1ge system accounts it while a job is submitted, and then it is forwarded to LS-DYNAs queue system. And the n1ge system begins charge the job at the stage. Please check your LS-DYNA job as described in the section 5.3.5 after submitting it. 5.1.18. FLUENT FLUENT 6.3.26 is available (1) Run FLUENT for multi-CPU $ n1ge -mpi 4:4 -fore fluent for 1CPU(single) $ n1ge -fore fluent [Forground]

- 75 -

5.2.

ISV applications (On Windows terminal) Material Explorer, Materials Studio, and Discovery Studio Modeling are executed on Windows

machine. 5.2.1. Materials Explorer On Windows, Click Materials Explorer to startup it. On the sub-window, input the necessary information as follows,
: a choice of 172.17.75.1722, : 10002 SelectNQS : Please input a queue name by the following rules. novice queue-q_novice bes1,bes2 queue-q_bes1_-g_BESGROUP, -q_bes2_-g_BESGROUP If you want to specify a job name Add _-N_JOBNAME : /usr/apps/isv10/MaterialsExplorer/binLinux64 : <INPUTFILE> : <OUTPUTFILE>

<> If a user of the BESGROUP 4B070000 submit a job to the bes1 queue with the job name TEST please input -q_bes1_-g_4B070000_-N_TEST. </> Please use a directory under your home directory or under the directory /work/yourID to save your input and output data.

5.2.2.

Materials Studio On Windows, Click Materials Studio to startup it, Press the [Gateway location] button in the [Job Control] menu and input the data as follows
Gateway Gateway Gateway Gateway location: location: location: location: 172.17.75.181_18888 (For Materials Studio 4.0) 172.17.75.181_18889 (For Materials Studio 4.1) 172.17.75.17_18888 (For Materials Studio 4.3) 172.17.75.17_18889 (For Materials Studio 4.4)

- 76 -

The license server is 27060@172.17.75.167. 5.2.3. Discovery Studio Modeling On Windows, ClickDiscovery Studio Modeling to start it up. Click FileChange Server...

Input the IP address of the computing server toServer name

Server Server Server Server Server name: name: name: name: name: 172.17.75.181:9941 (For Discovery Studio v1.5) 172.17.75.181:9944 (For Discovery Studio v1.6) 172.17.75.181:9945 (For Discovery Studio v1.7) 172.17.75.181:9947 (For Discovery Studio v2.0) 172.17.75.17:9943 (For Discovery Studio v2.1)

- 77 -

The license server is 27060@172.17.75.167. 5.3. Confirm the status of license of ISV Applications Sometimes, a specific ISV application is unavailable temporarily because all available licenses have been used up. Here, I show a method to identify the usage of those licenses. The licenses of the ISV applications installed on TSUBAME are managed in four kinds of license servers. License Server Application MD Nastran MD Patran MATLAB ABAQUS ABAQUS/CAE IMSL ClearSpeed INTEL Maple FLUENT Materials Studio Discovery Studio PGI SDK AVS/Express Developer AVS/Express PCE EnSight Mathematica License number/token number 10 10 210 80 5 10000 4 5 1 102 In total 40 tokens 25 20 2(Max parallel number is 64) 5 6000

FLEXlm

LmServer Slim MathLM

How to confirm the license usage depends on the kind of the license server. And I would show you how to confirm the usage of each kind license server. 5.3.1. To confirm usage of FLEXlm You can confirm license usage by the following steps for applications using FLEXlm license server. Log in TSUBAME confirm the usage of a license with the lmutil command The following command will display all features registered with FLEXlm. $ /usr/apps/isv10/flexlm/lmutil lmstat a

- 78 -

For examples
$ /usr/apps/isv10/flexlm/lmutil lmstat -a lmutil - Copyright (c) 1989-2006 Macrovision Europe Ltd. and/or Macrovision Corporation. All Rights Reserved. Flexible License Manager status on Thu 5/8/2008 11:27 License server status: 27010@tggls License file(s) on tggls: /licenses/license.dat_27010_MD: tggls: license server UP (MASTER) v10.8 Vendor daemon status (on tggls): MSC: UP v10.8 Feature usage info: Users of DYTRAN: Users of MARC: (Total of 10 licenses issued; (Total of 10 licenses issued; Total of 0 licenses in use) Total of 0 licenses in use) Total of 0 licenses in use)

Users of MARC_Electrical: Users of MARC_Hexmesh: Users of MARC_Mesh2D: Users of MARC_Mesh3D:

(Total of 10 licenses issued;

(Total of 10 licenses issued; (Total of 10 licenses issued; (Total of 10 licenses issued;

Total of 0 licenses in use) Total of 0 licenses in use) Total of 0 licenses in use) Total of 0 licenses in use) Total of 0 licenses in use) Total of 0 licenses in use)

Users of MARC_MetalCutting: Users of MARC_ShapeMemory: Users of MARC_ViewFactor: Users of Mentat:

(Total of 10 licenses issued; (Total of 10 licenses issued; (Total of 10 licenses issued;

(Total of 10 licenses issued;

Total of 0 licenses in use) Total of 0 licenses in use) Total of 0 licenses in use)

Users of Mentat_CMOLD:

(Total of 10 licenses issued;

Users of Mentat_ITI_Access: Users of MD_NASTRAN:

(Total of 10 licenses issued;

(Total of 10 licenses issued;

Total of 0 licenses in use) Total of 0 licenses in use)

Users of MD_ADAMS_Integration: Users of MD_Connectors: Users of MD_Dynamics: Users of MD_Thermal: Users of MD_DDAM:

(Total of 10 licenses issued;

(Total of 10 licenses issued; (Total of 10 licenses issued;

Total of 0 licenses in use) Total of 0 licenses in use)

(Total of 10 licenses issued;

Total of 0 licenses in use)

(Total of 10 licenses issued;

Total of 0 licenses in use)

Omit

- 79 -

You may also display a specific daemon as follows.

$ /usr/apps/isv10/flexlm/lmutil lmstat S <daemon> c <port>@tggls

The application and corresponding daemon and ports are listed as follows. Application MSC.NastranMSC.Patran MATLAB ABAQUSABAQUS/CAE IMSL PGI SDK Materials StudioDiscovery Studio ClearSpeed FLUENT Intel Compiler Maple Daemon MSC MLM ABAQUSLM VNI pgroupd msi Clearsp FluentLm INTEL Maplelmg Port 27010 27020 27030 27040 27050 27060 27070 27090 28010 28020

For example, type the following command to show ABAQUSLM(ABAQUS,ABAQUS/CAE).

$ /usr/apps/isv10/flexlm/lmutil lmstat -S ABAQUSLM -c 27030@tggls lmutil - Copyright (c) 1989-2004 by Macrovision Corporation. All rights reserved. Flexible License Manager status on Tue 4/17/2007 17:46 DAEMONs in configuration file: ABAQUSLM Users of features served by ABAQUSLM: Users of abaqus: (Total of 80 licenses issued; "abaqus" v6.6, vendor: ABAQUSLM floating license necap tgg075045 /dev/tty (v6.6) (tggls/27030 2517), start Tue 4/17 17:43, 5 licenses Users of aqua: Users of design: Users of parallel: Users of cae: (Total of 80 licenses issued; (Total of 80 licenses issued; Total of 0 licenses in use) Total of 0 licenses in use) Total of 0 licenses in use)

Total of 5 licenses in use)

(Total of 16384 licenses issued;

(Total of 5 licenses issued;

Total of 0 licenses in use) Total of 5 licenses in use)

Users of standard:

(Total of 80 licenses issued;

"standard" v6.6, vendor: ABAQUSLM floating license necap tgg075045 /dev/tty (v6.5) (tggls/27030 3476), start Tue 4/17 17:43, 5 licenses Users of explicit: Users of foundation: (Total of 80 licenses issued; (Total of 80 licenses issued; Total of 0 licenses in use) Total of 0 licenses in use)

- 80 -

5.3.2.

To confirm the usage of LmServer The following applications use LmServer. AVS/Express Developer AVS/Express PCE You can confirm the usage of LmServer for the applications using LmServer by the following steps. Log in TSUBAME, on starting X server. Start a web browser by using mozilla command. Access the following URL with the web browser, http://172.17.75.167:33333/STATUS The messages will be displayed.
LM Server Status 4/5/07 7:01 PM Use-limited License: EXPRESS limit: 20 currently used: 1 Use-limited License: EXPRESS_PCE limit: 1 currently used: 0 Client: EXPRESS 131.112.189.134 (131.112.189.134), started:4/5/07 4:59 PM, user:necap, pid:712

5.3.3.

To confirm the usage of slim The following application use slim as its License Server. EnSight You can confirm the usage of slim for the applications using slim, by the following steps. Log in TSUBAME type the slimd8_status command $ /usr/apps/isv10/ensight/CEI/bin/slimd8_status The messages will be displayed as follows. $ /usr/apps/isv10/ensight/CEI/bin/slimd8_status /usr/apps/isv10/ensight/CEI/license8/machines/linux_2.6_64/slimd8_status version 8.7. Getting slimd8 status from host <tggls>: Feature 'cei' total tokens 375 busy 0 idle 0 free 375 Feature 'ensightgold' total tokens 5 busy 0 idle 0 free 5 - 81 -

5.3.4.

To confirm the usage of MathLM

Users cannot confirm the license usage of applications using MathLM as License Serve 5.3.5. Usage of Lsdyna Set environment variables
$ export LSTC_LICENSE_SERVER=tggls

Run the following command

$ /usr/apps/isv10/lsdyna/license/dyna/lstc_qrun

An example $ /usr/apps/isv10/lsdyna/license/dyna/lstc_qrun No programs running No programs queued 5.4. 5.4.1. Free applications GROMACS Both the V3.3 and the V3.3.1 are installed on TSUABME. You can select V3.3 by specifying environment variable GROMACS_VERSION as follows. $ export GROMACS_VERSION=3.3 # for bash/ksh/sh or $ setenv GROMACS_VERSION 3.3 # for csh/tcsh

The V3.3.1 will be executed without specifying GROMACS_VERSION. The mdrun can be executed on the batch with n1ge nodes after the completion of a grompp command. [Foreground] For Multi-cpu (also for 1CPU) $ n1ge -fore -mpi CPUS mdrun -s XX.tpr -o XX.trr -c XX.gro -v -g LOGFILE \ -np CPUS () Please set two CPUS to one identical number. [Background] For Multi-cpu (also for 1CPU) $ n1ge -mpi CPUS mdrun -s XX.tpr -o XX.trr -c XX.gro -v -g LOGFILE \ -np CPUS 5.4.2. POV-Ray [Foreground] For 1CPU For murk-CPU - 82 $ n1ge -fore povray [options] INPUTFILE [options]

$ n1ge -fore -mpi CPUS povray [options] INPUTFILE [options] [Background] For 1CPU For multi-CPU $ n1ge povray [options] INPUTFILE [options] $ n1ge -mpi CPUS povray [options] INPUTFILE [options] LIMIT The version 3.5C is use if the option -mpi is specified, because PNG(Portable Network Graphics) files for multi-CPU jobs can not be outputted normorly.. 5.4.3. Tinker [Foreground] CLI Batch Example. $ n1ge fore timer bench1 10 y [Background] $ n1ge timer Coordinate_file Repetitions Hessian_Evaluations 5.4.4. UTChem [Foreground] $ n1ge -fore -mpi CPUS utchem -w SCRATCH_DIRECTORY INPUTFILE [Background] $ n1ge -mpi CPUS utchem -w SCRATCH_DIRECTORY INPUTFILE 5.4.5. NWChem First, copy the setting file for NWChem to your home directory $ cp /usr/apps/free10/nwchem4.7/data/default.nwchemrc $HOME/.nwchemrc Then, specify scratch directory. A directory in the directory /gau will be created and used as scratch directory without specifying scratch directory. In addition, specify a temporary directory, unless the current directory will be used to storing temporary files. You can specify a temporary directory as follows, A sample of data file start h2o_freq charge 1 $ n1ge -fore timer $ n1ge -fore timer Coordinate_file Repetitions Hessian_Evaluations

start h2o_freq permanent_dir charge 1 - 83 -

/work/USER-ID

Submit your job as follows. [Foreground] $ n1ge -fore -mpi CPUS nwchem INPUTFILE [Background] $ n1ge -mpi CPUS nwchem INPUTFILE 5.4.6. Gamess The versions 22 FEB 2006(R5), 24 MAR 2007(R3), and 11Apr2008(R1) are available on TSUBAME. You can select the version 11 Apr 2008(R1) by specifying environment variable GAMESS_VERSION as follows, or you will use the version 11Apr 2008(R1). to use version 25MAR2007(R3). $ export GAMESS_VERSION=2007R3 or $ setenv GAMESS_VERSION 2007R3 [Background] Set scratch directory for Gamess, or a directory in the directory /scr/JOBID/ will be created and used as scratch directory. Dont specify your home directory or a directory in it as a scratch firectory. $ export SCR=/work/USER-ID/scrh Submit a Gamess job as follows, Execute the following command, if you dont need profiler. $ n1ge -ddi CPUS rungms INPUTFILE or $ n1ge -ddi total-CPUS:nodo-CPUS rungms INPUTFILE Execute the following command, if you want to ouput profiler. $ n1ge -ddi CPUS rungms_prof INPUTFILE or $ n1ge -ddi total-CPUS:nodo-CPUS rungms_prof INPUTFILE Generally, Run GAMESS with the command rungms JOB VERNO NCPUS And, the options following the command rungms are not been taken as batch options by the N1GE system. So please omit the VERNO option and specify the CPUs with the option. -ddi as submitting an n1ge job. In addition, the -ddi option has the same format as -mpi one, -ddi num_cpu[:node_cpu]. - 84 -

bash/ksh/sh csh/tcsh

Without setting the environment, you use 11 Apr 2008(R1).

And, specify memory for each process with the "-mem" option, which is same as the option used in an mpi job. The versions 22 FEB 2006(R5) only runs on the node with SLES9. $ssh Y login.cc.titech.ac.jp t sles9 Run gamess $ /usr/apps/free/gamess/rungms

- 85 -

5.5.

ClearSpeed The ClearSpeed board is accelerator made by ClearSpeed Inc., and it is available for the programs satisfying the following conditions. Programs using Blas/Lapack libraries sander.CS (tuned by ClearSpeed Inc.) Matlab programs using ISML library (being investigated) All the computing nodes are equipped with the ClearSpeed accelerators, but CS batch jobs can be submitted to the following queues, sla1, sla2, sla3, inno1, inno2, inno3, pinno1, pinno2, pinno3,cs1,cs2 Therefore, the batch queues that can use the accelerators are the above queues. Besides, there is an interactive queue for execution of ClearSpeed debugging. The queue is csdebug queue. You can use this queue by connecting to TSUBAME as follows. $ ssh t User-ID@login.cc.titech.ac.jp csdebug

5.5.1. The usage with program sources The accelerators are available in the way below, if the programs refer to the Blas/Lapack libraries by dynamic link. 1) Set the environments for compiling $ source /usr/apps/isv10/clearspeed/3.11/bin/bashrc 2) Link the program with the ClearSpeed libraries for C $ g++ -lm lg2c -lpthread -L/usr/apps/isv10/clearspeed/3.11/lib lcsxl_acml -o sample sample.c for Fortran $ pgf90 -lm -lg2c -lpthread -L/usr/apps/isv10/clearspeed/3.1/lib \ -L/usr/apps/isv/pgi/linux86-64/6.1/libso -lcsxl_acml -o pre_lapack.cs pre_lapack.f 3) Set the environments for execution $ source /usr/apps/isv10/clearspeed/3.11/bin/bashrc $ export CS_HOST_BLAS=/usr/apps/isv10/pgi/linux86-64/7.2-4/libso/libacml.so $ export CS_BLAS_HOST_ASSIST_PERCENTAGE=10 Set the variable CS_BALS_HOST_ASSIST_PERCENTAGE to specify how work will be shared between the host processors and the ClearSpeed accelerator board. If you set it to 0, then 100% of a BLAS call will be performed on the ClearSpeed accelerator board. If 100, then 100% of a BLAS call will be done on the host processors. The variable is usually set to 515. 4) Execution $ csreset Av $ ./sample [options] 5.5.2. The usage without program sources - 86 -

Notes: The program has to been dynamic linked, or the CS is not been used. 1) Set the environment valuables $ source /usr/apps/isv10/clearspeed/3.11/bin/bashrc $ export LD_PRELOAD=/usr/apps/isv10/clearspeed/3.11/lib//libcsxl_acml.so $ export CS_HOST_BLAS=/usr/apps/isv10/pgi/linux86-64/7.2-4/libso/libacml.so $ export CS_BLAS_HOST_ASSIST_PERCENTAGE=10 CS_BALS_HOST_ASSIST_PERCENTAGErefer to the above description 2) Execution
$ csreset Av $ ./program [options]

5.5.3. Making Clear Speed execution module by SDK Here, the procedure from the compilation of the sample program to execution is shown. 1) Setting environment variables $ source /usr/apps/isv10/clearspeed/3.11/bin/bashrc $ export CS_HOST_BLAS=/usr/apps/isv10/pgi/linux86-64/7.2-4/libso/libacml.so $ export CS_BLAS_HOST_ASSIST_PERCENTAGE=10 The csdebug queue is used. 2) Compilation(using sample program ) $ cp -fr /usr/apps/isv10/clearspeed/3.11/examples/sdk/sinewave . $ cd sinewave $ cscn sinewave_mono.cn The extension of the source program is *.cn, and the extension of execution module is *.csx. 3) $ csreset -Av $ csrun ./a.csx 0.0000000 0.032719083 : Return value = 0 For details please refer to ClearSpeed Users' Guide ( PDF in Opening IO room of GSIC) . 5.5.4. Use ClearSeed on batch nodes You have to write a job script in which the ClearSpeed environment setting, CS Initialization, and execution line of the program are arranged, then submit it cs2 queue or SLA queue,in order to submit a batch job to execute your program with CS library. As an example, it is shown that a CS job is submitted with a program linked with the CS library. files on TSUBAME or a manual

- 87 -

1) to write a job script $ cat go.sh #!/bin/sh source /usr/apps/isv10/clearspeed/3.11/bin/bashrc export CS_HOST_BLAS=/usr/apps/isv10/pgi/linux86-64/7.2-4/libso/libacml.so export CS_BLAS_HOST_ASSIST_PERCENTAGE=10 csreset Av ./a.out 2) Submit (according to the kind of charge groups) Single program $ n1ge g <BESgroup> q cs2 go.sh -q cs2 must be specified for BES group $ n1ge g <SLAgroup> rt <minutes> go.sh -rt option must be specified for SLA MPI program $ n1ge g <BESgroup> q cs2 mpi <nodes> go.sh $ n1ge g <SLAgroup> mpi <nodes> rt <minutes> go.sh

5.5.5. isv application with ClearSpeed 1) MATLAB The CS version of Matlab 7.2 is available, $ source /usr/apps/isv10/clearspeed/3.11/bin/bashrc $ matlab_csx_cs311

Please submit a Matlab job as follows, to use CS library, Job script $ cat matlab_ex.sh source /usr/apps/isv10/clearspeed/3.11/bin/bashrc matlab_csx_cs311 nodisplay r sample # where, sample.m is a Matlab program. Submit $ n1ge g <BESgroup> q cs2 matlab_ex.sh 2) Mathematica The CS version of Mathematica 5.2 is available using the following steps, $ source /n1ge/TITECH_GRID/tools/bin/n1ge_bin/apps_rc/Mathematica.5.2.rc $ source /opt/clearspeed/csx600_m512_le/bin/bashrc $ math_csx

3) AMBER Please refer to ! AMBER (p.! - 88 -

- 89 -

5.6.

TESLA TESLA accelerators are available via the TSUBAME batch queues listed below. You can run

your GPU program developed by CUDA tool kit and programs using CUBLAS and CUFFT libraries that are CUDA port of BLAS, FFT libraries. For the use of best effort service queues, it is pre-required to register the users and programs for system stability. tesladebug (for interactive use) sla1tes2, sla2tes2, sla3tes2 (service level agreement queues) bes1tes2, bes2tes2 (best effort service queues *now preparing) (1) How to login tesladebug nodes tesladebug nodes are available for TESLA programming and debugging. You can login and use them interactively in the following manner. $ ssh Y t User-ID@login.cc.titech.ac.jp tesladebug (2) Development environments CUDA tool kit which includes a compiler driver and libraries is installed on /opt/cuda directory. The environment strings required to use them are automatically set when you login the tesladebug nodes. CUDA SDK that includes various CUDA sample programs (source files and executables) is installed on /usr/apps/nosupport10/NVIDIA_CUDA_SDK directory. (ex. 1) Compiling your GPU program with CUDA tool kit. $ nvcc myprog.cu o myprog (ex. 2) Compiling your CUBLAS program. $ gcc I/opt/cuda/include L/opt/cuda/lib o myprog myprog.c -lcublas (ex. 3) Compiling your CUFFT program. $ gcc I/opt/cuda/include L/opt/cuda/lib o myprog myprog.c -lcufft (3) Submitting batch jobs Submitting your TESLA jobs to sla?tes2 and bes?tes2 queues are done by the same way as other programs, with assigning TESLA queues. To submit bes?tes2 queues, the users and programs must be registered beforehand, otherwise cause an error on submitting. (*bes?tes2 queues are under preparing) $ n1ge q sla?tes2 g SLA group ID (other n1ge options) myprog $ n1ge q bes?tes2 g BES group ID (other n1ge options) myprog

For more details on how to use TESLA, please refer to TESLA Users Guide on the GSIC Website. - 90 -

6. MAIL
The portal email system (@m.titech.ac.jp) has been deployed in April, 2007 which supports SSL pop3/IMAP and SSL SMTP. http://portal.titech.ac.jp/ezguide/mailsetup.html#server-info And, the mail service (o.cc.titech.ac.jp and n.cc.titech.ac.jp ) provided by TSUBAME has been obsolete.

- 91 -

7. Application for the system

You can apply for a new account or modify your account information from the following page, URL http://www.gsic.titech.ac.jp/~ccwww/index.html

You may access the following web page to modify your application. URLhttps://gsicriyou.cc.titech.ac.jp/UZUserMenu.htm

- 92 -

8. Question and answer

If you have questions, please mail to the following email address. Global Scientific Information and Computing Center, Tokyo Institute of Technology E-Mailsoudan@cc.titech.ac.jp And please often read news published by GSIC. URLhttp://www.gsic.titech.ac.jp/

Please include the following information in a question mail. Name, Department, Grade Mail address to reply to Purpose Machine or Hostname Login name in the system Files and directory on the questionPlease change their permission to be accessible from support members Error messagespresent) Detailed command or operation Urgent or not

Note : Do not send mail on debug questions, please.

- 93 -

9. English Version History

Date 00 01 02 1.7 2006.07.18 2006.08.28 2006.11.10 2006.11.15 1.4

Objects ALL objects

Introduction New, based on Japanese version 1.4 Based on Japanese version 1.6

Queue Change Based on Japanese version 1.7 Unitary description on CPU Add: some new applications Limits on batch nodes and interactive nodes Modify: disk space queue change/limit on memRank0 in an MPI job Add: some n1ge options Add: batch job of Mathematica Add: servers and port System draft Modify: on ClearSpeed Add: Discovery Studio Modeling v1.6 Delete: DALTON Modify: Queues Modify: Queues Delete: /work_bk Delete: Mail server Delete: DALTON Modify: Queue Modify: use situation of queues Modify: how to delete a job Modify: Execution Add: on Intel compilers Add: License Server Name Delete: Set Mail Modify: Section 4.4 into Chapter 5 Add: Confirm available queues Update to TSYBAME 1.2 1.5.1, 1.5.2 2.1.2, 3.4.3 2.1.4

1.8

2007.03.15

3.5.2 3.5.7 3.5.9 4.4.1(11) 4.4.1(16) 1.1 1.2.2 1.5.1 1.5.2 2.1.1 2.1.4 2.1.7 2.2 2.2.1(3) 3.5.2 3.5.4 3.5.5 3.5.10 4.1.1 4.4.1(18) 5

1.9

2007.05.02

2.0 2.1 2.2

2007.05.02 2007.05.21 2007.8.14

4.4 3.5.3 All objects

3.5.9(3.5.8/V2.0) Modify: Memory restriction on batch jobs

- 94 -

Date 3.3.5 2.3 2007.08.27 5.1.2 5.1.3 2.1.4 2.4 2007.9.12 3.5.3 5.1 5.1.9

Objects

Introduction Modify: Lustre environments Add: ABAQUS 6.7.1 Add: ABAQUS 6.7.1 Modify: Working directory Modify: Confirm available queues Add: New version Add: MATLAB 7.1 Add: MSC/NASTRAN 2007.0(2007r1) Add: MSC/PATRAN 2006(2006r1) Add: EnSight 8.2.6_c Add: proc option Add: Gauss View 4.1.2 Add: GROMACS 3.3.1 Add: Gamess 24MAR 2007(R3) select Ouput profiler or not Add:Compiler version , other compiler Add:SDK Add:Sander.CS Modify: Intel compilers version Modify: proc option Modify: Materials Studio Modify: NWchem Modify: GAMESS Modify: Default change in application Modify: AVS/Express Modify: MD NASTRAN Modify: MD PATRAN Add: Gaussian 03 Rev. E01 Add: Maple 11

5.1.13 2.5 2007.11.16 5.1.14 5.1.6 3.5.8 2.6 2007.11.30 5.1.8 5.4.1 5.4.6 2.7 2007.12.14 4.1.2 , 4.1.3 5.5.3 5.1.4 1.5.1 , 4.1.3 2.8 2008.01.29 3.5.8 5.2.2 5.4.5 2.9 3.0 2008.02.18 5.4.6 2008.04.03 All 5.1.1 , 5.1.5 3.1 2008.05.08 5.1.13 5.1.14 5.1.7 3.2 2008.05.19 5.1.16

- 95 -

Data 3.3 3.4 3.5 3.6 2008.05.27 2008.06.09 2008.07.01 2008.07.25 5.4.6 3.5.9 3.5 5.1.4 1.1.2 2.1.4

Objects

Introduction Add: Gamess 11 Apr 2008(R1) Modify: on MPI parameter file Modify: Using n1ge command Array substitution of item Add: AMBER 10 Modify: nodes with ClearSpeed card Modify: Disk space Modify: Batch queue Modify: On sander.CS Modify: Matalab Modify: ClearSpeed Add: batch job on ClearSpeed Add: Matlab CS

3.5.2,3.5.3 3.7 2008.8.28 5.1.4 5.1.9 5.5 4.0 4.1 4.2 4.3 4.4 2008.11.4 2008.12.09 2008.12.26 2009.01.20 2009.04.08 2009.05.18 All items 5.2.2 5.2.3 5.6 3.5.7 5.1.17 All items

Add: Materials Studio 4.3 Add: Discovery Studio 2.1 Add: TESLA Modify: description on MPI parameters Add: Notice part Update for new fiscal year Add: ABAQUS 6.8.4 5.1.2,5.1.3,5.1.9 Add: MATLAB 7.8 ,5.1.12 Add: Molpro 2002.6

- 96 -

A User Guide of TSUBAME

Global Scientific Information and Computing Center (GSIC) In Tokyo Institute of Technology 2009/05 English Ver. 4.4

If you have questions on the document, please mail to soudan@cc.titech.ac.jp.

- 97 -