<div class="gmail_quote">Hi Everyone!<div><br></div><div>We are Sorry for the EVO technical problems faced for today's meeting. Here is the summary of the meeting for the people who missed the presentation.</div><div>The presentation is about Pilot-Jobs and their usage on LONI & XSEDE machines.</div>
<div><br></div><div><b>Problem statement:</b></div><div> 1. With traditional PBS systems on an average or highly busy systems like LONI or XSEDE, we face queue wait time for each job we submit. A simulation with bunch of jobs takes more time to complete, since there is a waiting time involved for each job. </div>
<div> 2. It is difficult to monitor free HPC resources and submit jobs using PBS. This involves manual process like logging onto each HPC resource and checking the queue status and then submitting the job.</div><div><br>
</div><div><b>Solution:</b></div>
<div>
Pilot Job is an effective abstraction to handle the above two problems. </div><div> Pilot Job is a container job to which a number of tasks assigned upfront. The user's responsibility is just to mention the HPC resources(could be multiple) and their attributes ( like walltime, ppn, working directory) which need to be used and set of jobs/tasks ( number of processes, spmd_variation, working directory, input, output, error) to be executed.</div>
<div> The Pilot-Job once become active takes care of assigning jobs by distributing them to the free resources among the resources requested. </div><div><br></div><div><b>Advantage:</b></div><div> The Pilot-Job uses the resources effectively in two ways.</div>
<div> 1. Scaling simulations over multiple HPC resources is easy.</div><div> 2. Allows execution of jobs without necessity of queuing each individual job.</div><div> 3. There is no waiting time involved between task, i.e when a job is completed, the next job is scheduled without waiting in the queue<font face="Verdana, Arial, 'Bitstream Vera Sans', Helvetica, sans-serif">. This continues </font><span class="Apple-style-span" style="font-family: Verdana, Arial, 'Bitstream Vera Sans', Helvetica, sans-serif; ">until all the requested tasks are completed or walltime requested is completed.</span></div>
<div><font face="Verdana, Arial, 'Bitstream Vera Sans', Helvetica, sans-serif"><br>
</font></div><div><font face="Verdana, Arial, 'Bitstream Vera Sans', Helvetica, sans-serif"><b>Types of Pilot-Jobs:</b></font></div><div><font face="Verdana, Arial, 'Bitstream Vera Sans', Helvetica, sans-serif"> 1. ManyJob </font></div>
<div><font face="Verdana, Arial, 'Bitstream Vera Sans', Helvetica, sans-serif"> 2. BigJob </font></div><div><font face="Verdana, Arial, 'Bitstream Vera Sans', Helvetica, sans-serif"><br>
</font></div>
<div><b>Requirements of Pilot-Jobs:</b></div><div> ManyJob needs python (>=2.4) , ssh/gsissh authentication to/from remote cluster head nodes to the host machine without password by use of password keys(ssh key) . </div>
<div>
BigJob requries SAGA, SAGA dependencies(like boost, postgresql), SAGA python bindings & python ( preferrably >=2.4)</div><div><br></div><div><b>Questions asked</b></div><div><br></div><div><b>BigJob questions:</b></div>
<div><br></div><div>1. Can BigJob perform the Input data transfer required, to the the remote HPC cluster where job need to be executed?</div><div> No, Its is not a part of BigJob framework to perform the input data transfer required for the jobs/simulations. The application which uses bigjob should perform the data transfer by using ftp tools like scp/gridftp.</div>
<div><br></div><div>2. Can BigJob be used on only eric?</div><div> No, It can be installed & executed on any HPC cluster which have SAGA , its dependencies and its python bindings. </div><div> Please see example </div>
<div> - <a href="https://svn.cct.lsu.edu/repos/saga-projects/applications/bigjob/trunk/generic/examples/example_manyjob_local.py" target="_blank">https://svn.cct.lsu.edu/repos/saga-projects/applications/bigjob/trunk/generic/examples/example_manyjob_local.py</a></div>
<div><br></div><div>3. <span style="font-family:sans-serif;font-size:13px;line-height:19px"> I want to run a job for each input file. For example; there are 45 files, each input file requires 20 hours to process and 1 node. process per node of HPC resource is 4. The max wall time allowed is 48hrs and 128 cores. Requesting more nodes involve more initial queue waiting. So what is the optimal way I can use bigjob to run all the jobs in 40hrs.</span></div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> There are many ways to handle this usecase.</span></div><div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> 1. Use a "single" bigjob to request 45 nodes(180 processors) - 20hrs(wall time) - which will complete all jobs in 20hrs. But getting 180 processors might not be possible & lot of initial queue waiting time is involved.</span></div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> 2. Use "two" BigJobs, where each BigJob request 23 nodes(92 processor) - 20hrs(wall time) - which will complete 23 jobs in 20hrs( easy to get 92 processors & less waiting time ). The second BigJob will complete 22 jobs in 20 hrs. So all jobs are completed in 20hrs with reduced initial queue waiting time.</span></div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> 3. Use a "single" BigJob to request 23 nodes and 40 hrs(wall time) - which will schedule first 23 jobs parallelly at a time. Once 23 jobs are completed the remaining 22 jobs are scheduled automatically, and there is no waiting time in involved in doing that. </span></div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"><br></span></div><div><span style="font-family:sans-serif;font-size:13px;line-height:19px"><b>ManyJob questions:</b></span></div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> 1. Can we schedule ManyJob to start a single job after 3 jobs completed their execution, if their execution time varies?</span></div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> No. We can't define multiple dependencies for a single job.</span></div><div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> 2. </span>Can ManyJob perform the Input data transfer required, to the the remote HPC cluster where job need to be executed?</div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> Yes. </span></div><div><span style="font-family:sans-serif;font-size:13px;line-height:19px"> </span></div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"><span style="font-family:arial;line-height:normal;font-size:small"><div>Manuals & Tutorial:</div>
<div> <span style="font-family:Verdana,Arial,'Bitstream Vera Sans',Helvetica,sans-serif">1. ManyJob - (<a href="http://dna.engr.latech.edu/ManyJobs/" target="_blank">http://dna.engr.latech.edu/ManyJobs/</a>)</span></div>
<div> <span style="font-family:Verdana,Arial,'Bitstream Vera Sans',Helvetica,sans-serif">2. BigJob - (<a href="http://faust.cct.lsu.edu/trac/bigjob/" target="_blank">http://faust.cct.lsu.edu/trac/bigjob/</a>)</span></div>
<div><br></div><div>Please send your questions & use cases </div><div> BigJob - <a href="mailto:bigjob-users@cct.lsu.edu" target="_blank">bigjob-users@cct.lsu.edu</a> </div><div> ManyJob - <a href="mailto:rajibm@latech.edu" target="_blank">rajibm@latech.edu</a> , <a href="mailto:fuji@tulane.edu" target="_blank">fuji@tulane.edu</a>, <a href="mailto:bishop@latech.edu" target="_blank">bishop@latech.edu</a></div>
</span></span></div><div><span style="font-family:sans-serif;font-size:13px;line-height:19px"><br></span></div><div><span style="font-family:sans-serif;font-size:13px;line-height:19px">Please find the today's tutorial at </span><a href="https://wiki.cct.lsu.edu/lasigma-execution-management/images/2/23/Dec2Tutorial_V2.ppt" target="_blank">Dec2Tutorial_V2.ppt</a><span style="font-family:sans-serif;font-size:13px;line-height:19px"> </span><span style="font-family:sans-serif;font-size:13px;line-height:19px">and for more details refer to the BigJob & ManyJob websites or write to the mailing lists mentioned above.</span></div>
<div><span style="font-family:sans-serif;font-size:13px;line-height:19px"><br></span></div><div> thanks</div><span class="HOEnZb"><font color="#888888"><div>pradeep</div><div><br></div>
</font></span></div><br>