LITESOPH Job Submission, Monitoring and File Management

LITESOPH Job Submission, Monitoring and File Management is a set of modules capable of handling tasks like submission of job to remote machine, tracking the status running job and transferring of files from remote to local machine.

Features

  1. Job Submission module supports queue (qsub, sbatch etc) as well as non-queue (bash) based job submission on remote machine through password and passwordless approach. Passwordless approach is generally used to avoid typing password repeatively for every job submission. It can also help to bypass captcha based login authentication. See How to enable passwordless SSH?

  2. Monitoring and File Management modules handles the job of tracking the status as well as terminating the running job on remote machine. File Management module handles the downloading of all files as well as selective transfer the files from remote to local machine via different transfer methods (direct transfer along with compression and split transfer).

Functionalities

  1. Function to submit a job at local as well as remote machine

  2. Function to check the running status of job submitted at local machine as well as remote machine

  3. Function to get the information about files generated by running project at local as well as remote machine.

  4. Function to view different files generated by running job at local as well as remote machine.

  5. Function to kill running job at remote machine.

  6. Function to selectively transfer files with varying methods (direct, compress, split).

Modules, Functions and Working

LITESOPH Job Submission, Runtime Query and File Management modules depends on two main Functionalities. Function to make connection with remote machine and execute commands on it via password as well as passwordless login authentication.

Furthermore, Runtime Query and File Management uses mainly four modules to work:

  • Module litesoph/common/submit.py handles methods requires to establish connection between local and remote machine, submitting jobs at local and remote machine, and downloading files.

  • Module litesoph/common/lfm_database.py defines are all predefined rules, methods and database required for file management and query related Functionalities.

  • Module litesoph/gui/task_controller.py handles methods related to runtime query for local as well as remote job submission.

  • Module litesoph/gui/views.py handles the GUI for all backend Functionalities related to LITESOPH Job Submission, Runtime Query and File Management.

  1. Module: litesoph/common/job_submit.py

    • function execute_cmd_local(): execute command locally by using argument command and directory.

    • function execute_cmd_remote(): execute command on remote machine through local machine shell.

    • class SubmitLocal: handles local job submission.

    • class SubmitNetwork: handles remote job submission.

    • function download_files_from_remote()

      • get the list of files from remote directory

      • convert the list of files into file-metadata-dictionary

      • filter the dictionary into sub-dictionary of priority-levels

      • give each file priority level according to the assigned tags of files

      • classify the files on the basis of priority and transfer priority1 files first

      • determine the method to transfer the files

      • transfer the file

  2. Module: litesoph/common/lfm_database.py

    • function lfm_file_info_dict(): generate a dictionary database of containing metadata information of all the possible files.

    • varibles: include python list and dictionary type variables different file tags used for generating master file metadata information.

  3. Module: litesoph/gui/task_controller.py

    • class TaskController: handles individual task in a workflow or a task in task mode.

      • function _on_run_network_button() : handles set of functions responsible for job submission and monitoring at remote machine

        • function _on_create_remote_job_script()

        • function _on_save_job_script()

        • function _run_network()

        • function _on_check_job_status_remote()

        • function _on_check_file_status_remote()

        • function _on_kill_job_remote()

        • function _on_kill_job_remote()

        • function _on_download_all_files()

        • function _on_download_specific_file()

      • function _on_run_local_button() : handles set of functions responsible for job submission and monitoring at local machine

        • function _on_create_local_job_script()

        • function _on_save_job_script()

        • function _run_local()

        • function _on_check_job_status_local()

        • function _on_check_file_status_local()

        • function _on_view_specific_file_local()

  4. Module: litesoph/gui/views.py

    • class JobSubPage: handles job submission GUI page.

      • function runtime_query_local(): handles GUI buttons related to runtime query for local job submission

      • function runtime_query_remote(): handles GUI buttons related to runtime query for remote job submission

      • function show_run_local(): handles GUI buttons related for remote job submission

      • function show_run_network(): handles GUI buttons related for remote job submission