Code Architecture

This document describes the overall code architecture.

litesoph_stack

Full Picture of the litesoph backend:

           +---------+
           |LSManager|
           +---------+
         ______|_______
        |              |
        V              V
ProjectInfo <---------ProjectManager
    |                   |
    V                   V
WorkflowInfo <--------WorkflowManager
    |                   |
    |                   V
    |                  EngineManager
    |                     |
    V                     V
TaskInfo  <-------------Task
                    _______|______
                    |             |
                    V             V
                SubmitLocal   SubmitNetwork
                                    |
                                    V
                            NetworkJobSubmission
class litesoph.common.ls_manager.LSManager[source]

This is the main interface to the litesoph backend. This class is responsible for creating and managing all the projects in the litesoph. All the data generated from the project is stored in the project_info which is serialized into json format and written to the project_data file in the .litesoph directory, which is in the project directory.

new_project(name, path, description='')[source]
Creates Project directory and instantiates ProjectInfo and project manager

then return project manager.

Parameters:

  • name (str) – name of the project.

  • path (str | Path) – path of the project directory.

  • description (str) – description of the project.

Return type:

ProjectManager object.

open_project(path)[source]

Opens a litesoph project.

Parameters:

path (str | Path) – path of the project directory

Return type:

ProjectManager object.

read_config()[source]

Reads the lsconfig file and converts it into dict.

class litesoph.common.project_manager.ProjectManager(ls_manager, project_info)[source]

This class is responsible for creating, loading and managing all the workflows in the projects.

Parameters:

  • ls_manager – LSManager object loaded with a project.

  • project_info (ProjectInfo) – ProjectInfo to store all the information generated in the project.

clone_workflow(workflow_info_uuid, traget_workflow_type, branch_point, label, description='')[source]

This method creates new workflow by clone any workflows in the project into

Parameters:

  • workflow_info_uuid (str) – The uuid of the source workflow

  • traget_workflow_type (str) – The workflow type of the cloned workflow.

  • branch_point (int) – It’s is the block id in the workflow. It is the point upto which workflow is cloned.

  • label (str) – label of the new cloned workflow

  • description (str) – description or comments about the cloned workflow.

get_workflow_info(workflow_uuid)[source]

returns WorkflowInfo object from the uuid.

list()[source]

returns list of tuples contain label and uuids of all the workflows in the project.

list_available_workflows()[source]

Returns a list of all the predefined workflow types.

new_workflow(label, description='')[source]

This method creates a new workflow info and saves it in the project.

Parameters:

  • label (str) – user given name of the workflow

  • description (str) – description or comments about the workflow.

open_workflow(workflow_uuid)[source]

This method opens already existing and defined workflow.

remove(workflow_uuid)[source]

This removes a workflow from the project. It will also delete the corresponding directory.

start_workflow(workflow_type, param)[source]

This method instantiates the workflow manager with the workflow_type and returns the workflow manager.

Parameters:

  • workflow_type (str) – The workflow indentifier.

  • param (Dict[str, Any]) – parameters.

class litesoph.common.workflow_manager.WorkflowManager(project_manager, workflow_info, config)[source]

This is the main interface to edit, modify and run workflows.

In litesoph, workflow is modeled as a chain of blocks, where each block contains a list of simple tasks that the user can create and run.

For example, consider the average spectrum workflow. We represent the average spectrum workflow as a chain of four blocks, where each block contains the same types of tasks but with different input parameters.

    Block-1                  Block-2                  Block-3                  Block-4
|---------------|      |-----------------|      |--------------------|    |----------------|
|               |      |                 |      |                    |    |                |
| 1. ground     |----> |  2. RT-TDDFT- x |----->| 5.compute-spectra-x|--->| 8. compute     |
|    state      |      |  3. RT-TDDFT- y |      | 6.compute-spectra-y|    | average spectra|
|---------------|      |  4. RT-TDDFT- z |      | 7.compute-spectra-z|    |----------------|
                       |-----------------|      |--------------------|

The dependenices_map maps how each simple task depends on the previous tasks. so, for above workflow

1 --> None : ground state doesn't depend on any tasks
2 --> 1    : RT-TDDFT-x dependents on ground state.
3 --> 1    : RT-TDDFT-y dependents on ground state.
4 --> 1    : RT-TDDFT-z dependents on ground state.
5 --> 2    : compute_spectra-x dependents on RT-TDDFT-x.
6 --> 3    : compute_spectra-y dependents on RT-TDDFT-y.
7 --> 4    : compute_spectra-z dependents on RT-TDDFT-z.
8 --> (5, 6, 7) : compute average spectra dependent on compute_spectra-x.
                    compute_spectra-y and compute-spectra-z.
Parameters:

  • project_manager – The class that manages creation and deletion of workflows:

  • workflow_info (WorkflowInfo) – This objects is used to store all the information about a workflow.

  • config (Dict[str, str]) – The configurations used to run the tasks in the workflow.

add_block(block_id, name, store_same_task_type=False, task_type=None, metadata={})[source]

This method inserts a block into the workflow.

Parameters:

  • block_id (int) – The index where block to be palce in the workflow.

  • name (str) – The name of the block.

  • store_same_task_type (bool) – the variable which indicative if the block contain same type of the tasks.

  • task_type – task type if the store_same_task_type is true.

  • metadata – This stores information about the tasks in the blocks in the context of the workflow.

add_dependency(task_uuid, dependent_tasks_uuid=[])[source]

Adds a dependency task list to the given task.

add_task(task_name, block_id, step_id, parameters={}, env_parameters={}, dependent_tasks_uuid=[])[source]

This method adds a task into the workflow.

Parameters:

  • task_name (str) – The task type.

  • block_id (int) – The index of the block to which the task to be added.

  • step_id (int) – The index in the task execution list to where task to be added.

  • paremeter – The default input parameters of the task.

  • env_parameters – This stores information about the task in the context of the workflow.

  • dependent_tasks_uuid (str | list) – The list of task_uuids to which the task dependents on.

check_engine(engine)[source]

Checks whether a given engine implements the current workflow type

choose_default_engine()[source]

Chooses a default engine for a given workflow type and sets the engine for the workflow.

clone(clone_workflow, branch_point)[source]

This method clones a new workflow_info from the existing workflow_info. It clones the task_info from the existing workflow_info and with that it copies all the input and output files generated from that task into the new directory of the cloned task.

get_engine_task()[source]

This method returns the Task object of the current_task in the workflow.

get_task_dependencies()[source]

Returns a list of previous task_infos that the present task in dependent on.

get_taskinfo(task_name)[source]

This method returns the list of TaskInfo object with the given name in the workflow.

next()[source]

This method changes the current task to the next task in the workflow.

save()[source]

saves the workflow_info into the hard drive.

set_engine(engine)[source]

sets the engine of the workflow

class litesoph.common.engine_manager.EngineManager[source]

Base class for all litesoph engines.

Parameters:

  • Name – name of the engine.

  • implemented_tasks (List[str]) – list of task identifiers of the tasks that are implemented.

  • implemented_workflows (List[str]) – list of workflow identifiers of the workflows that are implemented.

check_task(name)[source]

This method checks if a given task is implemented in this engine if not it raises a TaskNotImplementedError

abstract get_default_task_param(name, dependent_task)[source]

This mentods return the default parameters for a given particular task.

abstract get_task(config, workflow_type, task_info, dependent_task)[source]

This class checks if a task is implemented and if it is implemented it returns the task object else raises a TaskNotImplementedError.

get_task_list()[source]

This method returns the list of implemented tasks in this engine

get_workflow_list()[source]

This method returns the list of implemented tasks in this engine

class litesoph.common.Task(lsconfig, task_info, dependent_tasks=None)[source]

Base-class for all tasks. All the tasks in the litesoph inherets from this class. This class is interfaced with SubmitLocal and SubmitNetwork class.

Parameters:

  • lsconfig – The configuration dictionary the has information about path of engine binarys.

  • task_info (TaskInfo) – This is the empty TaskInfo object to store all the data generated by the task.

  • dependent_tasks (List[TaskInfo] | None) – This is a list of TaskInfo objects of other tasks that this task is dependent on.

abstract create_template()[source]

This method creates engine input and stores it in the task info. Store the engine input in taskinfo.input[‘engine_input’][data] filepath in taskinfo.input[‘engine_input’][‘path’]

abstract write_input()[source]

This method creates engine directory and task directory and writes the engine input to a file.

create_job_script()[source]

Create the bash script to run the job and “touch Done” command to it, to know when the command is completed.

add_proper_path(path)[source]

This replaces the local paths to remote paths in the engine input.

class litesoph.common.data_sturcture.data_classes.ProjectInfo(_uuid, label, path, description='', config=<factory>, workflows=<factory>)[source]

This class stores all the information about a project.

Parameters:

  • label (str) – Name of the project.

  • path (pathlib.Path) – Path to the project directory.

  • description (str) – string describing the project.

  • config (Dict[Any, Any]) – configuration used in the project.

  • workflows (List[litesoph.common.data_sturcture.data_classes.WorkflowInfo]) – List of all the workflow_info of workflows in the project.

class litesoph.common.data_sturcture.data_classes.WorkflowInfo(_uuid, label, path, _name='', description='', engine=None, task_mode=False, param=<factory>, steps=<factory>, containers=<factory>, state=<factory>, dependencies_map=<factory>, tasks=<factory>, current_step=<factory>)[source]

This class store all the information of a workflow.

The workflow is modeled ordered sequence of blocks, where each contains a list of same type of task but with different parameters. For example Average specrtrum workflow is:

block_1(Ground state tasks) -> block_2(RT TDDFT tasks) -> block_3(compute spectrum tasks) -> block_4(compute average spectrum)
        ground_state               rt tddft in x             compute spectrum in x                   compute average spectrum
                                    rt tddft in y             compute spectrun in y
                                    rt tddft in z             compute spectrun in z
Parameters:

  • label (str) – User given name of the workflow.

  • path (pathlib.Path) – Path to workflow directory.

  • name – The type of workflow, for example: spectrum, ksd.

  • description (str) – string, description about the workflow.

  • engine (str | None) – The engine used in the workflow.

  • task_mode (bool) – If false, the workflow comes with a defined sequence of tasks. For example, the spectrum workflow is: ground_state -> RT TDDFT -> compute spectrum. If true, the user is given full control over the workflow to add any kind of task to it.

  • param (Dict[Any, Any]) – any parameters related to the workflow.

  • steps (List[litesoph.common.data_sturcture.data_classes.Block]) – It’s a list of blocks.

  • containers (List[litesoph.common.data_sturcture.data_classes.Container]) – It’s a list of containers. In a workflow, each task is associated with a container, which stores information about the task in context of the workflow. Currently, this list is used to navigate one task to another in a workflow.

  • state (litesoph.common.data_sturcture.data_classes.State) – It stores information about what tasks are running and what step the workflow is in. (currently this variable is not in use)

  • dependencies_map (Dict[str, str]) – It’s a dictionary that maps each tasks with the list of tasks that it depend on it.

  • tasks (Dict[str, litesoph.common.data_sturcture.data_classes.TaskInfo]) – It’s a dictionary that maps that task uuid with the task_info objects.

  • current_step (list) – It’s a list that store the current task the workflow is in.

class litesoph.common.data_sturcture.data_classes.TaskInfo(_uuid, _name, engine=None, state=<factory>, path=None, task_data=<factory>, param=<factory>, engine_param=<factory>, input=<factory>, output=<factory>, local_copy_files=<factory>, remote_copy_files=<factory>, job_info=<factory>, network=<factory>, local=<factory>)[source]

This class stores all the information about a task.

Parameters:

  • name – Task identifier.

  • engine (str | None) – Which engine to use to run the task.

  • state (litesoph.common.data_sturcture.data_classes.State) – store information about the state of the task.

  • path (pathlib.Path | None) – path of the workflow directory.

  • task_data (Dict[Any, Any]) – It store any miscellaneous information about the task that depends on the engine.

  • param (Dict[Any, Any]) – The input parameters of the task.

  • engine_param (Dict[Any, Any]) – The input parameters of the task in the format of the engine used.

  • input (Dict[Any, Any]) – It’s a dictionary store that stores input files generated for the task.

  • output (Dict[Any, Any]) – It stores output files generated for by the task.

  • local_copy_files (List[Any]) – list of relative paths of files to be copied to clone a task.

  • remote_copy_files (List[Any]) – list of relative paths of files to be copied from the remote machine.

  • job_info (litesoph.common.data_sturcture.data_classes.JobInfo) – Containes all the information about submitting and running the job. (This is new feature in development)

  • network (Dict[Any, Any]) – Contains information about the job that was submitted to network.

  • local (Dict[Any, Any]) – Contains information about the job that was submitted locally. (network and the local variable will be removed once the job_info is incorporated.)