a d@sddlZddlZddlZddlmZmZmZmZmZddl m Z ddl m Z ddl mZe dZGdddeZeed d d Zeeeeefd d dZdS)N)AnyCallableOptionalSequenceTuple)RequirementCache)ClusterEnvironment) _Launcherz hydra-corecsneZdZdZdeeddfdd Zeeddd Ze e e e d d d Z ddd dZ ddddZ ZS)_SubprocessScriptLauncheraA process laucher that invokes the current script as many times as desired in a single node. This launcher needs to be invoked on each node. In its default behavior, the main process in each node then spawns N-1 child processes via :func:`subprocess.Popen`, where N is the number of devices (e.g. GPU) per node. It is very similar to how :mod:`torch.distributed.run` launches processes. For example, if the script gets invoked with the command .. code-block:: bash python train.py --devices 4 The launcher will create three additional subprocesses that get called like so: .. code-block:: bash LOCAL_RANK=1 python train.py --devices 4 LOCAL_RANK=2 python train.py --devices 4 LOCAL_RANK=3 python train.py --devices 4 It is implied that the main process which launched the others has ``LOCAL_RANK=0``. Beside the local rank, the following other environment variables also get set, but unlike the local rank, these get determined by the cluster environment: 1. `MASTER_ADDR`: The IP address of the main node. 2. `MASTER_PORT`: The port number of the main node through which all processes communicate. 3. `NODE_RANK`: The index of the node the current process is running on. Ranges from 0 to ``num_nodes - 1``. 4. `WORLD_SIZE`: The total number of processes across all nodes, i.e., ``num_processes * num_nodes``. Arguments: cluster_environment: A cluster environment that provides access to world size, node rank, etc. num_processes: The number of processes to launch in the current node. num_nodes: The total number of nodes that participate in this process group. rN)cluster_environment num_processes num_nodesreturncs t||_||_||_dS)N)super__init__r r r )selfr r r  __class__/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/lightning_fabric/strategies/launchers/subprocess_script.pyr@s z"_SubprocessScriptLauncher.__init__rcCsdS)NFrrrrris_interactive_compatibleKsz3_SubprocessScriptLauncher.is_interactive_compatible)functionargskwargsrcOs|jjs|||i|S)aCreates new processes, then calls the given function. Arguments: function: A callback function to execute after all processes have been created. It is up to the implementation of this function to synchronize the processes, e.g., with barriers. *args: Optional positional arguments to be passed to the given function. **kwargs: Optional keyword arguments to be passed to the given function. )r Zcreates_processes_externally_call_children_scripts)rrrrrrrlaunchOs z _SubprocessScriptLauncher.launchcCs||jjtjd<t|jjtjd<t|jtjd<t|jtjd<|j |j tjd<t d|j D]}tj }||d<tj ddurd|vr|d=d}d}trd d lm}|}|rt|d \}}nt}tj|||d qpdS) NZ MASTER_ADDRZ MASTER_PORTZ NODE_RANKZ LOCAL_RANKZ WORLD_SIZEZPL_GLOBAL_SEEDFr) HydraConfig) local_rank)envcwd)_check_can_spawn_childrenr Z main_addressosenvironstrZ main_portZ node_rankr r r rangecopyget_HYDRA_AVAILABLEZhydra.core.hydra_configrZ initialized_hydra_subprocess_cmd_basic_subprocess_cmd subprocessPopen)rr Zenv_copyZ hydra_in_user"rcommandrrrr\s(   z0_SubprocessScriptLauncher._call_children_scriptscCs|jdkrtddS)NraLightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.)r r RuntimeErrorrrrrr#sz3_SubprocessScriptLauncher._check_can_spawn_children)__name__ __module__ __qualname____doc__intrpropertyboolrrrrrr# __classcell__rrrrr s&  #r rcCsZddl}|jdur8tjtjtjdgtjddStjd|jjgtjddSdS)Nrr-m) __main____spec__sys executabler$pathabspathargvname)r:rrrr,s &r,)r rcCsddl}ddlm}m}|jdur8tj|tjdg}ntjd|jjg}|tjdd7}|}dt d}|d|d|g7}||fS)Nr)get_original_cwdto_absolute_pathr9r"zhydra.run.dir=z!hydra.job.name=train_ddp_process_) r:Z hydra.utilsrBrCr;r<r=r@rAr$getcwd)r r:rBrCr/r"Zos_cwdrrrr+s r+)r$r-r<typingrrrrrZ lightning_utilities.core.importsrZ9lightning_fabric.plugins.environments.cluster_environmentrZ*lightning_fabric.strategies.launchers.baser r*r r&r,r5r+rrrrs   m