a d@sddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZeeZGddde Zed d d ZdS) N)Optional)ClusterEnvironment) _IS_WINDOWS)rank_zero_warn)PossibleUserWarningcs6eZdZdZd)eeejddfdd Ze eddd Z e e dd d Z e e dd d ZeedddZeee dddZeee dddZe dddZe ddddZe dddZe ddddZe dddZe dd d!Zee e d"d#d$Zeddd%d&Zeddd'd(ZZS)*SLURMEnvironmentaCluster environment for training on a cluster managed by SLURM. Args: auto_requeue: Whether automatic job resubmission is enabled or not. How and under which conditions a job gets rescheduled gets determined by the owner of this plugin. requeue_signal: The signal that SLURM will send to indicate that the job should be requeued. Defaults to SIGUSR1 on Unix. TN) auto_requeuerequeue_signalreturncs<t||_|dur"ts"tj}||_||dS)N) super__init__rrsignalSIGUSR1r _validate_srun_used_validate_srun_variables)selfrr  __class__t/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/lightning_fabric/plugins/environments/slurm.pyr )s  zSLURMEnvironment.__init__r cCsdS)NTrrrrrcreates_processes_externally2sz-SLURMEnvironment.creates_processes_externallycCs<tjdd}||}|tjd<tdtjd|S)NZSLURM_NODELISTz 127.0.0.1Z MASTER_ADDRz MASTER_ADDR: )osenvirongetresolve_root_node_addresslogdebug)rZnodelistZ root_noderrr main_address6s   zSLURMEnvironment.main_addresscCsttjd}|dur.|dd}t|d}nd}dtjvrLttjd}nt|tjd<tdtjd|S)N SLURM_JOB_IDi:in2Z MASTER_PORTz MASTER_PORT: )rrrintstrrr)rjob_id default_portrrr main_port>s   zSLURMEnvironment.main_portcCsttS)aReturns ``True`` if the current process was launched on a SLURM cluster. It is possible to use the SLURM scheduler to request resources and then launch processes manually using a different environment. For this, the user can set the job name in SLURM to 'bash' (``SLURM_JOB_NAME=bash``). This will then avoid the detection of ``SLURMEnvironment`` and another environment can be detected automatically. )rr _is_srun_usedrrrrdetectYs zSLURMEnvironment.detectcCs tjdS)NZSLURM_JOB_NAME)rrrrrrrjob_nameeszSLURMEnvironment.job_namecCsPtdk}|rdStjd}|dur,dSz t|WStyJYdS0dS)Nbashr )rr)rrrr" ValueError)Zin_slurm_interactive_moder$rrrr$is    zSLURMEnvironment.job_idcCsttjdS)N SLURM_NTASKSr"rrrrrr world_sizexszSLURMEnvironment.world_size)sizer cCstddS)Nz[SLURMEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.rr)rr/rrrset_world_size{szSLURMEnvironment.set_world_sizecCsttjdS)NZ SLURM_PROCIDr-rrrr global_rank~szSLURMEnvironment.global_rank)rankr cCstddS)Nz]SLURMEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.r0)rr3rrrset_global_ranksz SLURMEnvironment.set_global_rankcCsttjdS)NZ SLURM_LOCALIDr-rrrr local_rankszSLURMEnvironment.local_rankcCsttjdS)NZ SLURM_NODEIDr-rrrr node_rankszSLURMEnvironment.node_rank)nodesr cCs4tdd|}tdd|}|ddddS)aThe node selection format in SLURM supports several formats. This function selects the first host name from - a space-separated list of host names, e.g., 'host0 host1 host3' yields 'host0' as the root - a comma-separated list of host names, e.g., 'host0,host1,host3' yields 'host0' as the root - the range notation with brackets, e.g., 'host[5-9]' yields 'host5' as the root z\[(.*?)[,-].*\]z\1z \[(.*?)\] r,)resubsplit)r7rrrrs z*SLURMEnvironment.resolve_root_node_addresscCs^trdStddu}|rZtsZddtjtj gtj dd}t d|dt ddS)aTChecks if the `srun` command is available and used. Parallel jobs (multi-GPU, multi-node) in SLURM are launched by prepending `srun` in front of the Python command. Not doing so will result in processes hanging, which is a frequent user error. Lightning will emit a warning if `srun` is found but not used. NZsrunr8@zThe `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: z ...)category) rshutilwhichr'joinrpathbasenamesys executableargvrr)Z srun_existshintrrrrs &z$SLURMEnvironment._validate_srun_usedcCs>ttjdd}|dkr:dtjvr:td|d|ddS) a1Checks for conflicting or incorrectly set variables set through `srun` and raises a useful error message. Right now, we only check for the most common user errors. See `the srun docs `_ for a complete list of supported srun variables. r,1ZSLURM_NTASKS_PER_NODEzYou set `--ntasks=z^` in your SLURM bash script, but this variable is not supported. HINT: Use `--ntasks-per-node=z ` instead.N)r"rrr RuntimeError)Zntasksrrrrsz)SLURMEnvironment._validate_srun_variables)TN)__name__ __module__ __qualname____doc__boolrr Signalsr propertyrr#rr"r& staticmethodr(r)r$r.r1r2r4r5r6rrr __classcell__rrrrrs4    rrcCsdtjvotdkS)Nr,r*)rrrr)rrrrr'sr')loggingrr:r?r rDtypingrZ9lightning_fabric.plugins.environments.cluster_environmentrZ"lightning_fabric.utilities.importsrZ$lightning_fabric.utilities.rank_zerorZ#lightning_fabric.utilities.warningsr getLoggerrKrrrOr'rrrrs