a  d0 @shddlZddlZddlZddlZddlmZddlmZ d ddZ ddZ dddZ d d Z d d ZdS)NncclcKs`tjdddurtd|dkr4t|fi|n(|dkrNt|fi|ntd|dS)NT) allow_nonespawnZpytorchZslurmzInvalid launcher type: )mpZget_start_methodZset_start_method_init_dist_pytorch_init_dist_slurm ValueError)launcherbackendkwargsr `/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/basicsr/utils/dist_util.py init_dist s rcKs@ttjd}tj}tj||tjfd|i|dS)NRANKr ) intosenvirontorchcuda device_count set_devicedistinit_process_group)r r ranknum_gpusr r r rs rcCsttjd}ttjd}tjd}tj}tj||td|d}|durjt |tjd<ndtjvrvn dtjd<|tjd <t |tjd <t ||tjd <t |tjd <t j |d dS)aInitialize slurm distributed training environment. If argument ``port`` is not specified, then the master port will be system environment variable ``MASTER_PORT``. If ``MASTER_PORT`` is not in system environment variable, then a default port ``29500`` will be used. Args: backend (str): Backend of torch.distributed. port (int, optional): Master port. Defaults to None. Z SLURM_PROCIDZ SLURM_NTASKSZSLURM_NODELISTzscontrol show hostname z | head -n1NZ MASTER_PORTZ29500Z MASTER_ADDRZ WORLD_SIZEZ LOCAL_RANKr)r ) rrrrrrr subprocess getoutputstrrr)r portZproc_idZntasksZ node_listraddrr r r rs      rcCs<trt}nd}|r,t}t}nd}d}||fS)NFr)rZ is_availableZis_initializedZget_rankZget_world_size)Z initializedrZ world_sizer r r get_dist_info<s  r!cstfdd}|S)Ncs$t\}}|dkr |i|SdS)Nr)r!)argsr r_funcr r wrapperLs zmaster_only..wrapper) functoolswraps)r%r&r r$r master_onlyJsr))r)N)r'rrrZtorch.distributedZ distributedrZtorch.multiprocessingmultiprocessingrrrrr!r)r r r r s