a d '@sddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z ddl mZddlmZddlZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&m'Z'e dZ(e(rddl)m Z*Gddde Z+dS)) ExitStack)AnyDictListOptionalTupleUnionN)module_available)Tensor) Optimizer) CheckpointIO)_distributed_available)group)ReduceOp)LightningOptimizer)PrecisionPlugin)ParallelStrategy) TBroadcast)MisconfigurationException)rank_zero_deprecationrank_zero_onlyz horovod.torchcseZdZdZdZd;edeeejee ee dfdd Z e e dd d Ze e dd d Ze e dd dZe ejdddZe eeefdddZe edddZdddfdd ZeeddddZdHorovodStrategyz4Plugin for Horovod distributed training integration.horovodNzpl.accelerators.Accelerator) acceleratorparallel_devices checkpoint_ioprecision_plugincs<tdtstdtj||d||d|jt_d|_dS)Nz`The `HorovodStrategy`: `Trainer(strategy='horovod')` has been deprecated in v1.9.0 and will be removed in v2.0.0. You can try using the `Trainer(strategy='ddp')` instead.zRequested `strategy="horovod"`, but Horovod is not installed. Install with `HOROVOD_WITH_PYTORCH=1 pip install horovod[pytorch]`)rrZcluster_environmentrr) r_HOROVOD_AVAILABLErsuper__init__ global_rankrrank _exit_stack)selfrrrr __class__m/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/pytorch_lightning/strategies/horovod.pyr-s zHorovodStrategy.__init__)returncCstSN)hvdr!r#r&r&r'r GszHorovodStrategy.global_rankcCstSr))r* local_rankr+r&r&r'r,KszHorovodStrategy.local_rankcCstSr))r*sizer+r&r&r' world_sizeOszHorovodStrategy.world_sizecCst|jtsJ|j|jSr)) isinstancerlistr,r+r&r&r' root_deviceSszHorovodStrategy.root_devicecCst|j|jd}|S)N)Z num_replicasr!)dictr.r )r#distributed_sampler_kwargsr&r&r'r3Xsz*HorovodStrategy.distributed_sampler_kwargscCsdS)zt|t_j|js0dSttdddj}fdd|D}|D]"}|j D]}|dj 9<qfq\j }|D]*}|j }t |drfdd|jD|_qjdusJtjjd d |D]}tj|d d q|j}|jd gkr td ||j_jD]}j|q"dS) N)optr(cSst|tr|jS|Sr))r/rZ _optimizer)r6r&r&r'_unpack_lightning_optimizernsz:HorovodStrategy.setup.._unpack_lightning_optimizercsg|] }|qSr&r&.0r6)r7r&r' rz)HorovodStrategy.setup..lrbase_lrscsg|]}|jqSr&)r.)r9r<r+r&r'r:r;r)Z root_rankz[Horovod currently does not support different `accumulate_grad_batches` at different epochs.)model_to_devicersetuprr" __enter__Ztrainingr optimizers param_groupsr.lr_scheduler_configs schedulerhasattrr=lightning_moduler*Zbroadcast_parametersZ state_dictZbroadcast_optimizer_stateaccumulation_schedulerZepochsr_wrap_optimizersaccumulate_grad_batches enter_contextZskip_synchronize) r#r5rA optimizerZ param_grouprCconfigrDrGr$)r7r#r'r?bs:     zHorovodStrategy.setup)argskwargsr(cOstr|dSr))r join)r#rMrNr&r&r'barrierszHorovodStrategy.barrierr)objsrcr(cCst||}|Sr))r*Zbroadcast_object)r#rQrRr&r&r' broadcasts zHorovodStrategy.broadcastcCs:|jjdkrtj|j|jdus(J|j|jdSNcuda)r1typetorchrUZ set_devicemodeltor+r&r&r'r>s zHorovodStrategy.model_to_devicecCs&|jjdkrt|jntdSrT)r1rVr*rOr,r+r&r&r'rOs zHorovodStrategy.joinmean)tensorr reduce_opr(cCsZ|durtd|dvr tj}n$|dtjfvr6tj}ntd||tj||dS)aReduces a tensor from several distributed processes to one aggregated tensor. Args: tensor: the tensor to sync and reduce group: the process group to gather results from. Defaults to all processes (world) reduce_op: the reduction operation. Defaults to 'mean'/'avg'. Can also be a string 'sum' to calculate the sum during reduction. Return: reduced value, except when the input was not a tensor the output remains is unchanged NzWHorovod does not support allreduce using a subcommunicator at this time. Unset `group`.)NavgrZsumzunrecognized `reduce_op`: )op) ValueErrorr*ZAveragerZSUMZSumrOZ allreduce)r#r[rr\r&r&r'reduceszHorovodStrategy.reduceF)resultr sync_gradsr(cCsD|dur|tjkrtdt|jdkr2|d}|t|S)NzWHorovod does not support allgather using a subcommunicator at this time. Unset `group`.r) dist_groupWORLDr`lenshapeZreshaperOr*Z allgather)r#rbrrcr&r&r' all_gathers  zHorovodStrategy.all_gather) closure_lossr(cCs|jD] }|qdSr))rAZ synchronize)r#rjrKr&r&r' post_backwards zHorovodStrategy.post_backwardzhvd.DistributedOptimizer)rArIr(cs"jdusJfdd|DS)z?Wraps optimizers to perform gradient aggregation via allreduce.Nc s8g|]0}dt|jvr0tj|j|dn|qS)r)Zbackward_passes_per_stepnamed_parameters)strr%r*ZDistributedOptimizer_filter_named_parametersrFr8rIr#r&r'r:s  z4HorovodStrategy._wrap_optimizers..)rF)r#rArIr&ror'rHs z HorovodStrategy._wrap_optimizers)rXrKr(cs&dd|jDfdd|DS)NcSs"h|]}|dgD]}|qqS)params)get)r9rpr&r&r' r;z;HorovodStrategy._filter_named_parameters..cs g|]\}}|vr||fqSr&r&)r9namerrZ opt_paramsr&r'r:r;z.)rBrl)rXrKr&rur'rnsz(HorovodStrategy._filter_named_parameters)strategy_registryr(cCs|j|j||jjddS)N) description)register strategy_namer%__name__)clsrvr&r&r'register_strategiess z#HorovodStrategy.register_strategiescs2|jr|jdddd|_|tdSr))r"__exit__rOrteardownr+r$r&r'r~s zHorovodStrategy.teardown)NNNN)r)NrZ)2rz __module__ __qualname____doc__ryrrrWZdevicer rrpropertyintr r,r.r1rrmrr3boolr4r?rPrrSr>rOrr rrarerfrirkr rH staticmethodnnModuler Parameterrn classmethodr|r~ __classcell__r&r&r$r'r(s^ 0     $r), contextlibrtypingrrrrrrrWZtorch.nnrZ lightning_utilities.core.importsr r Z torch.optimr Zpytorch_lightningplZlightning_fabric.pluginsr Z&lightning_fabric.utilities.distributedr rreZ lightning_fabric.utilities.typesrZ pytorch_lightning.core.optimizerrZ#pytorch_lightning.plugins.precisionrZ%pytorch_lightning.strategies.parallelrZ%pytorch_lightning.strategies.strategyrZ&pytorch_lightning.utilities.exceptionsrZ%pytorch_lightning.utilities.rank_zerorrrZ horovod.torchr*rr&r&r&r's*