a dE @sjdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZmZmZmZmZmZmZdd lmZdd lmZmZmZdd lmZd d lm Z m!Z!m"Z"m#Z#m$Z$d dl%m&Z&d dl'm(Z(m)Z)erd dl*m+Z+e#,e-Z.e/dZ0dZ1dddZ2eGdddZ3e$d>ee3ee4e4e4ee4eee5ee5fdddZ6Gddde&Z7Gddde&Z8deee4ddd d!Z9e5d"d#d$Z:e5d%d&d'Z;e5d(d)d*Ze4dd.d/d0Z?ee3e>ee4d1d2d3Z@eee4e8d4d5d6ZAdee4e>eed7d8d9ZBdee4e>eed7d:d;ZCGdsz'post_lfs_batch_info..r) operationZ transfersobjectsZ hash_algoZ access_tokenT)headersjsonauthrSzMalformed response from servercSsg|]}d|vrt|qSerror)_validate_batch_actionsrPobjr+r+r,rQcSsg|]}d|vrt|qSrW)_validate_batch_errorrZr+r+r,rQr\) rrrpost LFS_HEADERSrrrrUget isinstancelist ValueError) rBrCrDrErFZ url_prefixZ batch_urlrespZ batch_inforSr+r+r,post_lfs_batch_info^s8#      rec@seZdZUeed<eed<dS) PayloadPartT partNumberetagN)r9r:r;r?r>rAr+r+r+r,rfs rfc@s&eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rNpartsN)r9r:r;r<rAr>r rfr+r+r+r,ris rir)rRlfs_batch_actionrCrGc CsFt||d}|dur2td|jddS|dd}t||dd}|durdt||di}|d}|durz t|}Wn&ttfytd |d Yn0t ||||d d nt ||d d |dur0t|t j |d t dt|pdd|jj|jjdd}t|t|jddS)a Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. token (`str`, *optional*): A [user access token](https://hf.co/settings/tokens) to authenticate requests against the Hub Raises: - `ValueError` if `lfs_batch_action` is improperly formatted - `HTTPError` if the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrJverifyheader chunk_sizezTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.href)rRrnro upload_url)rRrqUSERT)usernamepasswordrM)rVrUz: Upload successful)rYr`loggerdebugZ path_in_repo_validate_lfs_actionr?rc TypeError_upload_multi_part_upload_single_partrr^rr upload_inforrOrr) rRrkrCrl upload_action verify_actionrnroZ verify_respr+r+r, lfs_uploads<       r~Z lfs_actioncCs:t|dtr.|ddus6t|dts6td|S).validates response from the LFS batch endpointrprnNz"lfs_action is improperly formatted)rar`rAdictrcrr+r+r,rws rw)lfs_batch_actionscCspt|dtr t|dts(td|did}|did}|dur\t||durlt||S)rrNrz)lfs_batch_actions is improperly formattedrlrJrmN)rar`rAr?rcrw)rr|r}r+r+r,rYs rY)lfs_batch_errorcCsht|dtr t|dts(td|d}t|tr\t|dtr\t|dtsdtd|S)rrNrz'lfs_batch_error is improperly formattedrXmessagecode)rar`rAr?rcr)r error_infor+r+r,r]s  r])rRrqrGcCsF|jdd&}td||d}t|Wdn1s80YdS)aZ Uploads `fileobj` as a single PUT HTTP request (basic LFS transfer protocol) Args: upload_url (`str`): The URL to PUT the file to. fileobj: The file-like object holding the data to upload. Returns: `requests.Response` Raises: `requests.HTTPError` if the upload resulted in an error TZ with_tqdmPUTr.N)as_filerr)rRrqr3responser+r+r,rzsrz)rRrnrorqrGcCst||j|d}t}tr>t|jts>t|jts>tdd}|rPt |||dn t |||d}t j |t ||jjtd}t|dS)z@ Uploads file using HF multipart LFS transfer protocol. )rnr{rozlhf_transfer is enabled but does not support uploading from bytes or BinaryIO, falling back to regular uploadF)rRsorted_parts_urlsro)rUrTN)_get_sorted_parts_urlsr{rrapath_or_fileobjrArwarningswarn_upload_parts_hf_transfer_upload_parts_iterativelyrr^_get_completion_payloadrrOr_r)rRrnrorqrZuse_hf_transferresponse_headersZcompletion_resr+r+r,ry(s,   ry)rnr{rorGcCsNddtdd|DdddD}t|}|t|j|krJtd|S)NcSsg|] \}}|qSr+r+)rP_rqr+r+r,rQLsz*_get_sorted_parts_urls..cSs2g|]*\}}|rt|dkrt|d|fqS)r )isdigitr1r?)rPZpart_numrqr+r+r,rQOscSs|dSNrr+)tr+r+r,Tr\z(_get_sorted_parts_urls..)keyz0Invalid server response to upload large LFS file)sorteditemsr1rrrc)rnr{roZsorted_part_upload_urlsZ num_partsr+r+r,rKs r)rrNrGcCsfg}t|D]N\}}|d}|dus.|dkrFtd|d|d||d|dq ||dS)NrhrHzInvalid etag (`z`) returned for part r)rgrh)rNrj) enumerater`rcappend)rrNrjZ part_numberrnrhr+r+r,r]s r)rRrrorGc Csg}|jddv}t|D]\\}}t||||d2}td||d}t|||jWdq1sl0YqWdn1s0Y|S)NTr) seek_from read_limitrr.)rr SliceFileObjrrrrT) rRrrorTr3Zpart_idxZpart_upload_urlZ fileobj_sliceZpart_upload_resr+r+r,rlsJrc Csxzddlm}Wnty*tdYn0z||j||ddddWStyr}ztd|WYd}~n d}~00dS) Nr)multipart_uploadzFast uploading using 'hf_transfer' is enabled (HF_HUB_ENABLE_HF_TRANSFER=1) but 'hf_transfer' package is not available in your environment. Try `pip install hf_transfer`.) file_pathZ parts_urlsroZ max_filesZparallel_failures max_retriesz~An error occurred while uploading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.)Z hf_transferr ImportErrorrcr Exception RuntimeError)rRrrorer+r+r,r}s*  rc@speZdZdZeeedddZddZddZded d d Z ed ddZ e j feeedddZ ddZdS)ra Utility context manager to read a *slice* of a seekable file-like object as a seekable, file-like object. This is NOT thread safe Inspired by stackoverflow.com/a/29838711/593036 Credits to @julien-c Args: fileobj (`BinaryIO`): A file-like object to slice. MUST implement `tell()` and `seek()` (and `read()` of course). `fileobj` will be reset to its original position when exiting the context manager. seek_from (`int`): The start of the slice (offset from position 0 in bytes). read_limit (`int`): The maximum number of bytes to read from the slice. Attributes: previous_position (`int`): The previous position Examples: Reading 200 bytes with an offset of 128 bytes from a file (ie bytes 128 to 327): ```python >>> with open("path/to/file", "rb") as file: ... with SliceFileObj(file, seek_from=128, read_limit=200) as fslice: ... fslice.read(...) ``` Reading a file in chunks of 512 bytes ```python >>> import os >>> chunk_size = 512 >>> file_size = os.getsize("path/to/file") >>> with open("path/to/file", "rb") as file: ... for chunk_idx in range(ceil(file_size / chunk_size)): ... with SliceFileObj(file, seek_from=chunk_idx * chunk_size, read_limit=chunk_size) as fslice: ... chunk = fslice.read(...) ``` r3rrcCs||_||_||_dSNr)selfr3rrr+r+r,__init__szSliceFileObj.__init__cCsF|j|_|jdtj}t|j||j|_ |j|jt j |Sr) r3r7_previous_positionr5osSEEK_ENDminrr_lenr%r6)rZ end_of_streamr+r+r, __enter__s  zSliceFileObj.__enter__cCs|j|jtjdSr)r3r5rr%r6)rexc_type exc_value tracebackr+r+r,__exit__szSliceFileObj.__exit__ncCsB|}||jkrdS|j|}|j|dkr2|nt||}|S)Nr\r)r7rr3r4r)rrposZremaining_amountr/r+r+r,r4s   zSliceFileObj.read)rGcCs|j|jSr)r3r7rrr+r+r,r7szSliceFileObj.tell)offsetwhencerGcCs|j}||j}|tjtjfvrR|tjkr2||n||}t|t||}tj}n>|tjkr|j }t||t|||}nt d|d|j |||jS)Nz whence value z is not supported) rrrr6rmaxrSEEK_CURr3r7rcr5)rrrstartendZcur_posr+r+r,r5s   zSliceFileObj.seekccs|jddVdS)Ni@r)r4rr+r+r,__iter__szSliceFileObj.__iter__N)r)r9r:r;r<rr?rrrr4r7rr6r5rr+r+r+r,rs,r)N)Er<r%rrer contextlibrZ dataclassesrmathros.pathrpathlibrtypingrrr r r r r Z requests.authrZhuggingface_hub.constantsrrrZhuggingface_hub.utilsrutilsrrrrrZ utils._typingrZ utils.sharrZ _commit_apirZ get_loggerr9rucompileZ OID_REGEXZLFS_MULTIPART_UPLOAD_COMMANDr_rrArrerfrir~rwrYr]rzr?ryrrrrrr+r+r+r,sj     $      )H< #