a dd @sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZddlmZmZmZmZmZmZmZmZmZmZdd lmZdd lmZd d l m!Z!m"Z"d d l#m$Z$m%Z%m&Z&d dl'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.d dl'm/Z0d dl1m2Z2er"d dl3m4Z4e,5e6Z7e2dZ8dZ9e GdddZ:e GdddZ;e GdddZeer)r2r4r4r5r6js z!CommitOperationCopy.__post_init__) r7r8r9r:r;r<r?rr6r4r4r4r5r=Qs r=c@sxeZdZUdZeed<eeeee fed<e dddZ e ed<ddd d Z edeee d d d ZedddZdS)CommitOperationAdda Data structure holding necessary info to upload a file to a repository on the Hub. Args: path_in_repo (`str`): Relative filepath in the repo, for example: `"checkpoints/1fec34a/weights.bin"` path_or_fileobj (`str`, `Path`, `bytes`, or `BinaryIO`): Either: - a path to a local file (as `str` or `pathlib.Path`) to upload - a buffer of bytes (`bytes`) holding the content of the file to upload - a "file object" (subclass of `io.BufferedIOBase`), typically obtained with `open(path, "rb")`. It must support `seek()` and `tell()` methods. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is not one of `str`, `Path`, `bytes` or `io.BufferedIOBase`. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `str` or `Path` but not a path to an existing file. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `io.BufferedIOBase` but it doesn't support both `seek()` and `tell()`. r)path_or_fileobjF)initrepr upload_infoNreturnc Cs*t|j|_t|jtr$t|j|_t|jtrdtjtj |j}tj |s~t d|dnt|jt j tfs~t dt|jt j rz|j|jdtjWn2ttfy}zt d|WYd}~n d}~00t|jtrt|j|_n,t|jtrt|j|_nt|j|_dS)z6Validates `path_or_fileobj` and compute `upload_info`.zProvided path: 'z(' is not a file on the local file systemzpath_or_fileobj must be either an instance of str, bytes or io.BufferedIOBase. If you passed a file-like object, make sure it is in binary mode.rzNpath_or_fileobj is a file-like object but does not implement seek() and tell()N)r-r)r/rBrr;ospathnormpath expanduserisfiler1ioBufferedIOBasebytestellseekSEEK_CUROSErrorAttributeErrorr from_pathrE from_bytesZ from_fileobj)r3rBexcr4r4r5r6s4       z CommitOperationAdd.__post_init__) with_tqdmrGccst|jtst|jtr|rNt|j}|VWdq1sB0Yqt|jd}|VWdq1sv0YnLt|jtrt|jVn0t|jtj r|j }|jV|j |tj dS)u A context manager that yields a file-like object allowing to read the underlying data behind `path_or_fileobj`. Args: with_tqdm (`bool`, *optional*, defaults to `False`): If True, iterating over the file object will display a progress bar. Only works if the file-like object is a path to a file. Pure bytes and buffers are not supported. Example: ```python >>> operation = CommitOperationAdd( ... path_in_repo="remote/dir/weights.h5", ... path_or_fileobj="./local/weights.h5", ... ) CommitOperationAdd(path_in_repo='remote/dir/weights.h5', path_or_fileobj='./local/weights.h5') >>> with operation.as_file() as file: ... content = file.read() >>> with operation.as_file(with_tqdm=True) as file: ... while True: ... data = file.read(1024) ... if not data: ... break config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] >>> with operation.as_file(with_tqdm=True) as file: ... requests.put(..., data=file) config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] ``` Nrb) r/rBr;rr openrOrMBytesIOrNrPrQSEEK_SET)r3rXfileZprev_posr4r4r5as_files$ &&  zCommitOperationAdd.as_filecCs:|}t|WdS1s,0YdS)z[ The base64-encoded content of `path_or_fileobj` Returns: `bytes` N)r^base64 b64encoderead)r3r]r4r4r5 b64contents zCommitOperationAdd.b64content)F)r7r8r9r:r;r<rrrOr rrErr6rr0rr^rbr4r4r4r5rAos #1rA)r)rGcCs|dr|dd}|dks0|dks0|dr@td|d|drV|d d}td d |dDr~td |d |S)Nr,r.z..z../z,Invalid `path_in_repo` in CommitOperation: ''z./css|]}|dkVqdS)z.gitNr4).0partr4r4r5 z)_validate_path_in_repo..z^Invalid `path_in_repo` in CommitOperation: cannot update files under a '.git/' folder (path: 'z').) startswithr1anysplit)r)r4r4r5r-s    r-) operationsrGcCstt}|D]}|j}t|trr||dkr>td|d||d7<t|jD]}|t |d7<qXt|t r |t t|dkr |j rtd|dq td|dq dS)a Warn user when a list of operations is expected to overwrite itself in a single commit. Rules: - If a filepath is updated by multiple `CommitOperationAdd` operations, a warning message is triggered. - If a filepath is updated at least once by a `CommitOperationAdd` and then deleted by a `CommitOperationDelete`, a warning is triggered. - If a `CommitOperationDelete` deletes a filepath that is then updated by a `CommitOperationAdd`, no warning is triggered. This is usually useless (no need to delete before upload) but can happen if a user deletes an entire folder and then add new files to it. rzBAbout to update multiple times the same file in the same commit: 'z9'. This can cause undesired inconsistencies in your repo.rz_About to delete a folder containing files that have just been updated within the same commit: 'zLAbout to delete a file that have just been updated within the same commit: 'N) rintr)r/rAwarningswarnrparentsr;r(r+)rmZnb_additions_per_path operationr)parentr4r4r5warn_on_overwriting_operationss4   rt)endpoint num_threads) additions repo_typerepo_idtokenrvrwcs~g}t|ddD]R}tdd|D|||d\}} | rZddd| D} td| ||7}qd d |Dg} |D]:} | d d urtd | djdqz| | qzt | dkrtdd Sd dfdd } t rtdt | dt | D]} | | qn`t | dkr@td| | dn:tdt | d|dt | | dt | d|t dd S)a Uploads the content of `additions` to the Hub using the large file storage protocol. Relevant external documentation: - LFS Batch API: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: additions (`List` of `CommitOperationAdd`): The files to be uploaded repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) num_threads (`int`, *optional*): The number of concurrent threads to use when uploading. Defaults to 5. Raises: `RuntimeError` if an upload failed for any reason Raises: `ValueError` if the server returns malformed responses Raises: `requests.HTTPError` if the LFS batch endpoint returned an HTTP error ) chunk_sizecSsg|] }|jqSr4)rErfopr4r4r5 Xriz$upload_lfs_files..)Z upload_infosr{rzryrv c Ss0g|](}d|dd|didqS)z$Encountered error for file with OID oidz: `errormessage)get)rferrr4r4r5rbsz$LFS batch endpoint returned errors: cSsi|]}|jj|qSr4)rEsha256hex)rfZadd_opr4r4r5 jriz$upload_lfs_files..actionsNzContent of file rz/ is already present upstream - skipping upload.rzNo LFS files to upload.rFc s\z|d}t||dWn8tyV}z td|jd|WYd}~n d}~00dS)Nr)rrZlfs_batch_actionr{zError while uploading 'z ' to the Hub.)r Exception RuntimeErrorr))Z batch_actionrrrWZ oid2addopr{r4r5_wrapped_lfs_upload|s  z-upload_lfs_files.._wrapped_lfs_uploadz Uploading z* LFS files to the Hub using `hf_transfer`.rzUploading 1 LFS file to the Hubz" LFS files to the Hub using up to z threads concurrentlyzUpload z LFS files)desc max_workersZ tqdm_class) rrjoinr1rloggerdebugr)appendlenrhf_tqdmr)rxryrzr{rvrwZ batch_actionschunkZbatch_actions_chunkZbatch_errors_chunkrZfiltered_actionsactionrr4rr5upload_lfs_files,s\)        r)preupload_infocCsh|d}t|tstd|D]B}t|trZt|dtrZt|dtrZ|ddvs tdq |S)Nfilesz&preupload_info is improperly formattedrI uploadModer%z'preupload_info is improperly formatted:)rr/listr1dictr;)rr file_infor4r4r5_validate_preupload_infos    rF)rxryrzr{revisionrv create_prrGc Cs|dur |nt}t|d}i}t|dD]~} ddd| Di} tj|d|d|d || ||rld d indd } t| t| } |jfid d| dDq(|D]} | j j dkr| j }d||<q|S)av Requests the Hub "preupload" endpoint to determine whether each input file should be uploaded as a regular git blob or as git LFS blob. Args: additions (`Iterable` of :class:`CommitOperationAdd`): Iterable of :class:`CommitOperationAdd` describing the files to upload to the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[str, UploadMode]` Key is the file path, value is the upload mode ("regular" or "lfs"). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. N)r{r|rcSs8g|]0}|jt|jjd|jj|jjdqS)ascii)rIsamplesizesha) r)r_r`rErdecoderrrr~r4r4r5rs  z&fetch_upload_modes..z/api/zs/z /preupload/r1)jsonheadersparamscSsi|]}|d|dqS)rIrr4)rfr]r4r4r5rriz&fetch_upload_modes..rr') rrrrpostrrrupdaterErr))rxryrzr{rrvrr upload_modesrpayloadresprZadditionrIr4r4r5fetch_upload_modess,%      rr$)copiesryrzr{rrvrGcCsddlm}|||d}i}t|dddD]\} } t| } dd| D} td t| tD]L} |j|| | | t| px||d } | D] }|jst d |||j | f<qqZ| D].}|j | f|vrt d |j d | p|dqq,|S)a` Requests the Hub files information of the LFS files to be copied, including their sha256. Args: copies (`Iterable` of :class:`CommitOperationCopy`): Iterable of :class:`CommitOperationCopy` describing the files to copy on the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[Tuple[str, Optional[str]], RepoFile]]` Key is the file path and revision of the file to copy, value is the repo file. Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. r)HfApi)rvr{cSs|jSr@)r?)rr4r4r5riz)fetch_lfs_files_to_copy..)keycSsg|] }|jqSr4)r>r~r4r4r5rriz+fetch_lfs_files_to_copy..r)rzpathsrry)Copying a non-LFS file is not implementedz Cannot copy z at revision z: file is missing on repo.) hf_apirrrrangerFETCH_LFS_BATCH_SIZEZlist_files_infor&NotImplementedErrorZ rfilenamer>r)rryrzr{rrvrr files_to_copyr?rmroffsetZsrc_repo_filesZ src_repo_filerrr4r4r5fetch_lfs_files_to_copys2#   r)rmrrcommit_messagecommit_description parent_commitrGc csN|dur |nd}||d}|dur*||d<d|dV|D] }t|trz||jdkrzd||jd d dVq:t|tr||jd krd |jd |jj|jj ddVq:t|t r|j rdndd|jidVq:t|t r,||j |jf}|jstdd |jd |jd ddVq:td|d||jq:dS)aG Builds the payload to POST to the `/commit` API of the Hub. Payload is returned as an iterator so that it can be streamed as a ndjson in the POST request. For more information, see: - https://github.com/huggingface/huggingface_hub/issues/1085#issuecomment-1265208073 - http://ndjson.org/ N)summary descriptionZ parentCommitheader)rvaluer'r]r_)contentrIencodingr&ZlfsFiler)rIalgorrZ deletedFolderZ deletedFilerIr)rIrrz(Unknown operation to commit. Operation: z. Upload mode: )r/rArr)rbrrErrrr(r+r=r>r?r&rr1) rmrrrrr header_valuerrZ file_to_copyr4r4r5prepare_commit_payload/sP            r)NF)N)NN)Ir:r_rMrHro collectionsr contextlibrZ dataclassesrr itertoolsrpathlibrrtypingr r r r r rrrrrZtqdm.contrib.concurrentrZhuggingface_hubr constantsrrr&rrrutilsrrrrrr r!r"rZ utils._typingr#rr$Z get_loggerr7rZ UploadModerr(r=rAr;r-ZCommitOperationrtrnrrrr0rrrr4r4r4r5s   0  $    {.j JB