a  d@sddlZddlZddlZddlZddlmZmZddlZeeedddZ eeeeedddZ deeeeeefe dd d d Z deeeeddddZ dS)N)ListTuple)file_urlfolderreturncCstj|}tj||}tj|r7z._replace_remote_with_local..rrNz..w) rr dirnamereplacelensplitseprrr r) rr'r(r)Z relt_pathdepthr"r#urlfpathZpath_upfwrrr_replace_remote_with_local*s &  r9 docs/sourcefetched-s3-assets*.rst5https?://[-a-zA-Z0-9_]+\.s3\.[-a-zA-Z0-9()_\\+.\\/=]+)r' assets_folder file_patternretrieve_patternrc Cstjtj|d|dd}|s:td|d|ddSt||d}|sdtd |d |dStj||}tj|dd g}t t |D]J\}} td |d t |d| t | |} | | tj|| fq|D]} t| ||qdS)aSearch all URL in docs, download these files locally and replace online with local version. Args: docs_folder: the location of docs related to the project root assets_folder: a folder inside ``docs_folder`` to be created and saving online assets file_pattern: what kind of files shall be scanned retrieve_pattern: patter for reg. expression to search URL/S3 resources z**T) recursivez no files were listed in folder "z" and pattern ""N)rz"no resources/assets were match in z for )exist_okz >> downloading (/z): )globrr r r rr%infomakedirs enumeratesetr2rappendr9) r'r>r?r@rurlsZ target_folderZpairs_url_fileir6rr7rrrfetch_external_assetsCs"   rM)r&)r:r;r<r=)rEr rr typingrrrstrrr%intr9rMrrrrs.