a  d@s`ddlZddlZddlZddlmZddlmZddlmZdd d Z d d Z Gd ddZ dS)N)Pool)path)tqdmF(c  st|t|ks,Jdt|dt|td|d|dtdt||dsftdt|rtd |d td |r.iitd |d tt|ddfdd} t |} t ||D]&\} } | j t t || | |f| dq| | tdt|d|durtt ||dtj} td| tj|g\}}|j}td||t|}|d}tj||d}tt|dd|jdd}tt |dd}tt ||D]\}\} } d d | | d!}|r&| }| \}}}n$t t || | |\}}}|\}}}||||| d"|d#|d#|d$|d% ||dkr||jdd}q؈|||td&dS)'uMake lmdb from images. Contents of lmdb. The file structure is: example.lmdb ├── data.mdb ├── lock.mdb ├── meta_info.txt The data.mdb and lock.mdb are standard lmdb files and you can refer to https://lmdb.readthedocs.io/en/release/ for more details. The meta_info.txt is a specified txt file to record the meta information of our datasets. It will be automatically created when preparing datasets by our provided dataset tools. Each line in the txt file records 1)image name (with extension), 2)image shape, and 3)compression level, separated by a white space. For example, the meta information could be: `000_00000000.png (720,1280,3) 1`, which means: 1) image name (with extension): 000_00000000.png; 2) image shape: (720,1280,3); 3) compression level: 1 We use the image name without extension as the lmdb key. If `multiprocessing_read` is True, it will read all the images to memory using multiprocessing. Thus, your server needs to have enough memory. Args: data_path (str): Data path for reading images. lmdb_path (str): Lmdb save path. img_path_list (str): Image path list. keys (str): Used for lmdb keys. batch (int): After processing batch images, lmdb commits. Default: 5000. compress_level (int): Compress level when encoding images. Default: 1. multiprocessing_read (bool): Whether use multiprocessing to read all the images to memory. Default: False. n_thread (int): For multiprocessing. map_size (int | None): Map size for lmdb env. If None, use the estimated size from images. Default: None z.callback)argsrzFinish reading z images.Nr.pngzData size per image is: map_sizechunkTwrite meta_info.txtwzWrite ascii.png (,)  z Finish writing lmdb.)lenprintendswith ValueErrorospexistssysexitrrzipZ apply_asyncread_img_workerjoinclosecv2imreadIMREAD_UNCHANGEDimencodeIMWRITE_PNG_COMPRESSIONnbyteslmdbopenbegin enumeraterrencodeputrcommit)Z data_path lmdb_pathZ img_path_listkeysbatchcompress_levelZmultiprocessing_readZn_threadrrpoolrrimg_img_byteZdata_size_per_img data_sizeenvtxntxt_fileidxkey_bytehr c img_shaperrrmake_lmdb_from_imgs sn4            (rPc Cs\t|tj}|jdkr(|j\}}d}n |j\}}}td|tj|g\}}|||||ffS)aRead image worker. Args: path (str): Image path. key (str): Image key. compress_level (int): Compress level when encoding images. Returns: str: Image key. byte: Image byte. tuple[int]: Image shape. rr)r2r3r4ndimshaper5r6) rrrBrDrMr rNrErFrrrr/s   r/c@s*eZdZdZd ddZddZd d Zd S) LmdbMakera8LMDB Maker. Args: lmdb_path (str): Lmdb save path. map_size (int): Map size for lmdb env. Default: 1024 ** 4, 1TB. batch (int): After processing batch images, lmdb commits. Default: 5000. compress_level (int): Compress level when encoding images. Default: 1. rrcCs|dstdt|r6td|dtd||_||_||_ t j ||d|_ |j j dd|_t t|d d |_d |_dS) Nrr r r rrTrrr r)r(r)r*r+r'r,r-r?rArBr8r9rHr:rIr0rJcounter)selfr?rrArBrrr__init__s   zLmdbMaker.__init__c Cs|jd7_|d}|j|||\}}}|j|d|d|d|d|jd |j|jdkr|j|j j dd |_dS) Nrr!r"r#r$r%rTr) rVr<rIr=rJrrBrAr>rHr:)rWrFrrOrLrMr rNrrrr=s  , z LmdbMaker.putcCs"|j|j|jdS)N)rIr>rHr1rJ)rWrrrr1s  zLmdbMaker.closeN)rUrr)__name__ __module__ __qualname____doc__rXr=r1rrrrrTs  rT)rrFrN) r2r8r,multiprocessingrosrr*rrPr/rTrrrrs    {