a  d] @sdZddlmZddlZddlZddlZddlmZmZm Z ddl Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZddlZdd lmZmZddlmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'erddl(m)Z)m*Z*m+Z+m,Z,m-Z-dddddZ.d:ddddddddd Z/Gd!ddZ0Gd"d#d#e0Z1Gd$d%d%e0Z2eed&d'd;d*d+dd,d-dd.dd/d0 d1d2Z3eed&d'd(ddej4ej4ddfd3dd.dd4d5dd6d*d7 d8d9Z5dS)eZdZedddddZddddZd dd d d Zd S)rrNone)dfr cCst|tstddS)Nz+to_parquet only supports IO with DataFrames)r9rr(rHr*r*r+validate_dataframes zBaseImpl.validate_dataframerIcKs t|dSNr )selfrHr0 compressionkwargsr*r*r+writeszBaseImpl.writeNr cKs t|dSrKr )rLr0columnsrNr*r*r+readsz BaseImpl.read)N)r= __module__ __qualname__ staticmethodrJrOrRr*r*r*r+rsc @sZeZdZddddZdddd d d d dd ddZdddejddfddd ddddZdS)r%rGrPcCs&tdddddl}ddl}||_dS)Nr#z(pyarrow is required for parquet support.extrar)r Zpyarrow.parquetZ(pandas.core.arrays.arrow.extension_typesapi)rLr#pandasr*r*r+__init__s zPyArrowImpl.__init__snappyNrzFilePath | WriteBuffer[bytes] str | None bool | Noner.list[str] | None)rHr0rMindexr2partition_colsr cKsF||d|ddi} |dur*|| d<|jjj|fi| } |jrtdt|ji} | jj } i| | } | | } t |||d|dud\}}}t |t jrt|drt |jttfr|j}t |tr|}z^|dur|jjj| |f|||d|n|jjj| |f||d|W|durB|n|dur@|0dS) NschemaZpreserve_indexZ PANDAS_ATTRSwb)r2r3r4name)rMr` filesystem)rMrd)rJpoprXZTableZ from_pandasattrsjsondumpsrametadataZreplace_schema_metadatarFr9ioBufferedWriterhasattrrcrbytesdecodeparquetZwrite_to_datasetZ write_tableclose)rLrHr0rMr_r2r`rdrNZfrom_pandas_kwargstable df_metadataZexisting_metadataZmerged_metadatarCrEr*r*r+rOsf         zPyArrowImpl.writeFr/DtypeBackend | lib.NoDefault)use_nullable_dtypes dtype_backendr2r cKs(d|d<i} |dkr2ddlm} | } | j| d<n$|dkrFtj| d<ntrVt| d<td} | d krnd| d <t|||d d \} }}z|j j j | f|||d |}|j fi| }| d kr|j d dd}|jjrd|jjvr|jjd}t||_|W|dur |Sn|dur"|0dS)NTZuse_pandas_metadataZnumpy_nullabler)_arrow_dtype_mappingZ types_mapperr#zmode.data_managerarrayZ split_blocksr-)r2r3)rQrdfiltersF)copys PANDAS_ATTRS)pandas.io._utilrvgetpdZ ArrowDtyperrrrFrXroZ read_table to_pandasZ _as_managerrarirgloadsrfrp)rLr0rQrxrtrur2rdrNZto_pandas_kwargsrvmappingmanagerrCrEZpa_tableresultrrr*r*r+rRsX           zPyArrowImpl.read)r[NNNN)r=rSrTrZrOr no_defaultrRr*r*r*r+r%sDr%c@sBeZdZddddZdddd dd d d Zdd dd ddZdS)r&rGrPcCstddd}||_dS)Nr$z,fastparquet is required for parquet support.rV)r rX)rLr$r*r*r+rZ'szFastParquetImpl.__init__r[Nrz*Literal['snappy', 'gzip', 'brotli'] | Noner.)rHrMr2r c  s||d|vr"|dur"tdd|vr4|d}|durDd|d<|durTtdt|}t|rtdfdd|d <n rtd td d .|jj ||f|||d |Wdn1s0YdS)N partition_onzYCannot use both partition_on and partition_cols. Use partition_cols for partitioning dataZhiveZ file_scheme9filesystem is not implemented for the fastparquet engine.r7csj|dfipiS)Nrb)open)r0_r7r2r*r+Rsz'FastParquetImpl.write..Z open_withz?storage_options passed with file object or non-fsspec file pathT)record)rMZ write_indexr) rJr(rer:rrr rrXrO) rLrHr0rMr_r`r2rdrNr*rr+rO/s@   zFastParquetImpl.write)r2r c Ksi}|dd}|dtj} d|d<|r2td| tjurDtd|durTtdt|}d} t|rtd} | j|d fi|pij |d <n,t |t rt j |st|d d|d } | j}z>|jj|fi|} | jf||d |W| dur| Sn| dur| 0dS) NrtFruZ pandas_nullszNThe 'use_nullable_dtypes' argument is not supported for the fastparquet enginezHThe 'dtype_backend' argument is not supported for the fastparquet enginerr7r-r1r8)rQrx)rerrr(r:rrr rr1r9rr@r0rArrBrXZ ParquetFiler}rp) rLr0rQrxr2rdrNZparquet_kwargsrtrurEr7Z parquet_filer*r*r+rRdsH     zFastParquetImpl.read)r[NNNN)NNNN)r=rSrTrZrOrRr*r*r*r+r&&s 8r&r2)r2r!r[rz$FilePath | WriteBuffer[bytes] | Noner\r]r^z bytes | None) rHr0rrMr_r2r`rdr c Kstt|tr|g}t|} |dur(tn|} | j|| f|||||d||durlt| tjsdJ| SdSdS)a Write a DataFrame to the parquet format. Parameters ---------- df : DataFrame path : str, path object, file-like object, or None, default None String, path object (implementing ``os.PathLike[str]``), or file-like object implementing a binary ``write()`` function. If None, the result is returned as bytes. If a string, it will be used as Root Directory path when writing a partitioned dataset. The engine fastparquet does not accept file-like objects. .. versionchanged:: 1.2.0 engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. When using the ``'pyarrow'`` engine and no storage options are provided and a filesystem is implemented by both ``pyarrow.fs`` and ``fsspec`` (e.g. "s3://"), then the ``pyarrow.fs`` filesystem is attempted first. Use the filesystem keyword with an instantiated fsspec filesystem if you wish to use its implementation. compression : {{'snappy', 'gzip', 'brotli', 'lz4', 'zstd', None}}, default 'snappy'. Name of the compression to use. Use ``None`` for no compression. index : bool, default None If ``True``, include the dataframe's index(es) in the file output. If ``False``, they will not be written to the file. If ``None``, similar to ``True`` the dataframe's index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn't require much space and is faster. Other indexes will be included as columns in the file output. partition_cols : str or list, optional, default None Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. {storage_options} .. versionadded:: 1.2.0 filesystem : fsspec or pyarrow filesystem, default None Filesystem object to use when reading the parquet file. Only implemented for ``engine="pyarrow"``. .. versionadded:: 2.1.0 kwargs Additional keyword arguments passed to the engine Returns ------- bytes if no path argument is provided else None N)rMr_r`r2rd)r9rr,rjBytesIOrOgetvalue) rHr0rrMr_r2r`rdrNimplZ path_or_bufr*r*r+ to_parquets(F  rzFilePath | ReadBuffer[bytes]zbool | lib.NoDefaultrsz&list[tuple] | list[list[tuple]] | None) r0rrQr2rtrurdrxr c Ksft|} |tjur:d} |dur&| d7} tj| ttdnd}t|| j|f||||||d|S)a Load a parquet object from the file path, returning a DataFrame. Parameters ---------- path : str, path object or file-like object String, path object (implementing ``os.PathLike[str]``), or file-like object implementing a binary ``read()`` function. The string could be a URL. Valid URL schemes include http, ftp, s3, gs, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.parquet``. A file URL can also be a path to a directory that contains multiple partitioned parquet files. Both pyarrow and fastparquet support paths to directories as well as file URLs. A directory path could be: ``file://localhost/path/to/tables`` or ``s3://bucket/partition_dir``. engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. When using the ``'pyarrow'`` engine and no storage options are provided and a filesystem is implemented by both ``pyarrow.fs`` and ``fsspec`` (e.g. "s3://"), then the ``pyarrow.fs`` filesystem is attempted first. Use the filesystem keyword with an instantiated fsspec filesystem if you wish to use its implementation. columns : list, default=None If not None, only these columns will be read from the file. {storage_options} .. versionadded:: 1.3.0 use_nullable_dtypes : bool, default False If True, use dtypes that use ``pd.NA`` as missing value indicator for the resulting DataFrame. (only applicable for the ``pyarrow`` engine) As new dtypes are added that support ``pd.NA`` in the future, the output with this option will change to use those dtypes. Note: this is an experimental option, and behaviour (e.g. additional support dtypes) may change without notice. .. deprecated:: 2.0 dtype_backend : {{'numpy_nullable', 'pyarrow'}}, default 'numpy_nullable' Back-end data type applied to the resultant :class:`DataFrame` (still experimental). Behaviour is as follows: * ``"numpy_nullable"``: returns nullable-dtype-backed :class:`DataFrame` (default). * ``"pyarrow"``: returns pyarrow-backed nullable :class:`ArrowDtype` DataFrame. .. versionadded:: 2.0 filesystem : fsspec or pyarrow filesystem, default None Filesystem object to use when reading the parquet file. Only implemented for ``engine="pyarrow"``. .. versionadded:: 2.1.0 filters : List[Tuple] or List[List[Tuple]], default None To filter out data. Filter syntax: [[(column, op, val), ...],...] where op is [==, =, >, >=, <, <=, !=, in, not in] The innermost tuples are transposed into a set of filters applied through an `AND` operation. The outer list combines these sets of filters through an `OR` operation. A single list of tuples can also be used, meaning that no `OR` operation between set of filters is to be conducted. Using this argument will NOT result in row-wise filtering of the final partitions unless ``engine="pyarrow"`` is also specified. For other engines, filtering is only performed at the partition level, that is, to prevent the loading of some row-groups and/or files. .. versionadded:: 2.1.0 **kwargs Any additional kwargs are passed to the engine. Returns ------- DataFrame See Also -------- DataFrame.to_parquet : Create a parquet object that serializes a DataFrame. Examples -------- >>> original_df = pd.DataFrame( ... {{"foo": range(5), "bar": range(5, 10)}} ... ) >>> original_df foo bar 0 0 5 1 1 6 2 2 7 3 3 8 4 4 9 >>> df_parquet_bytes = original_df.to_parquet() >>> from io import BytesIO >>> restored_df = pd.read_parquet(BytesIO(df_parquet_bytes)) >>> restored_df foo bar 0 0 5 1 1 6 2 2 7 3 3 8 4 4 9 >>> restored_df.equals(original_df) True >>> restored_bar = pd.read_parquet(BytesIO(df_parquet_bytes), columns=["bar"]) >>> restored_bar bar 0 5 1 6 2 7 3 8 4 9 >>> restored_bar.equals(original_df[['bar']]) True The function uses `kwargs` that are passed directly to the engine. In the following example, we use the `filters` argument of the pyarrow engine to filter the rows of the DataFrame. Since `pyarrow` is the default engine, we can omit the `engine` argument. Note that the `filters` argument is implemented by the `pyarrow` engine, which can benefit from multithreading and also potentially be more economical in terms of memory. >>> sel = [("foo", ">", 2)] >>> restored_part = pd.read_parquet(BytesIO(df_parquet_bytes), filters=sel) >>> restored_part foo bar 0 3 8 1 4 9 zYThe argument 'use_nullable_dtypes' is deprecated and will be removed in a future version.TzFUse dtype_backend='numpy_nullable' instead of use_nullable_dtype=True.) stacklevelF)rQrxr2rtrurd) r,rrwarningswarn FutureWarningr rrR) r0rrQr2rtrurdrxrNrmsgr*r*r+ read_parquets0 r)Nr-F)Nr!r[NNNN)6__doc__ __future__rrjrgr@typingrrrrrZpandas._configrZ pandas._libsrZpandas.compat._optionalr Z pandas.errorsr Zpandas.util._decoratorsr Zpandas.util._exceptionsr Zpandas.util._validatorsrrYr|rrZpandas.core.shared_docsrrzrZpandas.io.commonrrrrrZpandas._typingrrrrrr,rFrr%r&rrrr*r*r*r+s`            %? q "]