.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "build/examples_auto_module/demo_auto_data.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_build_examples_auto_module_demo_auto_data.py: 01. Load web datasets with GluonCV Auto Module ================================================= This tutorial introduces the basic dataset preprocesses that can be used to download and load arbitrary custom dataset as long as they follow certain supported data formats. The current version supports loading datasets for - Image Classification(with csv lists and raw images, or folder separated raw images) - Object Detection(as in Pascal VOC format or COCO json annatations) Stay tuned for new applications and formats, we are also looking forward to seeing contributions that brings new formats to GluonCV! That's enough introduction, let's take a look at how web datasets can be loaded into a recommended formats supported in GluonCV auto module. .. GENERATED FROM PYTHON SOURCE LINES 20-45 Image Classification ------- Managing the labels of an image classification dataset is pretty simple. In this example we show a few ways to organize them. First of all, we could infer labels from nested folder structure automatically like:: root/car/0001.jpg root/car/xxxa.jpg root/car/yyyb.jpg root/bus/123.png root/bus/023.jpg root/bus/wwww.jpg or even more with train/val/test splits like:: root/train/car/0001.jpg root/train/car/xxxa.jpg root/train/bus/123.png root/train/bus/023.jpg root/test/car/yyyb.jpg root/test/bus/wwww.jpg where root is the root folder, `car` and `bus` categories are well organized in sub-directories, respectively .. GENERATED FROM PYTHON SOURCE LINES 45-47 .. code-block:: default from gluoncv.auto.tasks import ImageClassification .. GENERATED FROM PYTHON SOURCE LINES 48-53 We can use `ImageClassification.Dataset` to load dataset from a folder, here root can be a local path or url, if it's a url, the archieve file will be downloaded and extracted automatically to `~/.gluoncv` by default, to change the default behavior, you may edit `~/.gluoncv/config.yml` .. GENERATED FROM PYTHON SOURCE LINES 54-58 .. code-block:: default train, val, test = ImageClassification.Dataset.from_folders( 'https://autogluon.s3.amazonaws.com/datasets/shopee-iet.zip', train='train', val='val', test='test', exts=('.jpg', '.jpeg', '.png')) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Downloading /root/.gluoncv/archive/shopee-iet.zip from https://autogluon.s3.amazonaws.com/datasets/shopee-iet.zip... 0%| | 0/40895 [00:00`_ with built-in datasets. D8 datasets is fully compatible with gluoncv.auto, you can directly plug-in datasets loaded from d8 and train with `fit` functions. .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 46.529 seconds) .. _sphx_glr_download_build_examples_auto_module_demo_auto_data.py: .. only :: html .. container:: sphx-glr-footer :class: sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: demo_auto_data.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: demo_auto_data.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_