CK workflows in this repository were originally created by dividiti Limited and used to prepare ~50% of all results (~900 out of 1800) submitted to MLPerf Inference v0.5 and v0.7. These workflows are now being maintained and extended by Krai Ltd.
Install Collective Knowledge (CK)
$ export CK_PYTHON=/usr/bin/python3 $ $CK_PYTHON -m pip install --ignore-installed pip setuptools testresources --user $ $CK_PYTHON -m pip install ck --user $ echo 'export PATH=$HOME/.local/bin:$PATH' >> $HOME/.bashrc $ source $HOME/.bashrc $ ck version V1.55.2
NB: CK works fine under Python 2, but we are playing safe here (as Python 2 has reached End-Of-Life).
$ ck pull repo --url=https://github.com/krai/ck-mlperf
$ export CK_PYTHON=/usr/bin/python3 $ ck detect soft:compiler.python --full_path=$CK_PYTHON $ ck show env --tags=compiler,python Env UID: Target OS: Bits: Name: Version: Tags: 6dd55d431ea19241 linux-64 64 python 3.8.5 64bits,compiler,host-os-linux-64,lang-python,python,target-os-linux-64,v3,v3.8,v3.8.5
NB: CK can normally detect available Python interpreters automatically, but we are playing safe here.
$ ck detect platform.os --platform_init_uoa=generic-linux-dummy OS CK UOA: linux-64 (4258b5fe54828a50) OS name: Ubuntu 20.04.1 LTS Short OS name: Linux 5.4.0 Long OS name: Linux-5.4.0-58-generic-x86_64-with-glibc2.29 OS bits: 64 OS ABI: x86_64 Platform init UOA: -
Unfortunately, the ImageNet 2012 validation dataset (50,000 images) cannot be freely downloaded.
If you have a copy of it e.g. under /datasets/dataset-imagenet-ilsvrc2012-val/, you can register it with CK ("detect") as follows:
$ ck detect soft:dataset.imagenet.val --extra_tags=full \ --full_path=/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00000001.JPEG
The reference code uses OpenCV for preprocessing image data. The minimal OpenCV Python package can be installed via CK as follows:
$ ck install package --tags=opencv-python-headless
The standard image resolution used for benchmarking ImageNet models is 224x224. The reference code uses bilinear interpolation by default. For ResNet50, however, area interpolation is used. In addition, ResNet50 requires means to be subtracted.
To preprocess ImageNet specifically for ResNet50 (use --ask to confirm the destination):
$ ck install package --tags=dataset,imagenet,preprocessed,using-opencv,full,for-resnet --ask
The dataset preprocessed for ResNet50 requires 29G on disk, as pixel components are stored as 32-bit floats:
$ du -hs $(ck locate env --tags=dataset,imagenet,preprocessed,using-opencv,full,for-resnet) 29G /home/anton/CK-TOOLS/dataset-imagenet-preprocessed-using-opencv-crop.875-for-resnet-full-side.224-unmutilated
An alternative method, dubbed universal, uses bilinear interpolation and stores pixels as 8-bit integers. The dataset preprocessed using the alternative method requires 7.1G on disk.
At load time, however, minor additional processing may be required depending on the model (e.g. see sample code for subtracting means and for normalizing to the (-1,+1) range).
$ ck install package --tags=dataset,imagenet,preprocessed,using-opencv,full,universal --ask $ du -hs $(ck locate env --tags=dataset,imagenet,preprocessed,using-opencv,full,universal) 7.1G /home/anton/CK-TOOLS/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224-universal-unmutilated
| Preprocessing method | OpenCV for ResNet50 | OpenCV universal |
|---|---|---|
| ResNet50 Top1 accuracy | 0.76456 | 0.76422 |
| ResNet50 Top5 accuracy | 0.93016 | 0.93074 |
| Matches reference? | Yes | No |
| Additional tags | using-opencv,for-resnet |
using-opencv,universal |
| Supported models | ResNet50 only | ResNet50, MobileNet-224 |
| Supported platforms | x86 | x86 |
| Data format | rgbf32 (float32) | rgb8 (int8) |
| Data size | 29G | 7.1G |
For calibration, MLPerf provides two equivalent sets of 500 images, each randomly selected from the ImageNet validation dataset.
$ ck install package --tags=dataset,imagenet,cal,mlperf.option1 --dep_add_tags.imagenet-val=full $ ck locate env --tags=dataset,imagenet,cal,mlperf.option1 /home/anton/CK-TOOLS/dataset-imagenet-calibration-mlperf.option1 $ du -hs $(ck locate env --tags=dataset,imagenet,cal,mlperf.option2) 65M /home/anton/CK-TOOLS/dataset-imagenet-calibration-mlperf.option2
$ ck install package --tags=dataset,imagenet,cal,mlperf.option2 --dep_add_tags.imagenet-val=full $ ck locate env --tags=dataset,imagenet,cal,mlperf.option2 /home/anton/CK-TOOLS/dataset-imagenet-calibration-mlperf.option2 $ du -hs $(ck locate env --tags=dataset,imagenet,cal,mlperf.option2) 71M /home/anton/CK-TOOLS/dataset-imagenet-calibration-mlperf.option2
To install the COCO 2017 validation dataset (use --ask to confirm the destination):
$ ck install package --tags=dataset,coco,val,2017 --ask
The reference code uses OpenCV for preprocessing image data. The minimal OpenCV Python package can be installed via CK as follows:
$ ck install package --tags=opencv-python-headless
SSD-ResNet34 requires resizing images to the 1200x1200 resolution:
$ ck install package --tags=dataset,coco.2017,preprocessed,using-opencv,full,side.1200 --ask $ du -hs $(ck locate env --tags=dataset,coco.2017,preprocessed,using-opencv,full,side.1200 21G /home/anton/CK-TOOLS/dataset-object-detection-preprocessed-using-opencv-coco.2017-full-side.1200
SSD-MobileNet-v1 requires resizing images to the 300x300 resolution:
$ ck install package --tags=dataset,coco.2017,preprocessed,using-opencv,full,side.300 --ask $ du -hs $(ck locate env --tags=dataset,coco.2017,preprocessed,using-opencv,full,side.300 1.3G /home/anton/CK-TOOLS/dataset-object-detection-preprocessed-using-opencv-coco.2017-full-side.300