The speed of MxNet differently built is different

[Rivendile]

I found that the training speed of MxNet v1.5.0 built by pip and from source is different. I tested on 2 servers and 2 workers, ran resnet50, and recorded the average throughput of 100 iterations. However, the results are different, as 232.37 samples/sec (built from source) and 325.97 samples/sec (built by pip). Any suggestions?

[Zha0q1]

Hi, it might be due to that the build configurations are different. For example you might be using the mkl/cuda optimized pip wheel

[Rivendile]

I find that some errors occur when install using https://github.com/apache/incubator-mxnet/tree/v1.5.x/tools/staticbuild/build.sh. But I can install MxNet from source using https://github.com/Rivendile/incubator-mxnet/blob/bytescheduler/docs/install/build_from_source.md.
Also I find there are at least three methods to build mxnet from source, https://github.com/Rivendile/incubator-mxnet/blob/bytescheduler/docs/install/ubuntu_setup.md#quick-mxnet-build, https://github.com/Rivendile/incubator-mxnet/blob/bytescheduler/docs/install/ubuntu_setup.md#manual-mxnet-installation, and https://github.com/apache/incubator-mxnet/tree/v1.5.x/tools/staticbuild. Could you please tell me what's the difference among these methods?

[leezu]

The scripts referenced above are used by to build a static libmxnet that can be distributed to different computers. It's not documented on the website as it's not intended for end-users. The script is not in a good shape in the v1.5.x branch, but you may be able to make it run if you need to. You can just read the script to know the build configurations of the pip wheel (which you asked for).

You can also just check your source build configuration and see if you missed anything. For example, did you enable cudnn?

[Rivendile]

Thanks to your reply. I tried to use https://github.com/apache/incubator-mxnet/blob/1.5.0/make/pip/pip_linux_cu100.mk as config.mk, which should be the configuration of pip installation. I tried to build MxNet with this config but get slower throughput, 199 images/second. The only difference is that USE_S3 is set to be 0 in my setting, because when USE_S3=1, I encountered the error: "mxnet build error.: g++: error: /home/ubuntu/mxnet/3rdparty/dmlc-core/libdmlc.a: No such file or directory". I have tried solutions in #10828, but still cannot make it. So my questions are:

1. Will USE_S3 affect the performance?
2. What should I do to build with USE_S3=1?

[leezu]

The performance difference may also be due to some of the dependencies used by MXNet, such as OpenCV or libturbojpeg. In the published pip wheel, all dependencies are bundled and optimized. You can refer to https://github.com/apache/incubator-mxnet/tree/master/tools/dependencies for the optimized build configuration of the dependencies

[Rivendile]

I will try it. Thanks a lot.