Starting commit. Forked from github.com/tensorpack/tensorpack at comm…

…it a9dce5b220dca34b15122a9329ba9ff055e8edc6

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,36 @@
+
+# If you meet any unexpected problems when running the code, or want to report bugs, please STOP here. Go to the following link instead and fill out the information there:
+https://github.com/tensorpack/tensorpack/issues/new?template=unexpected-problems---bugs.md
+
+
+Otherwise, you can post here for:
+1. Feature Requests:
+  + Note that you can implement a lot of features by extending Tensorpack
+   (See http://tensorpack.readthedocs.io/en/latest/tutorial/index.html#extend-tensorpack).
+   It does not have to be added to Tensorpack unless you have a good reason.
+
+2. Questions on Using/Understanding Tensorpack:
+  + Your question is probably answered in [tutorials](http://tensorpack.readthedocs.io/en/latest/tutorial/index.html#user-tutorials). Read it first.
+  + We answer "HOW to do X with Tensorpack" for a well-defined X.
+    We also answer "HOW/WHY Tensorpack does X" for some X that Tensorpack or its examples are doing.
+
+Some typical questions that we DO NOT answer:
+
++ "Could you improve/implement an example/paper ?" --
+
+  We have no plans to do so. We don't consider feature
+  requests for examples or implement a paper for you.
+  If you don't know how to do something yourself, you may ask a usage question.
++ "The examples do not perform as expected after I change the models/dataset/parameters/etc."
+
+  Tensorpack maintainers make sure the examples perform well without modifications.
+  But it's your job to pick the model and parameters that are suitable for your own situation.
+  We cannot help with such questions unless they appear to be a bug in tensorpack.
+
++ "Why my own model doesn't perform well?", "I don't understand this paper you implement.",
+  "How should I change the parameters for my own dataset?"
+
+  We do not answer machine learning questions.
+
+
+You can also use gitter (https://gitter.im/tensorpack/users) for more casual discussions.

.github/ISSUE_TEMPLATE/feature-requests.md

@@ -0,0 +1,15 @@
+---
+name: Feature Requests
+about: Suggest an idea for Tensorpack
+
+---
+
++ Note that you can implement a lot of features by extending Tensorpack
+  (See http://tensorpack.readthedocs.io/en/latest/tutorial/index.html#extend-tensorpack).
+  It does not have to be added to Tensorpack unless you have a good reason.
+
++ "Could you improve/implement an example/paper ?"
+  -- The answer is: we have no plans to do so. We don't consider feature
+  requests for examples or implement a paper for you, unless it demonstrates 
+  some Tensorpack features not yet demonstrated in the existing examples.
+  If you don't know how to do something yourself, you may ask a usage question.

.github/ISSUE_TEMPLATE/unexpected-problems---bugs.md

@@ -0,0 +1,53 @@
+---
+name: Unexpected Problems / Bugs
+about: Report unexpected problems about Tensorpack or its examples.
+
+---
+
+If you're asking about an unexpected problem you met, use this template.
+__PLEASE DO NOT DELETE THIS TEMPLATE, FILL IT__:
+
+### 1. What you did:
+
+(1) **If you're using examples, what's the command you run:**
+
+(2) **If you're using examples, have you made any changes to the examples? Paste them here:**
+
+(3) **If not using examples, tell us what you did here:**
+
+Note that we may not be able to investigate it if there is no reproducible code.
+It's always better to paste what you did instead of describing them.
+
+### 2. What you observed:
+
+(1) **Include the ENTIRE logs here:**
+
+It's always better to paste what you observed instead of describing them.
+
+A part of logs is sometimes enough, but it's always better to paste as much as possible.
+
+You can run a command with `CMD 2>&1 | tee logs.txt` to save all stdout & stderr logs to one file.
+
+(2) **Other observations, if any:**
+For example, CPU/GPU utilization, output images, tensorboard curves, if relevant to your issue.
+
+### 3. What you expected, if not obvious.
+
+If you expect higher accuracy, only in one of the two conditions can we help with it:
+(1) You're unable to match the accuracy documented in tensorpack examples.
+(2) It appears to be a tensorpack bug.
+
+Otherwise, how to get high accuracy is a machine learning question and is
+not our responsibility to figure out.
+
+### 4. Your environment:
+  + Python version:
+  + TF version: `python -c 'import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)'`.
+  + Tensorpack version: `python -c 'import tensorpack; print(tensorpack.__version__);'`.
+      You can install Tensorpack master by `pip install -U git+https://github.com/ppwwyyxx/tensorpack.git`
+      and see if your issue is already solved.
+  + If you're not using tensorpack under a normal command line shell (e.g.,
+    using an IDE or jupyter notebook), please retry under a normal command line shell. 
+  + Hardware information, e.g. number of GPUs used.
+
+About efficiency issues, PLEASE first read http://tensorpack.readthedocs.io/en/latest/tutorial/performance-tuning.html

.github/ISSUE_TEMPLATE/using-understanding-tensorpack.md

@@ -0,0 +1,37 @@
+---
+name: Using/Understanding Tensorpack
+about: More general questions about Tensorpack.
+
+---
+
++ If you did something with tensorpack and it failed, please use the "Unexpected Problems /
+  Bugs" category.
+
++ Your question is probably answered in [tutorials](http://tensorpack.readthedocs.io/en/latest/tutorial/index.html#user-tutorials). Read it first.
+
++ We answer "HOW to do X with Tensorpack" for a well-defined specific X.
+  X must be something that you conceptually know how to do, but are unable to do due to lack of knowledge about Tensorpack.
+  We don't answer vague questions like "How to do semantic segmentation in Tensorpack".
+
++ We also answer "HOW/WHY Tensorpack does X" for some X that Tensorpack or its examples are doing.
+
+
+Some typical questions that we DO NOT answer:
+
++ "Could you improve/implement an example/paper?":
+
+  We have no plans to do so. We don't consider feature
+  requests for examples or implement a paper for you.
+  If you don't know how to do something yourself, you may ask a usage question.
++ "The examples do not perform as expected after I change the models/dataset/parameters/etc.":
+
+  Tensorpack maintainers make sure the examples perform well without modifications.
+  But it's your job to pick the model and parameters that are suitable for your own situation.
+  We cannot help with such questions unless they appear to be a bug in tensorpack.
+
++ "Why my own model doesn't perform well?", "I don't understand this paper you implement.",
+  "How should I change the parameters for my own dataset?":
+
+  We do not answer machine learning questions.
+
+You can also use gitter (https://gitter.im/tensorpack/users) for more casual discussions.

.github/pull_request_template.md

@@ -0,0 +1,18 @@
+Thanks for your contribution!
+
+Unless you want to send a simple several lines of PR that can be easily merged, please note the following:
+
+* If you want to add a new feature, 
+  please open an issue first and indicate that you want to contribute.
+
+  There are features that we prefer to not add to tensorpack, e.g. symbolic models
+  (see details at https://tensorpack.readthedocs.io/tutorial/symbolic.html).
+  Therefore it's good to have a discussion first.
+
+* If you want to add a new example, please note that:
+
+  1. We prefer to not have an example that is too similar to existing ones in terms of the tasks.
+
+  2. Examples have to be able to reproduce (preferrably in some mesurable metrics) published or well-known experiments and results.
+
+* Please run `flake8 .` under the root of this repo to lint your code, and make sure the command produces no output.

.lgtm.yml

@@ -0,0 +1,16 @@
+queries:
+  - exclude: py/unguarded-next-in-generator
+  - exclude: py/explicit-call-to-delete
+  - exclude: py/polluting-import
+  - exclude: py/import-and-import-from
+  - exclude: py/similar-function
+  - exclude: py/unused-local-variable
+  # buggy: https://discuss.lgtm.com/t/python-false-positive-about-super/1330/3
+  - exclude: py/super-not-enclosing-class
+  - exclude: py/unreachable-statement
+extraction:
+  python:
+    prepare:
+      packages:
+        - libcap-dev
+

.travis.yml

@@ -0,0 +1,108 @@
+sudo: false
+dist: trusty
+language: python
+
+cache:
+  pip: true
+  apt: true
+  directories:
+    - $HOME/tensorpack_data
+
+addons:
+  apt:
+    packages:
+    - libprotobuf-dev
+    - protobuf-compiler
+
+# Test with TF 1.3-latest
+matrix:
+  fast_finish: true
+  include:
+  - os: linux
+    python: 2.7
+    env: TF_VERSION=1.3.0 TF_TYPE=release
+  - os: linux
+    python: 3.6
+    env: TF_VERSION=1.3.0 TF_TYPE=release
+  - os: linux
+    python: 2.7
+    env: TF_VERSION=1.12.0 TF_TYPE=release
+  - os: linux
+    python: 3.6
+    env: TF_VERSION=1.12.0 TF_TYPE=release PYPI=true
+  - os: linux
+    python: 2.7
+    env: TF_TYPE=nightly
+  - os: linux
+    python: 3.6
+    env: TF_TYPE=nightly
+  allow_failures:
+  - env: TF_TYPE=nightly
+
+install:
+- pip install -U pip    # the pip version on travis is too old
+- pip install .
+- pip install flake8 scikit-image opencv-python lmdb h5py msgpack
+  # check that dataflow can be imported alone without tensorflow
+- python -c "import tensorpack.dataflow"
+- ./tests/install-tensorflow.sh
+
+before_script:
+- flake8 --version
+- protoc --version
+- python -c "import cv2; print('OpenCV '+ cv2.__version__)"
+- python -c "import tensorflow as tf; print('TensorFlow '+ tf.__version__)"
+# Check that these private names can be imported because tensorpack is using them
+- python -c "from tensorflow.python.client.session import _FetchHandler"
+- python -c "from tensorflow.python.training.monitored_session import _HookedSession"
+- python -c "import tensorflow as tf; tf.Operation._add_control_input"
+
+script:
+- flake8 .
+- if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then cd examples && flake8 .; fi  # some examples are py3 only
+- mkdir -p $HOME/tensorpack_data
+- export TENSORPACK_DATASET=$HOME/tensorpack_data
+- $TRAVIS_BUILD_DIR/tests/run-tests.sh
+- cd $TRAVIS_BUILD_DIR  # go back to root so that deploy may work
+
+notifications:
+  email:
+    recipients:
+      - [email protected]
+    on_success: never
+    on_failure: always
+  webhooks:
+    urls:
+      - https://webhooks.gitter.im/e/cede9dbbf6630b3704b3
+    on_success: change  # options: [always|never|change] default: always
+    on_failure: always  # options: [always|never|change] default: always
+    on_start: never     # options: [always|never|change] default: always
+
+
+# see https://docs.travis-ci.com/user/deployment/pypi/
+deploy:
+- provider: pypi
+  user: ppwwyyxx
+  distributions: "sdist bdist_wheel"
+  skip_upload_docs: true
+  password:
+    secure: lnNfzPFSk4HF7seuu63CoUa0g4V0JPs42H5FTzWecEIro8IqjdaAvzNKuhu9E4rkrMXPzoYYA6gC4YhseT7N/jg7lyV/Nn1koGXK1gmzu5JnXZXHw5/Ri0I3hOF3OFpEvkR39tzV0r5HsO0JIU3pl11+bBS8iloEtzdTPaUaRgyYxyZGrHl8l3VnUyM50PRnPGDouK6DHxJmknnFCOZFCrEpwN4zpOs55c1rChjJ8aql69rSsXQOUP8++jRtGwgqEvHh0K86uGm1AJUYvSWmcoD+5Urcg8LGaQzySmXtJnFgxtzsORactIEJoAteSMXufWZITn4OQ5VQkc4/CfU0HgHioQw86dpdJrfOLkhEx65JxfUsQiOTgpcTOgYYEda6+dY4cnTTpj2fASVDhQMQ/vo+Ab/W14nYG8z0IPwGJ1qDXRO6AtPD7vbah2LdHQTUTiAbbtva7NWuKbFiVxD2tdrVQHKWqmKXw+JF5F8TBDEnUxFtulW2hbM+vwu6mPxbYQZEpONjLKSa7qiZ8jQZ8cy9KatQYkqPLeGEbgI+IsmA4bnJJennToyWZ2N4W071ddtLB7hDH4ZRVdaLVYtfeKW/b/+YGX3N8p5cMKGIDjpGyF0BocFalQ7gYfg2ouAn1RyEPkCaw6ntA2uzIgvTqxU5inWJCFn20Ogst0oIaPs=
+  on:
+    tags: true
+    branch: master
+    repo: tensorpack/tensorpack
+    python: "3.6"
+    condition: "$PYPI = true"
+
+- provider: pypi
+  server: https://testpypi.python.org/pypi
+  user: ppwwyyxx
+  distributions: "sdist bdist_wheel"
+  skip_upload_docs: true
+  password:
+    secure: lnNfzPFSk4HF7seuu63CoUa0g4V0JPs42H5FTzWecEIro8IqjdaAvzNKuhu9E4rkrMXPzoYYA6gC4YhseT7N/jg7lyV/Nn1koGXK1gmzu5JnXZXHw5/Ri0I3hOF3OFpEvkR39tzV0r5HsO0JIU3pl11+bBS8iloEtzdTPaUaRgyYxyZGrHl8l3VnUyM50PRnPGDouK6DHxJmknnFCOZFCrEpwN4zpOs55c1rChjJ8aql69rSsXQOUP8++jRtGwgqEvHh0K86uGm1AJUYvSWmcoD+5Urcg8LGaQzySmXtJnFgxtzsORactIEJoAteSMXufWZITn4OQ5VQkc4/CfU0HgHioQw86dpdJrfOLkhEx65JxfUsQiOTgpcTOgYYEda6+dY4cnTTpj2fASVDhQMQ/vo+Ab/W14nYG8z0IPwGJ1qDXRO6AtPD7vbah2LdHQTUTiAbbtva7NWuKbFiVxD2tdrVQHKWqmKXw+JF5F8TBDEnUxFtulW2hbM+vwu6mPxbYQZEpONjLKSa7qiZ8jQZ8cy9KatQYkqPLeGEbgI+IsmA4bnJJennToyWZ2N4W071ddtLB7hDH4ZRVdaLVYtfeKW/b/+YGX3N8p5cMKGIDjpGyF0BocFalQ7gYfg2ouAn1RyEPkCaw6ntA2uzIgvTqxU5inWJCFn20Ogst0oIaPs=
+  on:
+    branch: test-travis
+    repo: tensorpack/tensorpack
+    python: "3.6"
+    condition: "$PYPI = true"

CHANGES.md

@@ -0,0 +1,74 @@
+
+## Breaking API changes.
+
+Tensorpack is in development, and API changes can happen.
+The backward compatibilty will be __preserved for at least several months__, with a deprecation warning,
+so you don't need to look at here very often.
+
+Here are a list of things that were changed, starting from an early version.
+TensorFlow itself also changes API and those are not listed here.
+
++ [2018/08/27] msgpack is used again for "serialization to disk", because pyarrow
+  has no compatibility between versions. To use pyarrow instead, `export TENSORPACK_COMPATIBLE_SERIALIZE=pyarrow`.
++ [2018/04/05] msgpack is replaced by pyarrow in favor of its speed. If you want old behavior,
+	`export TENSORPACK_SERIALIZE=msgpack`. It's later found that pyarrow is unstable and may lead to crash.
++ [2018/03/20] `ModelDesc` starts to use simplified interfaces:
+	+ `_get_inputs()` renamed to `inputs()` and returns `tf.placeholder`s.
+	+ `build_graph(self, tensor1, tensor2)` returns the cost tensor directly.
+	+ `_get_optimizer()` renamed to `optimizer()`.
+	Old interface will still be available for a while, but new ones are recommended.
++ [2018/03/12] `JSONWriter` use a different file name, and will not automatically restore epoch number.
+	`AutoResumeTrainConfig` was added to support resuming.
++ [2017/10/21]
+	tensorpack is gradually switching to a new Trainer API.
+	The old API will keep working for a while. See [issue](https://github.com/tensorpack/tensorpack/issues/458)
+	for details.
++ [2017/10/18]
+	`TrainConfig(predict_tower)` was deprecated. You can set the inference device directly when creating the `InferenceRunner` callback.
++ [2017/10/12](https://github.com/tensorpack/tensorpack/commit/7e963996f615b85f7459455596b4ee9bbd0bce8e).
+	`tensorpack.RL` was deprecated. The RL examples are written with OpenAI gym interface instead.
++ [2017/10/10](https://github.com/tensorpack/tensorpack/commit/7d40e049691d92018f50dc7d45bba5e8b140becc).
+	`tfutils.distributions` was deprecated in favor of `tf.distributions` introduced in TF 1.3.
++ [2017/08/02](https://github.com/tensorpack/tensorpack/commit/875f4d7dbb5675f54eae5675fa3a0948309a8465).
+	`Trainer.get_predictor` now takes GPU id. And `Trainer.get_predictors` was deprecated.
++ 2017/06/07. Now the library explicitly depends on msgpack-numpy>=0.3.9. The serialization protocol
+	becomes incompatible if you've been using <0.3.9.
++ [2017/05/06](https://github.com/tensorpack/tensorpack/commit/0774ec66e66075486f6a36aba63cc2a151b9fec8).
+	`replace_get_variable` was deprecated in favor of the official `custom_getter` interface.
+	`{freeze,remap}_get_variable` was renamed to `{freeze,remap}_variables`.
++ [2017/04/09](https://github.com/tensorpack/tensorpack/commit/5beab907895aec36bdcaed62e25b976aad7979b8).
+	`ParamRestore` was renamed to `DictRestore`.
++ [2017/03/16](https://github.com/tensorpack/tensorpack/commit/ccae46f4a3ca89dc3df901a338eef8447d19a730).
+	`session_config` option in `PredictConfig` is deprecated. Use `session_creator` to define how to create session instead.
++ 2017/02/20. The interface of step callbacks are changed to be the same as `tf.train.SessionRunHook`.
+	If you haven't written any custom step callbacks, there is nothing to do. Otherwise please refer
+	to the [existing callbacks](https://github.com/tensorpack/tensorpack/blob/master/tensorpack/callbacks/steps.py).
++ [2017/02/12](https://github.com/tensorpack/tensorpack/commit/d1041a77a9c59d8c9abf64f389f3b605d65b483e).
+	`TrainConfig(optimizer=)` was deprecated. Now optimizer is set in `ModelDesc`. And gradient processors become part of an optimizer.
+* [2017/02/11](https://github.com/tensorpack/tensorpack/commit/5b29bda9f17d7b587259e13963c4c8093e8387f8).
+	`_get_input_vars()` in `ModelDesc` was renamed to `_get_inputs`. `InputVar` was renamed to `InputDesc`.
+* [2017/01/27](https://github.com/tensorpack/tensorpack/commit/a9dd0b8ec34209ab86a92875589dbbc4716e73ef).
+	`TrainConfig(step_per_epoch)` was renamed to `steps_per_epoch`.
+* [2017/01/25](https://github.com/tensorpack/tensorpack/commit/2df3dcf401a99fe61c699ad719e95528872d3abe).
+	Argument order of `models.ConcatWith` is changed to follow the API change in TensorFlow upstream.
+* [2017/01/25](https://github.com/tensorpack/tensorpack/commit/243e957fe6d62a0cfb5728bd77fb3e005d6603e4).
+	`TrainConfig(callbacks=)` now takes a list of `Callback` instances.
+* [2017/01/06](https://github.com/tensorpack/tensorpack/commit/bbf41d9e58053f843d0471e6d2d87ff714a79a90).
+	`summary.add_moving_summary` now takes any number of positional arguments instead of a list.
+* [2017/01/05](https://github.com/tensorpack/tensorpack/commit/651a5aea8f9aacad7147542021dcf106fc824bc2).
+	The argument `TrainConfig(dataset=)` is renamed to `TrainConfig(dataflow=)`.
+* [2016/12/15](https://github.com/tensorpack/tensorpack/commit/99c70935a7f72050f45891fbbcc49c4ce43aedce).
+	The `predict_tower` option is in `TrainConfig` now instead of `Trainer`.
+* [2016/11/10](https://github.com/tensorpack/tensorpack/commit/77bcc8b1afc984a569f6ec3eda0a3c47b4e2923a).
+	The `{input,output}_var_names` argument in `PredictConfig` is renamed to `{input,output}_names`.
+* [2016/11/06](https://github.com/tensorpack/tensorpack/commit/740e9d8ca146af5a911f68a369dd7348243a2253).
+	The inferencer `ClassificationError` now expects the vector tensor returned by `prediction_incorrect` instead of the "wrong" tensor.
+* [2016/10/17](https://github.com/tensorpack/tensorpack/commit/6eb0bebe60d6f38bcad9ddb3e6091b0b154a09cf).
+	`Conv2D` and `FullyConnect` use `tf.identity` by default instead of `tf.nn.relu`.
+* [2016/09/01](https://github.com/tensorpack/tensorpack/commit/fc9e45b0208ff09daf454d3bd910c540735b7f83).
+	The method `_build_graph` of `ModelDesc` doesn't take `is_training` argument anymore.
+	The `is_training` attribute can be obtained from tower context.
+* [2016/05/15](https://github.com/tensorpack/tensorpack/commit/e69034b5c9b588db9fb52295b1e63c89e8b42654).
+	The method `_get_cost` of `ModelDesc` is replaced by `_build_graph`.
+
+