img2img_allen_brain problem defs (#872)

* img2img_allen_brain problem defs

only outstanding issue blocking this should be that the test in allen_brain_test.py that uses tf.eager breaks many/all other t2t tests not compatible with eager mode.

squashed these:

commit 76c8cd3f105c07919f84d70b02b1fbd38b5376fe
Author: cwbeitel <[email protected]>
Date:   Fri Jun 15 18:35:13 2018 +0000

    fix mock_raw_data

commit f0dc3a0
Merge: 3e17473 4b7360d
Author: cwbeitel <[email protected]>
Date:   Thu Jun 14 22:54:33 2018 +0000

    Merge branch 'allen-deps' of https://github.com/cwbeitel/tensor2tensor into allen-deps

commit 3e17473
Author: cwbeitel <[email protected]>
Date:   Fri Jun 8 17:04:39 2018 +0000

    cleanup

commit 3e8ef9d
Author: cwbeitel <[email protected]>
Date:   Wed Jun 6 23:07:28 2018 +0000

    removed image summary writing from metrics

    training was 10x slower than CIFAR img2img problem, profiled and found 91% of time was being sp
ent writing ImageSummary's, removed that from metrics and saw the speed increase as expected.

commit 4814556
Author: cwbeitel <[email protected]>
Date:   Tue May 29 16:00:17 2018 +0000

    basic img2img prob. defs that maybe get data

commit 320f06c
Author: cwbeitel <[email protected]>
Date:   Thu May 24 18:53:02 2018 +0000

    test without accessing network, follow rcfile

    * made test that accesses remote API optional, off by default
    * included test that runs sub-image extractor on mocked raw input image within mocked directory
 structure mirroring what would be produced by running actual data downloader
    * refactoring according to rcfile

commit 503123b
Author: cwbeitel <[email protected]>
Date:   Wed May 23 22:09:31 2018 +0000

    add [allen] deps to travis setup

commit 239188b
Author: cwbeitel <[email protected]>
Date:   Wed May 23 21:39:13 2018 +0000

    data download and sub-image gen utility

    * if additional [allen] deps not present suggest running `pip install tensor2tensor[allen]`
    * obtains a list of section dataset ID's from Allen Institute API
    * downloads specified number of raw images from specified number of section datasets
    * produces specified number of sub-images of specified size from raw images.
    * simple e2e test that runs the data downloader and sub-image generator with default parameters

commit af34cc6
Author: cwbeitel <[email protected]>
Date:   Wed May 23 17:47:35 2018 +0000

    deps. for allen brain img probs. via extras

* run allen tests separately on ci

* travis debug

* include previously excluded setup.py that defines [allen] step

* smaller e2e test, maybe addresses travis OOM

* linter fixes

* Remove allensdk dependency; various improvements

* Simplify allen_brain_utils* to download smaller static list of images, avoiding allensdk dependency.

* Simplify _generator given input is already in [0, 255]

* _generator ignores likely non-tissue/background regions based on max intensity

* Fractional in-painting step added to pre-processor, depends on inpaint_fraction property, randomly positioned square region

* Random crop to target dimensions instead of downsampling leaving full resolution of target intact

* Re-using same generated data for different problems that share examples (differing in terms of how these are pre-processed)

* cleanup

Author: cwbeitel

.travis.yml

@@ -42,7 +42,7 @@ install:
   - pip install -q .
   - t2t-trainer --registry_help
   # Then install the test dependencies
-  - pip install -q .[tests]
+  - pip install -q .[tests,allen]
   # Make sure to install the atari extras for gym
   - pip install "gym[atari]"
   # Make sure we have the latest version of numpy - avoid problems we were
@@ -59,6 +59,8 @@ script:
   #   * trainer_lib_test
   #   * visualization_test
   #   * model_rl_experiment_test
+  #   * allen_brain_test
+  #   * allen_brain_utils_test
   #   * model_rl_experiment_stochastic_test
   #   * models/research
   # algorithmic_math_test: flaky
@@ -71,11 +73,15 @@ script:
     --ignore=tensor2tensor/data_generators/algorithmic_math_test.py
     --ignore=tensor2tensor/models/research/universal_transformer_test.py
     --ignore=tensor2tensor/rl/model_rl_experiment_test.py
+    --ignore=tensor2tensor/data_generators/allen_brain_test.py
+    --ignore=tensor2tensor/data_generators/allen_brain_utils_test.py
     --ignore=tensor2tensor/rl/model_rl_experiment_stochastic_test.py
     --ignore=tensor2tensor/models/research
   - pytest tensor2tensor/utils/registry_test.py
   - pytest tensor2tensor/utils/trainer_lib_test.py
   - pytest tensor2tensor/visualization/visualization_test.py
+  - pytest tensor2tensor/data_generators/allen_brain_test.py
+  - pytest tensor2tensor/data_generators/allen_brain_utils_test.py
   - if [[ "$TF_VERSION" == "$TF_LATEST"  ]] || [[ "$TF_VERSION" == "tf-nightly"  ]];
     then
       pytest tensor2tensor/models/research;

setup.py

@@ -58,6 +58,7 @@
             # explicit pip install gym[atari] for the tests.
             # 'gym[atari]',
         ],
+        'allen': ['Pillow==5.1.0', 'pandas==0.23.0']
     },
     classifiers=[
         'Development Status :: 4 - Beta',

tensor2tensor/data_generators/all_problems.py

@@ -23,6 +23,7 @@
 MODULES = [
     "tensor2tensor.data_generators.algorithmic",
     "tensor2tensor.data_generators.algorithmic_math",
+    "tensor2tensor.data_generators.allen_brain",
     "tensor2tensor.data_generators.audio",
     "tensor2tensor.data_generators.babi_qa",
     "tensor2tensor.data_generators.bair_robot_pushing",