Merge branch 'unstable'

Author: Grokzen

CHANGES

@@ -1,3 +1,12 @@
+* 0.3.0
+    * simple benchmark now uses docopt for cli parsing
+    * New make target to run some benchmarks 'make benchmark'
+    * simple benchmark now support pipelines tests
+    * Renamed RedisCluster --> StrictRedisCluster
+    * Implement backwards compatible redis.Redis class in cluster mode. It was named RedisCluster and everyone updating from 0.2.0 to 0.3.0 should consult docs/Upgrading.md for instructions how to change your code.
+    * Added comprehensive documentation regarding pipelines
+    * Meta retrieval commands(slots, nodes, info) for Redis Cluster. (iandyh)
+
 * 0.2.0
     * Moved pipeline code into new file.
     * Code now uses a proper cluster connection pool class that handles 

MANIFEST.in

@@ -3,5 +3,4 @@ include docs/License.txt
 include CHANGES
 include setup.py
 include README.md
-include docs/ALPHA.md
-include docs/Authorrs
+include docs/Authors

Makefile

@@ -120,6 +120,7 @@ help:
 	@echo "  sdist           make a source distribution"
 	@echo "  bdist           make an egg distribution"
 	@echo "  install         install package"
+	@echo "  benchmark       runs all benchmarks. assumes nodes running on port 7001 and 7007"
 	@echo " *** CI Commands ***"
 	@echo "  start           starts a test redis cluster"
 	@echo "  stop            stop all started redis nodes (Started via 'make start' only affected)"
@@ -209,4 +210,18 @@ redis-install:
 	gem install redis
 	sleep 3
 
+benchmark:
+	@echo ""
+	@echo " -- Running Simple benchmark with StrictRedis lib and non cluster server --"
+	python benchmarks/simple.py --port 7007 --timeit --nocluster
+	@echo ""
+	@echo " -- Running Simple benchmark with StrictRedisCluster lib and cluster server --"
+	python benchmarks/simple.py --port 7001 --timeit
+	@echo ""
+	@echo " -- Running Simple benchmark with pipelines & StrictRedis lib and non cluster server --"
+	python benchmarks/simple.py --port 7007 --timeit --pipeline --nocluster
+	@echo ""
+	@echo " -- Running Simple benchmark with StrictRedisCluster lib and cluster server"
+	python benchmarks/simple.py --port 7001 --timeit --pipeline
+
 .PHONY: test

README.md

@@ -4,19 +4,35 @@ Redis cluster client in python for the official cluster support targeted for red
 
 This project is a port of `redis-rb-cluster` by antirez, with alot of added functionality. The original source can be found at https://github.com/antirez/redis-rb-cluster
 
-[![Build Status](https://travis-ci.org/Grokzen/redis-py-cluster.svg?branch=master)](https://travis-ci.org/Grokzen/redis-py-cluster) [![Coverage Status](https://coveralls.io/repos/Grokzen/redis-py-cluster/badge.png)](https://coveralls.io/r/Grokzen/redis-py-cluster) [![Latest Version](https://pypip.in/version/redis-py-cluster/badge.svg)](https://pypi.python.org/pypi/redis-py-cluster/) [![Downloads](https://pypip.in/download/redis-py-cluster/badge.svg)](https://pypi.python.org/pypi/redis-py-cluster/) [![Supported Python versions](https://pypip.in/py_versions/redis-py-cluster/badge.svg)](https://pypi.python.org/pypi/redis-py-cluster/) [![License](https://pypip.in/license/redis-py-cluster/badge.svg)](https://pypi.python.org/pypi/redis-py-cluster/) [![Gitter](https://badges.gitter.im/Join Chat.svg)](https://gitter.im/Grokzen/redis-py-cluster?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Code Health](https://landscape.io/github/Grokzen/redis-py-cluster/unstable/landscape.svg)](https://landscape.io/github/Grokzen/redis-py-cluster/unstable)
+[![Build Status](https://travis-ci.org/Grokzen/redis-py-cluster.svg?branch=master)](https://travis-ci.org/Grokzen/redis-py-cluster) [![Coverage Status](https://coveralls.io/repos/Grokzen/redis-py-cluster/badge.png)](https://coveralls.io/r/Grokzen/redis-py-cluster) [![PyPI version](https://badge.fury.io/py/pykwalify.svg)](http://badge.fury.io/py/pykwalify) [![Gitter](https://badges.gitter.im/Join Chat.svg)](https://gitter.im/Grokzen/redis-py-cluster?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Code Health](https://landscape.io/github/Grokzen/redis-py-cluster/unstable/landscape.svg)](https://landscape.io/github/Grokzen/redis-py-cluster/unstable)
+
+
+
+# Project status
+
+The project is not dead but not much new development is done right now. I do awnser issue reports and pull requests as soon as possible and if you have a problem you can ping me inside the gitter channel that you can find here [![Gitter](https://badges.gitter.im/Join Chat.svg)](https://gitter.im/Grokzen/redis-py-cluster?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) and i will help you out with problems or usage of this lib.
+
+As of release `0.3.0` this project will be considered stable and usable in production. Just remember that if you are going to use redis cluster to please reda up on the documentation that you can find in the bottom of this Readme. It will contain usage examples and descriptions of what is implemented and what is not implemented and why things are the way they are.
+
+On the topic about porting/moving this code into `redis-py` there is currently work over here https://github.com/andymccurdy/redis-py/pull/604 that will bring cluster uspport based on this code. But my suggestion is that until that work is completed that you should use this lib.
+
+
+
+## Upgrading instructions
+
+Please read the [following](docs/Upgrading.md) documentation that will go through all changes that is required when upgrading `redis-py-cluster` between versions.
 
 
 
 ## Dependencies & supported python versions
 
-- redis >= 2.10.2
-- Cluster enabled redis servers. Only Redis 3.0 beta.7 and above is supported because of CLUSTER SLOTS command was introduced in that release.
-- Optional: hiredis >= 0.1.3
+- Python: redis >= `2.10.2` is required
+- Redis server >= `3.0.0` is required
+- Optional Python: hiredis >= `0.1.3`
 
 Hiredis is tested and supported on all supported python versions.
 
-Supported python versions:
+Supported python versions, all minor releases in each major version should be supported unless otherwise stated here:
 
 - 2.7.x
 - 3.2.x
@@ -48,20 +64,23 @@ $ python setup.py install
 Small sample script that show how to get started with RedisCluster. `decode_responses=True` is required to have when running on python3.
 
 ```python
->>> from rediscluster import RedisCluster
+>>> from rediscluster import StrictRedisCluster
 >>> startup_nodes = [{"host": "127.0.0.1", "port": "7000"}]
->>> rc = RedisCluster(startup_nodes=startup_nodes, decode_responses=True)
+>>> rc = StrictRedisCluster(startup_nodes=startup_nodes, decode_responses=True)
 >>> rc.set("foo", "bar")
 True
 >>> rc.get("foo")
 'bar'
 ```
 
-The following imports can be imported from `redis` package. 
+The following imports can be imported from `redis` package.
 
-- `RedisCluster` 
-- `StrictClusterPipeline` 
-- `ClusterPubSub` 
+- `StrictRedisCluster`
+- `RedisCluster`
+- `StrictClusterPipeline`
+- `ClusterPubSub`
+
+`StrictRedisCluster` is based on `redis.StrictRedis` and `RedisCluster` has the same functionality as `redis.Redis` even if it is not directly based on it.
 
 
 
@@ -71,38 +90,36 @@ All tests are currently built around a 6 redis server cluster setup (3 masters +
 
 The easiest way to setup a cluster is to use either a Docker or Vagrant. They are both described in [Setup a redis cluster. Manually, Docker & Vagrant](docs/Cluster_Setup.md).
 
+To run all tests in all supported environments with `tox` read this [Tox multienv testing](docs/Tox.md)
 
 
-### Tox - Multi environment testing
-
-Tox is the easiest way to run all tests because it will manage all dependencies and run the correct test command for you.
-
-TravisCI will use tox to run tests on all supported python & hiredis versions.
-
-Install tox with `pip install tox`
-
-To run all environments you need all supported python versions installed on your machine. (See supported python versions list) and you also need the python-dev package for all python versions to build hiredis.
-
-To run a specific python version use either `tox -e py27` or `tox -e py34`
-
 
 ## More documentation
 
 More detailed documentation can be found in `docs` folder.
 
+- [Benchmarks](docs/Benchmarks.md)
 - [Pubsub](docs/Pubsub.md)
 - [Setup a redis cluster. Manually, Docker & Vagrant](docs/Cluster_Setup.md)
 - [Command differences](docs/Commands.md)
 - [Limitations and differences](docs/Limits_and_differences.md)
 - [Redisco support (Django ORM)](docs/Redisco.md)
+- [Pipelines](docs/Pipelines.md)
 - [Threaded Pipeline support](docs/Threads.md)
+- [Cluster Management class](docs/ClusterMgt.md)
 - [Authors](docs/Authors)
 
 
 
 ## Disclaimer
 
-Both this client and Redis Cluster are a work in progress that is not suitable to be used in production environments. This is only my current personal opinion about both projects.
+Both Redis cluster and redis-py-cluster is considered stable and production ready.
+
+But this depends on what you are going to use clustering for. In the simple use cases with SET/GET and other single key functions there is not issues. If you require multi key functinoality or pipelines then you must be very carefull when developing because they work slightly different from the normal redis server.
+
+If you require advance features like pubsub or scripting, this lib and redis do not handle that kind of use-cases very well. You either need to develop a custom solution yourself or use a non clustered redis server for that.
+
+Finally, this lib itself is very stable and i know of atleast 2 companies that use this in production with high loads and big cluster sizes.
 
 
 

benchmarks/simple.py

@@ -2,10 +2,9 @@
 
 # python std lib
 import time
-import argparse
 
-# rediscluster imports
-from rediscluster.rediscluster import RedisCluster
+# 3rd party imports
+from docopt import docopt
 
 
 def loop(rc, reset_last_key=None):
@@ -35,67 +34,79 @@ def loop(rc, reset_last_key=None):
         except Exception as e:
             print("error {0}".format(e))
 
-        time.sleep(0.1)
+        time.sleep(0.05)
 
 
 def timeit(rc, itterations=50000):
-    """ Time how long it take to run a number of set/get:s
+    """
+    Time how long it take to run a number of set/get:s
     """
     t0 = time.time()
     for i in xrange(0, itterations):  # noqa
-        try:
-            s = "foo{0}".format(i)
-            rc.set(s, i)
-            rc.get(s)
-        except Exception as e:
-            print("error {0}".format(e))
+        s = "foo{0}".format(i)
+        rc.set(s, i)
+        rc.get(s)
 
     t1 = time.time() - t0
-    print("{}k SET and then GET took: {} seconds... {} itterations per second".format((itterations / 1000), t1, (itterations / t1)))
+    print("{}k SET/GET operations took: {} seconds... {} operations per second".format((itterations / 1000) * 2, t1, (itterations / t1) * 2))
+
+
+def timeit_pipeline(rc, itterations=50000):
+    """
+    Time how long it takes to run a number of set/get:s inside a cluster pipeline
+    """
+    t0 = time.time()
+    for i in xrange(0, itterations):  # noqa
+        s = "foo{0}".format(i)
+
+        p = rc.pipeline()
+        p.set(s, i)
+        p.get(s)
+        p.execute()
+
+    t1 = time.time() - t0
+    print("{}k SET/GET operations inside pipelines took: {} seconds... {} operations per second".format((itterations / 1000) * 2, t1, (itterations / t1) * 2))
 
 
 if __name__ == "__main__":
-    parser = argparse.ArgumentParser(
-        conflict_handler="resolve",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
-    )
-    parser.add_argument(
-        "-h",
-        "--host",
-        help="host of a cluster member",
-        default="127.0.0.1"
-    )
-    parser.add_argument(
-        "-p",
-        "--port",
-        help="port of a cluster member",
-        type=int,
-        default=7000
-    )
-    parser.add_argument(
-        "--timeit",
-        help="run a mini benchmark to test performance",
-        action="store_true"
-    )
-    parser.add_argument(
-        "--resetlastkey",
-        help="reset __last__ key",
-        action="store_true"
-    )
-    args = parser.parse_args()
-
-    startup_nodes = [
-        {"host": args.host, "port": args.port}
-    ]
-
-    rc = RedisCluster(startup_nodes=startup_nodes, max_connections=32, socket_timeout=0.1, decode_responses=True)
-
-    if args.timeit:
+    __docopt__ = """
+Usage:
+  simple [--host IP] [--port PORT] [--nocluster] [--timeit] [--pipeline] [--resetlastkey] [-h] [--version]
+
+Options:
+  --nocluster        If flag is set then StrictRedis will be used instead of cluster lib
+  --host IP          Redis server to test against [default: 127.0.0.1]
+  --port PORT        Port on redis server [default: 7000]
+  --timeit           run a mini benchmark to test performance
+  --pipeline         Only usable with --timeit flag. Runs SET/GET inside pipelines.
+  --resetlastkey     reset __last__ key
+  -h --help          show this help and exit
+  -v --version       show version and exit
+    """
+
+    args = docopt(__docopt__, version="0.3.0")
+
+    startup_nodes = [{"host": args["--host"], "port": args["--port"]}]
+
+    if not args["--nocluster"]:
+        from rediscluster import StrictRedisCluster
+        rc = StrictRedisCluster(startup_nodes=startup_nodes, max_connections=32, socket_timeout=0.1, decode_responses=True)
+    else:
+        from redis import StrictRedis
+        rc = StrictRedis(host=args["--host"], port=args["--port"], socket_timeout=0.1, decode_responses=True)
+
+    if args["--timeit"]:
         test_itterstions = [
+            5000,
             10000,
-            25000,
+            20000,
         ]
-        for itterations in test_itterstions:
-            timeit(rc, itterations=itterations)
+
+        if args["--pipeline"]:
+            for itterations in test_itterstions:
+                timeit_pipeline(rc, itterations=itterations)
+        else:
+            for itterations in test_itterstions:
+                timeit(rc, itterations=itterations)
     else:
-        loop(rc, reset_last_key=args.resetlastkey)
+        loop(rc, reset_last_key=args["--resetlastkey"])

dev-requirements.txt

@@ -5,3 +5,4 @@ hiredis >= 0.1.3
 pytest >= 2.5.0
 testfixtures >= 4.0.1
 mock == 1.0.1
+docopt == 0.6.2

docs/ALPHA.md

@@ -17,3 +17,4 @@ Authors who contributed code or testing:
  - Dobrite - https://github.com/dobrite
  - 72squared - https://github.com/72squared
  - Neuron Teckid - https://github.com/neuront
+ - iandyh - https://github.com/iandyh

docs/Authors

@@ -0,0 +1,34 @@
+# Benchmarks
+
+There is a few benchmarks that is designed to test specific parts of the code that will show how big of a performance difference there is between using this lib and the normal Redis client.
+
+
+
+## Setup benchmarks
+
+Before running any benchmark you should install this lib in editable mode inside a virtualenv so it can import `StrictRedisCluster` lib.
+
+Install with
+
+```
+$ pip install -e .
+```
+
+You also need a few redis servers to test against. It is required to have 1 cluster with atleast one node on port `7001` and it also required to have a non-clustered server on port `7007`.
+
+
+## Bencmarks
+
+`simple.py` - This benchmark can be used to messure a simple `set` and `get` operation chain. It also support running pipelines bu adding the flag `--pipeline`
+
+
+
+## Run predefined benchmarks
+
+There is a set of predefined benchmarks that can be runned to messure performance drop from using this library.
+
+To run the benchmarks
+
+```
+make benchmark
+```