-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* add CONTRIBUTING.md * update pynsq docs * update README.md (add Authors/Contributors) * update protocol docs * update nsqd docs * update INSTALLING.md
- Loading branch information
1 parent
4d1f5a6
commit d9af9e5
Showing
6 changed files
with
237 additions
and
52 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Contributing | ||
|
|
||
| Thanks for your interest in contributing to NSQ! | ||
|
|
||
| ## Getting Started | ||
|
|
||
| * make sure you have a [GitHub account](https://github.com/signup/free) | ||
| * submit a ticket for your issue, assuming one does not already exist | ||
| * clearly describe the issue including steps to reproduce when it is a bug | ||
| * identify specific versions of the binaries and client libraries | ||
| * fork the repository on GitHub | ||
|
|
||
| ## Making Changes | ||
|
|
||
| * create a branch from where you want to base your work | ||
| * we typically name branches according to the following format: `helpful_name_<issue_number>` | ||
| * make commits of logical units | ||
| * make sure your commit messages are in a clear and readable format, example: | ||
|
|
||
| ``` | ||
| nsqd: fixed bug in protocol_v2 | ||
| * update the message pump to properly account for RDYness | ||
| * cleanup variable names | ||
| * ... | ||
| ``` | ||
|
|
||
| * if you're fixing a bug or adding functionality it probably makes sense to write a test | ||
| * make sure to run `fmt.sh` and `test.sh` in the root of the repo to ensure that your code is | ||
| properly formatted and that tests pass (NOTE: we integrate Travis with GitHub for continuous | ||
| integration) | ||
|
|
||
| ## Submitting Changes | ||
|
|
||
| * push your changes to your branch in your fork of the repository | ||
| * submit a pull request against bitly's repository | ||
| * comment in the pull request when you're ready for the changes to be reviewed: `"ready for review"` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,41 +1,51 @@ | ||
| nsqd | ||
| ==== | ||
| ## nsqd | ||
|
|
||
| `nsqd` is the daemon that receives, buffers, and delivers messages to clients. It optionally connects to `nsqlookupd` | ||
| instances to announce topic and channels. It has a TCP API for clients, and HTTP API for publishing messages, | ||
| administrative actions, and statistics. | ||
| `nsqd` is the daemon that receives, buffers, and delivers messages to clients. | ||
|
|
||
| HTTP API | ||
| -------- | ||
| It is normally run alongside `nsqlookupd` instances to announce topic and channels but can be run | ||
| standalone. | ||
|
|
||
| * `/put?topic=...` [message is POST body] | ||
| It listens on two TCP ports, one for clients and another for the HTTP API. | ||
|
|
||
| curl -d "<message>" http://127.0.0.1:4151/put?topic=message_topic | ||
| ### HTTP API | ||
|
|
||
| * `/mput?topic=...` [messages are new line separated POST body] | ||
| * `/put?topic=...` | ||
|
|
||
| curl -d "<message>\n<message>" http://127.0.0.1:4151/put?topic=message_topic | ||
| POST message body | ||
|
|
||
| `$ curl -d "<message>" http://127.0.0.1:4151/put?topic=message_topic` | ||
|
|
||
| * `/mput?topic=...` | ||
|
|
||
| POST message body (`\n` separated) | ||
|
|
||
| `$ curl -d "<message>\n<message>" http://127.0.0.1:4151/put?topic=message_topic` | ||
|
|
||
| * `/empty_channel?topic=...&channel=...` | ||
| * `/delete_channel?topic=...&channel=...` | ||
| * `/stats` [?format=json] | ||
| * `/ping` (returns "OK" for use with monitoring) | ||
| * `/info` returns server version information. | ||
|
|
||
|
|
||
| Command Line Options | ||
| -------------------- | ||
|
|
||
| Usage of ./nsqd: | ||
| -data-path="": path to store disk-backed messages | ||
| -debug=false: enable debug mode | ||
| -http-address="0.0.0.0:4151": <addr>:<port> to listen on for HTTP clients | ||
| -lookupd-tcp-address=[]: lookupd TCP address (may be given multiple times) | ||
| -max-bytes-per-file=104857600: number of bytes per diskqueue file before rolling | ||
| -mem-queue-size=10000: number of messages to keep in memory (per topic) | ||
| -msg-timeout=60000: time (ms) to wait before auto-requeing a message | ||
| -sync-every=2500: number of messages between diskqueue syncs | ||
| -tcp-address="0.0.0.0:4150": <addr>:<port> to listen on for TCP clients | ||
| -verbose=false: enable verbose logging | ||
| -version=false: print version string | ||
| -worker-id=0: unique identifier (int) for this worker (will default to a hash of hostname) | ||
| * `/stats` | ||
|
|
||
| supports both text and JSON via `?format=json` | ||
|
|
||
| * `/ping` | ||
|
|
||
| returns `OK`, helpful when monitoring | ||
|
|
||
| * `/info` | ||
|
|
||
| returns version information | ||
|
|
||
| ### Command Line Options | ||
|
|
||
| -data-path="": path to store disk-backed messages | ||
| -debug=false: enable debug mode | ||
| -http-address="0.0.0.0:4151": <addr>:<port> to listen on for HTTP clients | ||
| -lookupd-tcp-address=[]: lookupd TCP address (may be given multiple times) | ||
| -max-bytes-per-file=104857600: number of bytes per diskqueue file before rolling | ||
| -mem-queue-size=10000: number of messages to keep in memory (per topic) | ||
| -msg-timeout=60000: time (ms) to wait before auto-requeing a message | ||
| -sync-every=2500: number of messages between diskqueue syncs | ||
| -tcp-address="0.0.0.0:4150": <addr>:<port> to listen on for TCP clients | ||
| -verbose=false: enable verbose logging | ||
| -version=false: print version string | ||
| -worker-id=0: unique identifier (int) for this worker (will default to a hash of hostname) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| ## pynsq | ||
|
|
||
| `pynsq` is a Python NSQ client library. | ||
|
|
||
| It provides a high-level reader library for building consumers and two low-level modules for both | ||
| sync and async communication over the NSQ protocol (if you wanted to write your own high-level | ||
| functionality). | ||
|
|
||
| The async module is built on top of the [Tornado IOLoop][tornado] and as such requires `tornado` be | ||
| installed: | ||
|
|
||
| `$ pip install tornado` | ||
|
|
||
| ### Reader | ||
|
|
||
| Reader provides high-level functionality for building robust NSQ consumers in Python on top of the | ||
| async module. | ||
|
|
||
| Multiple reader instances can be instantiated in a single process (to consume from multiple | ||
| topics/channels at once). Each specifying a set of tasks that will be called for each message over | ||
| that channel. Tasks are defined as a dictionary of string names -> callables passed as | ||
| `all_tasks` during instantiation. | ||
|
|
||
| `preprocess_method` defines an optional callable that can alter the message data before other task | ||
| functions are called. | ||
|
|
||
| `validate_method` defines an optional callable that returns a boolean as to weather or not this | ||
| message should be processed. | ||
|
|
||
| `async` determines whether handlers will do asynchronous processing. If set to True, handlers must | ||
| accept a keyword argument called `finisher` that will be a callable used to signal message | ||
| completion (with a boolean argument indicating success). | ||
|
|
||
| The library handles backoff as well as maintaining a sufficient RDY count based on the # of | ||
| producers and your configured `max_in_flight`. | ||
|
|
||
| Here is an example that demonstrates synchronous message processing: | ||
|
|
||
| ```python | ||
| import nsq | ||
|
|
||
| def task1(message): | ||
| print message | ||
| return True | ||
|
|
||
| def task2(message): | ||
| print message | ||
| return True | ||
|
|
||
| all_tasks = {"task1": task1, "task2": task2} | ||
| r = nsq.Reader(all_tasks, lookupd_http_addresses=['127.0.0.1:4161'], | ||
| topic="nsq_reader", channel="asdf") | ||
| nsq.run() | ||
| ``` | ||
|
|
||
| And async: | ||
|
|
||
| ```python | ||
| """ | ||
| This is a simple example of async processing with nsq.Reader. | ||
| It will print "deferring processing" twice, and then print | ||
| the last 3 messages that it received. | ||
| Note in particular that we pass the `async=True` argument to Reader(), | ||
| and also that we cache a different finisher callable with | ||
| each message, to be called when we have successfully finished | ||
| processing it. | ||
| """ | ||
| import nsq | ||
|
|
||
| buf = [] | ||
|
|
||
| def process_message(message, finisher): | ||
| global buf | ||
| # cache both the message and the finisher callable for later processing | ||
| buf.append((message, finisher)) | ||
| if len(buf) >= 3: | ||
| print '****' | ||
| for msg, finish_fxn in buf: | ||
| print msg | ||
| finish_fxn(True) # use finish_fxn to tell NSQ of success | ||
| print '****' | ||
| buf = [] | ||
| else: | ||
| print 'deferring processing' | ||
|
|
||
| all_tasks = {"task1": process_message} | ||
| r = nsq.Reader(all_tasks, lookupd_http_addresses=['127.0.0.1:4161'], | ||
| topic="nsq_reader", channel="async", async=True) | ||
| nsq.run() | ||
| ``` | ||
|
|
||
| [tornado]: https://github.com/facebook/tornado |