Skip to content

Commit b3967ae

Browse files
jdhoekjdhoek
committed
Implement etcd backend
1 parent d70b558 commit b3967ae

19 files changed

+1766
-12
lines changed

‎README.md‎

Lines changed: 120 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ limited number of 'clusters' exists). The generator-ID can also be assigned manu
7575
usually it is desirable for processes to be able to exclusively claim a generator-ID
7676
automatically.
7777

78-
This library facilitates this using [ZooKeeper](http://zookeeper.apache.org/).
78+
This library facilitates this using [Etcd](https://etcd.io/) or [Apache ZooKeeper](http://zookeeper.apache.org/).
7979
Generators can stake a claim on a generator-ID for a short period of time (usually ten
8080
minutes), and repeat this whenever IDs are generated.
8181

@@ -122,25 +122,135 @@ For local usage the `uniqueid-core` module can be used:
122122
</dependency>
123123
```
124124

125-
### Distributed usage with a ZooKeeper quorum
125+
### Distributed usage
126126

127127
If you need to generate unique IDs within a distributed environment, automatic coordination of
128128
the generator-ID is also a possibility. The acquisition of a generator-ID can be handled by a
129-
`SynchronizedGeneratorIdentity` instance, which uses
130-
[Apache ZooKeeper](http://zookeeper.apache.org/) to claim its generator-ID — for a short while,
131-
or as long as it maintains a connection to the ZooKeeper quorum.
129+
`SynchronizedGeneratorIdentity` instance, which uses [Etcd](https://etcd.io/) or
130+
[Apache ZooKeeper](http://zookeeper.apache.org/) to claim its generator-ID
132131

133-
For this functionality the `uniqueid-zookeeper` module is used:
132+
#### With an Etcd cluster
133+
134+
For this Etcd the `uniqueid-etcd` module is used:
134135

135136
```xml
136137
<dependency>
137138
<groupId>org.lable.oss.uniqueid</groupId>
138-
<artifactId>uniqueid-core</artifactId>
139+
<artifactId>uniqueid-etcd</artifactId>
140+
<version>${uniqueid.version}</version>
141+
</dependency>
142+
```
143+
144+
##### Preparing the Etcd cluster
145+
146+
To use this method of generator-ID acquisition, a namespace on the Etcd cluster must
147+
be chosen to hold the resource pool used by `SynchronizedGeneratorIdentity`.
148+
149+
For example, if you choose `unique-id-generator/` as the namespace, these keys can be
150+
automatically created when the library is used:
151+
152+
```
153+
unique-id-generator/cluster-id
154+
unique-id-generator/pool/0
155+
unique-id-generator/pool/1
156+
…
157+
unique-id-generator/pool/255
158+
```
159+
160+
Note that if you do not create the `cluster-id` key yourself (recommended), the default
161+
value of `0` will be used. To use a different cluster ID, set the content of this key to
162+
one of the 16 permissible values (i.e., `0..15`).
163+
164+
If you have access to the `etcdctl` command line utility you can set the
165+
cluster-ID like so:
166+
167+
```
168+
etcdctl --endpoints=… put unique-id-generator/cluster-id 1
169+
```
170+
171+
##### Using the generator
172+
173+
To use an `IDGenerator` with a negotiated generator-Id, create a new instance like this:
174+
175+
```java
176+
// Change the values of etcdCluster and namespace as needed:
177+
final List<String> etcdCluster = Arrays.asList("https://etcd1:2379","https://etcd2:2379","https://etcd3:2379");
178+
final String namespace = "unique-id-generator/";
179+
final ByteSequence ns = ByteSequence.from(namespace, StandardCharsets.UTF_8);
180+
final Client client = Client.builder()
181+
.endpoints(etcdCluster)
182+
.namespace(ns)
183+
.build();
184+
IDGenerator generator = SynchronizedUniqueIDGeneratorFactory.generatorFor(client, Mode.SPREAD);
185+
// ...
186+
byte[] id = generator.generate()
187+
// ...
188+
```
189+
190+
If you expect that you will be using dozens of IDs in a single process, it is more
191+
efficient to generate IDs in batches:
192+
193+
```java
194+
Deque<byte[]> ids = generator.batch(500);
195+
// ...
196+
byte[] id = ids.pop();
197+
// etc.
198+
```
199+
200+
If you intend to generate more than a few IDs at a time, you can also wrap the generator in
201+
an `AutoRefillStack`, and simply call `generate()` on that whenever you need a new ID.
202+
It will grab IDs in batches from the wrapped `IDGenerator` instance for you. This is
203+
probably the simplest and safest way to use an `IDGenerator` in the default `SPREAD` mode.
204+
205+
```java
206+
final List<String> etcdCluster = Arrays.asList("https://etcd1:2379","https://etcd2:2379","https://etcd3:2379");
207+
final String namespace = "unique-id-generator/";
208+
final ByteSequence ns = ByteSequence.from(namespace, StandardCharsets.UTF_8);
209+
final Client client = Client.builder()
210+
.endpoints(etcdCluster)
211+
.namespace(ns)
212+
.build();
213+
IDGenerator generator = new AutoRefillStack(
214+
SynchronizedUniqueIDGeneratorFactory.generatorFor(client, Mode.SPREAD)
215+
);
216+
// ...
217+
byte[] id = generator.generate()
218+
// ...
219+
```
220+
221+
For the `TIME_SEQUENTIAL` mode the above is usually not what you want, if you intend to use
222+
the timestamp stored in the generated ID as part of your data model (the batched pre-generated
223+
IDs might have a timestamp that lies further in the past then you might want).
224+
225+
```java
226+
final List<String> etcdCluster = Arrays.asList("https://etcd1:2379","https://etcd2:2379","https://etcd3:2379");
227+
final String namespace = "unique-id-generator/";
228+
final ByteSequence ns = ByteSequence.from(namespace, StandardCharsets.UTF_8);
229+
final Client client = Client.builder()
230+
.endpoints(etcdCluster)
231+
.namespace(ns)
232+
.build();
233+
IDGenerator generator = SynchronizedUniqueIDGeneratorFactory.generatorFor(client, Mode.TIME_SEQUENTIAL);
234+
// ...
235+
byte[] id = generator.generate()
236+
// Extract the timestamp in the ID.
237+
long createdAt = IDBuilder.parseTimestamp(id);
238+
```
239+
240+
241+
#### With a ZooKeeper quorum
242+
243+
For ZooKeeper the `uniqueid-zookeeper` module is used:
244+
245+
```xml
246+
<dependency>
247+
<groupId>org.lable.oss.uniqueid</groupId>
248+
<artifactId>uniqueid-zookeeper</artifactId>
139249
<version>${uniqueid.version}</version>
140250
</dependency>
141251
```
142252

143-
#### Preparing the ZooKeeper quorum
253+
##### Preparing the ZooKeeper quorum
144254

145255
To use this method of generator-ID acquisition, a node on the ZooKeeper quorum must
146256
be chosen to hold the queue and resource pool used by `SynchronizedGeneratorIdentity`.
@@ -172,7 +282,7 @@ Or if the node already exists:
172282
set /unique-id-generator/cluster-id 1
173283
```
174284

175-
#### Using the generator
285+
##### Using the generator
176286

177287
To use an `IDGenerator` with a negotiated generator-Id, create a new instance like this:
178288

@@ -219,7 +329,7 @@ IDs might have a timestamp that lies further in the past then you might want).
219329
```java
220330
final String zookeeperQuorum = "zookeeper1,zookeeper2,zookeeper3";
221331
final String znode = "/unique-id-generator";
222-
IDGenerator SynchronizedUniqueIDGeneratorFactory.generatorFor(zookeeperQuorum, znode, Mode.TIME_SEQUENTIAL);
332+
IDGenerator generator = SynchronizedUniqueIDGeneratorFactory.generatorFor(zookeeperQuorum, znode, Mode.TIME_SEQUENTIAL);
223333
// ...
224334
byte[] id = generator.generate()
225335
// Extract the timestamp in the ID.

‎doc/eight-byte-id-structure.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ TTTTTTTT TTTTTTTT TTTTTTTT TTTTTTTT TTTTTTTT TTSSSSSS ...MGGGG GGGGCCCC
99
| | | || |
1010
v | | |\ |
1111
| v \ \ |
12-
Date of ID creation, meausred in milliseconds | \ \ |
12+
Date of ID creation, measured in milliseconds | \ \ |
1313
ellapsed since 1970-01-01T00:00:00.000, | Reserved \ \ |
1414
represented in reverse byte-order (in SPREAD | for future | | |
1515
mode) to guarantee an even spread of IDs. | use. | | |
@@ -59,7 +59,7 @@ resulting ID is unique. Secondly, the timestamp is a permanent record of the dat
5959
creation of the ID, which may be useful if is used as ID for a database record the moment
6060
it is created.
6161

62-
Because of the limited bitsize of the timestamp, only dates until the year2109 are supported.
62+
Because of the limited bit-size of the timestamp, only dates until the year 2109 are supported.
6363
As this library generates only 'now' timestamps, this should not be practical limitation.
6464

6565
### Sequence counter

‎pom.xml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,17 +38,20 @@
3838
<modules>
3939
<module>uniqueid-core</module>
4040
<module>uniqueid-zookeeper</module>
41+
<module>uniqueid-etcd</module>
4142
</modules>
4243

4344
<properties>
4445
<java.language.level>1.8</java.language.level>
4546

4647
<dynamicconfig.version>2.9-beta4</dynamicconfig.version>
4748
<zookeeper.version>3.4.6</zookeeper.version>
49+
<jetcd-version>0.4.1</jetcd-version>
4850
<slf4j.version>1.7.21</slf4j.version>
4951

5052
<!-- For testing only. -->
5153
<log4j.version>2.7</log4j.version>
54+
<hamcrest.optional>1.0</hamcrest.optional>
5255
</properties>
5356

5457
<licenses>

‎uniqueid-core/src/main/java/org/lable/oss/uniqueid/BaseUniqueIDGenerator.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,18 @@ public class BaseUniqueIDGenerator implements IDGenerator {
4646
* @param generatorIdentityHolder Generator identity holder.
4747
* @param mode Generator mode.
4848
*/
49+
public BaseUniqueIDGenerator(GeneratorIdentityHolder generatorIdentityHolder,
50+
Mode mode) {
51+
this(generatorIdentityHolder, null, mode);
52+
}
53+
54+
/**
55+
* Create a new UniqueIDGenerator instance.
56+
*
57+
* @param generatorIdentityHolder Generator identity holder.
58+
* @param clock System clock (optional; useful for tests).
59+
* @param mode Generator mode.
60+
*/
4961
public BaseUniqueIDGenerator(GeneratorIdentityHolder generatorIdentityHolder,
5062
Clock clock,
5163
Mode mode) {

‎uniqueid-etcd/pom.xml‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
4+
Copyright (C) 2014 Lable ([email protected])
5+
6+
Licensed under the Apache License, Version 2.0 (the "License");
7+
you may not use this file except in compliance with the License.
8+
You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing, software
13+
distributed under the License is distributed on an "AS IS" BASIS,
14+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
See the License for the specific language governing permissions and
16+
limitations under the License.
17+
18+
-->
19+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
20+
<parent>
21+
<artifactId>uniqueid</artifactId>
22+
<groupId>org.lable.oss.uniqueid</groupId>
23+
<version>2.6-beta5-SNAPSHOT</version>
24+
</parent>
25+
<modelVersion>4.0.0</modelVersion>
26+
27+
<artifactId>uniqueid-etcd</artifactId>
28+
<name>UniqueID :: Etcd</name>
29+
30+
<dependencies>
31+
<dependency>
32+
<artifactId>uniqueid-core</artifactId>
33+
<groupId>org.lable.oss.uniqueid</groupId>
34+
</dependency>
35+
<dependency>
36+
<groupId>io.etcd</groupId>
37+
<artifactId>jetcd-core</artifactId>
38+
<version>${jetcd-version}</version>
39+
</dependency>
40+
41+
<!-- Testing dependencies. -->
42+
<dependency>
43+
<artifactId>uniqueid-core</artifactId>
44+
<groupId>org.lable.oss.uniqueid</groupId>
45+
<classifier>tests</classifier>
46+
<scope>test</scope>
47+
<version>${project.version}</version>
48+
</dependency>
49+
<dependency>
50+
<groupId>io.etcd</groupId>
51+
<artifactId>jetcd-launcher</artifactId>
52+
<version>${jetcd-version}</version>
53+
<scope>test</scope>
54+
</dependency>
55+
<dependency>
56+
<groupId>com.github.npathai</groupId>
57+
<artifactId>hamcrest-optional</artifactId>
58+
<version>${hamcrest.optional}</version>
59+
<scope>test</scope>
60+
</dependency>
61+
</dependencies>
62+
63+
<build>
64+
<plugins>
65+
<plugin>
66+
<artifactId>maven-failsafe-plugin</artifactId>
67+
</plugin>
68+
69+
<plugin>
70+
<groupId>org.jacoco</groupId>
71+
<artifactId>jacoco-maven-plugin</artifactId>
72+
</plugin>
73+
74+
<plugin>
75+
<groupId>org.apache.maven.plugins</groupId>
76+
<artifactId>maven-shade-plugin</artifactId>
77+
<version>3.2.1</version>
78+
<executions>
79+
<execution>
80+
<phase>package</phase>
81+
<goals>
82+
<goal>shade</goal>
83+
</goals>
84+
<configuration>
85+
<shadedArtifactAttached>true</shadedArtifactAttached>
86+
<shadedClassifierName>shaded</shadedClassifierName>
87+
<relocations>
88+
<relocation>
89+
<pattern>com.google.</pattern>
90+
<shadedPattern>com.shaded.google.</shadedPattern>
91+
</relocation>
92+
<relocation>
93+
<pattern>io.grpc.</pattern>
94+
<shadedPattern>io.shaded.grpc.</shadedPattern>
95+
</relocation>
96+
<relocation>
97+
<pattern>io.netty.</pattern>
98+
<shadedPattern>io.shaded.netty.</shadedPattern>
99+
</relocation>
100+
</relocations>
101+
<transformers>
102+
<!-- GRPC uses service-loaders.
103+
If these don't get renamed to match the new package names, GRPC breaks. -->
104+
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
105+
</transformers>
106+
</configuration>
107+
</execution>
108+
</executions>
109+
</plugin>
110+
</plugins>
111+
</build>
112+
</project>

0 commit comments

Comments
 (0)