Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug fix and ui change #10

Open
wants to merge 33 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
4b20b84
Merge pull request #1 from chickling/master
Hulva Aug 23, 2017
559c387
update: 1. Kafka now store offsets info in broker default; 2. Elastic…
Hulva Sep 27, 2017
81917ea
google resource localization
Hulva Sep 29, 2017
efc6d8e
regular commit
Hulva Sep 29, 2017
1f8a305
merge from origin/master
Hulva Sep 29, 2017
c60fcad
dealing with warning in test
Hulva Oct 9, 2017
2fae1b5
thinking...
Hulva Oct 9, 2017
72d8ab7
logging thing
Hulva Oct 13, 2017
d1b9793
Create bootstrap.yml
Hulva Oct 14, 2017
01dcbce
Delete bootstrap.yml
Hulva Oct 14, 2017
a94cc74
mail alert bug fix
Hulva Dec 23, 2017
d6f0490
bug fix: mail alert
Hulva Dec 23, 2017
32896a3
1.重构Consumers Offsets 页面
TrumanDu Mar 14, 2018
e638b45
修改按group 查看offsets信息功能
Mar 14, 2018
ed2f45c
fix alert bug
Mar 16, 2018
0700c7b
Merge pull request #3 from TrumanDu/master
Hulva Apr 21, 2018
62e99da
update
TrumanDu Apr 21, 2018
caa9ab8
Merge pull request #4 from TrumanDu/master
Hulva Apr 21, 2018
78f9dfa
add OffsetFetchTest
Hulva Apr 21, 2018
b33105c
Merge branch 'master' of https://github.com/Hulva/kmanager.git
Hulva Apr 21, 2018
69f4cb9
regular commit
Hulva Apr 21, 2018
4ff8eba
Add OffsetFetchTest: KafkaStreams
Hulva Apr 21, 2018
2cd26e4
add kafka consumer demo; OffsetFetchTest
Hulva Apr 22, 2018
3f25217
优化从broker获取logsize以及offset的逻辑
Hulva Apr 23, 2018
3f14051
优化获取logEndOffset的代码;优化ElasticSearch处理:添加index template name 配置...
Hulva Apr 24, 2018
81047a4
delete unnecessary ZkConsumerGroupService
Hulva Apr 24, 2018
57d60a1
regular commit
Hulva Apr 25, 2018
669c9dc
fix: 没有消费者的消费信息没有被计入
Hulva May 4, 2018
81fdc24
cached JMXConnector
Hulva May 4, 2018
5dc4e1b
regular commit
Hulva May 5, 2018
f23217c
Merge branch 'master' of https://github.com/Hulva/kmanager
Hulva May 5, 2018
5155a40
fix: 当topic的消费组没有消费者的时候,topic页面不显示消费组(commitToBroker)
Hulva May 9, 2018
c1560d2
1.重新组织pom文件,移除重复依赖;
Hulva Nov 24, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,4 @@ tasks/
system.json
metrics.json
objectNames.json
.sts4-cache/
69 changes: 69 additions & 0 deletions config/jmx-mapping.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
{
"properties": {
"date": {
"type": "date"
},
"oneMinuteRate": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"fifteenMinuteRate": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"fiveMinuteRate": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"meanRate": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"metric": {
"fielddata": true,
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"count": {
"type": "long"
},
"broker": {
"fielddata": true,
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"timestamp": {
"type": "long"
}
}
}
57 changes: 57 additions & 0 deletions config/offset-mapping.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"properties": {
"date": {
"type": "date"
},
"owner": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"lag": {
"type": "long"
},
"partition": {
"type": "long"
},
"offset": {
"type": "long"
},
"logSize": {
"type": "long"
},
"modified": {
"type": "long"
},
"topic": {
"fielddata": true,
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"creation": {
"type": "long"
},
"group": {
"fielddata": true,
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"timestamp": {
"type": "long"
}
}
}
12 changes: 12 additions & 0 deletions config/template.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"template": "chicklingkmanager-*",
"settings": {
"index": {
"number_of_shards": "5",
"number_of_replicas": "1"
}
},
"aliases": {

}
}
Binary file added doc/temp/memo1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
90 changes: 45 additions & 45 deletions doc/test.java
Original file line number Diff line number Diff line change
@@ -1,45 +1,45 @@
import java.util.Date;
import java.util.Random;

public class test {
public static void main(String[] args) {
//System.out.println(getOffset("botdetection_tomysql3", "EC_bot_detection"));

System.out.println(Long.parseLong("1502830800000"));
String clusterName = "my cluster - e11";
String output = clusterName.replaceAll("\\W", "");
System.out.println(output);
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
}

public static String getOffset(String group, String topic) {
String query = "{\"size\": 0,\"aggs\": {\"aggs\": {\"date_histogram\": {\"field\": \"timestamp\","
+ "\"interval\": \"30m\""
+ ",\"time_zone\": \"America/Los_Angeles\"},\"aggs\": {\"group\": {\"terms\": {\"field\": \"group\",\"order\": {\"_term\": \"desc\"}}"
+ ",\"aggs\": {\"topic\": {\"terms\": {\"field\": \"topic\",\"order\": {\"_term\": \"desc\"}}"
+ ",\"aggs\": {\"data\": {\"terms\": {\"field\": \"partition\",\"order\": {\"_term\": \"desc\"}}"
+ ",\"aggs\": {\"offset\": {\"max\": {\"field\": \"offset\"}}, \"lag\": {\"max\": {\"field\": \"lag\"}}}}"
+ ",\"offset\":{\"sum_bucket\":{\"buckets_path\": \"data>offset\"}}"
+ ",\"lag\":{\"sum_bucket\":{\"buckets_path\": \"data>lag\"}}}}}}}}}"
+ ",\"query\": {\"filtered\": {\"query\": {\"query_string\": {\"analyze_wildcard\": true,\"query\": \"*\"}}"
+ ",\"filter\": {\"bool\": {\"must\": ["
+ "{\"query\": {\"match\": {\"_type\": {\"query\": \"kafkaoffset-e3\",\"type\": \"phrase\"}}}}, ";
if (group != null && group.length() > 0) {
query += "{\"query\": {\"match\": {\"group\": {\"query\": \"" + group + "\",\"type\": \"phrase\"}}}},";
}
if (topic != null && topic.length() > 0) {
query += "{\"query\": {\"match\": {\"topic\": {\"query\": \"" + topic + "\",\"type\": \"phrase\"}}}},";
}
query += "{\"range\": {\"timestamp\": {" + "\"gte\":" + (new Date().getTime() - 24 * 60 * 60000) + ",\"lte\": "
+ new Date().getTime() + ",\"format\": \"epoch_millis\"}}}" + "]}}}}}";
return query;
}
}
import java.util.Date;
import java.util.Random;
public class test {
public static void main(String[] args) {
//System.out.println(getOffset("botdetection_tomysql3", "EC_bot_detection"));
System.out.println(Long.parseLong("1502830800000"));
String clusterName = "my cluster - e11";
String output = clusterName.replaceAll("\\W", "");
System.out.println(output);
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
System.out.println(new Random().nextInt(10));
}
public static String getOffset(String group, String topic) {
String query = "{\"size\": 0,\"aggs\": {\"aggs\": {\"date_histogram\": {\"field\": \"timestamp\","
+ "\"interval\": \"30m\""
+ ",\"time_zone\": \"America/Los_Angeles\"},\"aggs\": {\"group\": {\"terms\": {\"field\": \"group\",\"order\": {\"_term\": \"desc\"}}"
+ ",\"aggs\": {\"topic\": {\"terms\": {\"field\": \"topic\",\"order\": {\"_term\": \"desc\"}}"
+ ",\"aggs\": {\"data\": {\"terms\": {\"field\": \"partition\",\"order\": {\"_term\": \"desc\"}}"
+ ",\"aggs\": {\"offset\": {\"max\": {\"field\": \"offset\"}}, \"lag\": {\"max\": {\"field\": \"lag\"}}}}"
+ ",\"offset\":{\"sum_bucket\":{\"buckets_path\": \"data>offset\"}}"
+ ",\"lag\":{\"sum_bucket\":{\"buckets_path\": \"data>lag\"}}}}}}}}}"
+ ",\"query\": {\"filtered\": {\"query\": {\"query_string\": {\"analyze_wildcard\": true,\"query\": \"*\"}}"
+ ",\"filter\": {\"bool\": {\"must\": ["
+ "{\"query\": {\"match\": {\"_type\": {\"query\": \"kafkaoffset-e3\",\"type\": \"phrase\"}}}}, ";
if (group != null && group.length() > 0) {
query += "{\"query\": {\"match\": {\"group\": {\"query\": \"" + group + "\",\"type\": \"phrase\"}}}},";
}
if (topic != null && topic.length() > 0) {
query += "{\"query\": {\"match\": {\"topic\": {\"query\": \"" + topic + "\",\"type\": \"phrase\"}}}},";
}
query += "{\"range\": {\"timestamp\": {" + "\"gte\":" + (System.currentTimeMillis() - 24 * 60 * 60000) + ",\"lte\": "
+ System.currentTimeMillis() + ",\"format\": \"epoch_millis\"}}}" + "]}}}}}";
return query;
}
}
Loading