Track the actor that triggers the minion task #14829
Conversation
shounakmk219
changed the title
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #14829 +/- ##
============================================
- Coverage 61.75% 56.24% -5.52%
- Complexity 207 796 +589
============================================
Files 2436 2120 -316
Lines 133233 114478 -18755
Branches 20636 18410 -2226
============================================
- Hits 82274 64384 -17890
+ Misses 44911 44862 -49
+ Partials 6048 5232 -816
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
shounakmk219
requested review from
tibrewalpratik17,
swaminathanmanish and
KKcorps
| * It might be called from the non-leader controller. | ||
| * Returns a map from the task type to the {@link TaskSchedulingInfo} of tasks scheduled. | ||
| */ | ||
| public synchronized Map<String, TaskSchedulingInfo> scheduleAllTasksForAllTables(@Nullable String minionInstanceTag) { |
There was a problem hiding this comment.
I would suggest rather than removing them add a @deprecated annotation on top, along with the method that they should be using. We can remove this in next release
There was a problem hiding this comment.
ok, will revert removed methods
| @@ -114,6 +114,10 @@ public class PinotTaskManager extends ControllerPeriodicTask<Void> { | |||
|
|
|||
| private final TaskManagerStatusCache<TaskGeneratorMostRecentRunInfo> _taskManagerStatusCache; | |||
|
|
|||
| public enum Triggers { | |||
There was a problem hiding this comment.
Better to move this outside
| Map<String, List<TableConfig>> enabledTableConfigMap = new HashMap<>(); | ||
| for (String tableNameWithType : tableNamesWithType) { | ||
| Map<String, Set<String>> tableToTasksMap = context.getTableToTaskNamesMap(); | ||
| if (context.getTableToTaskNamesMap().isEmpty()) { |
There was a problem hiding this comment.
can simply use tableToTasksMap here
| if (context.getTableToTaskNamesMap().isEmpty()) { | ||
| _pinotHelixResourceManager.getAllTables().forEach(table -> tableToTasksMap.put(table, null)); | ||
| } | ||
| for (Map.Entry<String, Set<String>> entry : context.getTableToTaskNamesMap().entrySet()) { |
| /** | ||
| * Helper method to schedule tasks (all task types) for the given tables that have the tasks enabled. | ||
| * Returns a map from the task type to the {@link TaskSchedulingInfo} of the tasks scheduled. | ||
| */ | ||
| protected synchronized Map<String, TaskSchedulingInfo> scheduleTasks(List<String> tableNamesWithType, | ||
| boolean isLeader, @Nullable String minionInstanceTag) { | ||
| public synchronized Map<String, TaskSchedulingInfo> scheduleTasks(TaskSchedulingContext context) { |
There was a problem hiding this comment.
do we need this to be public? let's keep it as protected
There was a problem hiding this comment.
This method is being called everywhere hence kept it public
| for (String tableNameWithType : tableNamesWithType) { | ||
| Map<String, Set<String>> tableToTasksMap = context.getTableToTaskNamesMap(); | ||
| if (context.getTableToTaskNamesMap().isEmpty()) { | ||
| _pinotHelixResourceManager.getAllTables().forEach(table -> tableToTasksMap.put(table, null)); |
There was a problem hiding this comment.
Adding an entry for all tables is probably not optimal, see if there is a way around this
There was a problem hiding this comment.
good point, let me refactor the code to avoid adding entries
|
Add missing license headers |
shounakmk219
force-pushed
the
task-trigger-tracking
branch
from
79a8879 to
7040e3b
Compare
Description
Minion task can be triggered by 3 ways as of now
To identify which actor triggered a particular task run, this PR injects the information in task configs so that it gets tracked in zk for reference.
The
triggeredByinfo is also made available onTaskDebugInfoandSubtaskDebugInfo.Refactor on
PinotTaskManagerThere were too many derivations of schedule methods in PinotTaskManager based on what combination of tables and task types are supposed to be scheduled. This made the changes on schedule inputs hard to manage.
This PR adds a wrapper class
TaskSchedulingContextthat will be passed to only onescheduleTasks()method to achieve all the derivations.