Skip to content

DM-48713: Write to the APDB in chunks to avoid overwhelming the database #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

DM-48713: Write to the APDB in chunks to avoid overwhelming the database #267

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion python/lsst/ap/association/diaPipe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,16 @@ class DiaPipelineConfig(pipeBase.PipelineTaskConfig,
doc="Pad the image by this many pixels before removing off-image "
"diaObjects for association.",
)
maximumTableLength = pexConfig.RangeField(
dtype=int,
default=500,
min=0,
max=65535,
doc="Maximum length of tables allowed to be written in one operation"
" to the APDB. Set to 0 to disable."
"The maximum is the hard limit on the number of records in the"
" Cassandra APDB that can be updated in one transaction",
)
idGenerator = DetectorVisitIdGeneratorConfig.make_field()

def setDefaults(self):
Expand Down Expand Up @@ -786,7 +796,8 @@ def writeToApdb(self, updatedDiaObjects, associatedDiaSources, diaForcedSources)
DateTime.now().toAstropy(),
diaObjectStore,
diaSourceStore,
diaForcedSourceStore)
diaForcedSourceStore,
maximum_table_length=self.config.maximumTableLength)
self.log.info("APDB updated.")

def testDataFrameIndex(self, df):
Expand Down