Skip to content

DM-48279: APDB simulation #263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 26 commits into
base: main
Choose a base branch
from
Draft

DM-48279: APDB simulation #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
90ccf93
Make testDataFrameIndex a staticmethod
isullivan Jan 8, 2025
e00d198
Make method static in diaPipe
isullivan Jan 4, 2025
052de15
Initial working version of an APDB simulator
isullivan Jan 2, 2025
0885384
Add final table conversions
isullivan Jan 8, 2025
9830a47
Fix angle from area calculation
isullivan Jan 9, 2025
921a1ef
Fix reported number of real sources
isullivan Jan 9, 2025
e6a3e5e
Write testApdb outputs
isullivan Jan 9, 2025
44d5ca2
Add apdbTest_marker output
isullivan Jan 9, 2025
9bcb1b9
Add option to limit the number of fake sources by max database capacity
isullivan Jan 9, 2025
8315989
Write to the APDB in chunks for large tables
isullivan Jan 10, 2025
17a3561
Enlarge the region to load diaObjects from
isullivan Jan 13, 2025
8c9b8b4
Restore forced source table formatting
isullivan Jan 13, 2025
04c5371
Skip writing forced source table if empty
isullivan Jan 13, 2025
000157a
ForcedSource 'band' must be a string
isullivan Jan 13, 2025
098417e
band needs to be a single character for the APDB schema
isullivan Jan 14, 2025
7befd15
Only update nDiaSources once
isullivan Jan 14, 2025
d0822cd
Only write new and updated diaObjects to APDB
isullivan Jan 14, 2025
b2d9cf5
Add timing log messages
isullivan Jan 14, 2025
fac465e
Temporarily delete failing tests not relevant to the APDB simulation
isullivan Jan 24, 2025
5376061
Change default false positive rate to 2-1
isullivan Jan 26, 2025
8d3656d
Fill diaSource table
isullivan Jan 26, 2025
19bd887
Fill diaObject table
isullivan Jan 26, 2025
1e77c4b
Ensure new random-filled dataframes have the right schema
isullivan Jan 26, 2025
79015f3
Add correct format of band to DiaSource table
isullivan Jan 26, 2025
2d74cfc
Fix diaObject format for nearbyLowzGal column
isullivan Jan 26, 2025
c97b6b0
Add option to raise early if there will likely be duplicate diaSources
isullivan Jan 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions python/lsst/ap/association/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,4 @@
from .version import *
from .mpSkyEphemerisQuery import *
from .ssSingleFrameAssociation import *
from .testApdb import *
6 changes: 4 additions & 2 deletions python/lsst/ap/association/diaPipe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -789,7 +789,8 @@ def writeToApdb(self, updatedDiaObjects, associatedDiaSources, diaForcedSources)
diaForcedSourceStore)
self.log.info("APDB updated.")

def testDataFrameIndex(self, df):
@staticmethod
def testDataFrameIndex(df):
"""Test the sorted DataFrame index for duplicates.

Wrapped as a separate function to allow for mocking of the this task
Expand Down Expand Up @@ -922,7 +923,8 @@ def mergeCatalogs(self, originalCatalog, newCatalog, catalogName):
mergedCatalog = pd.concat([originalCatalog], sort=True)
return mergedCatalog.loc[:, originalCatalog.columns]

def updateObjectTable(self, diaObjects, diaSources):
@staticmethod
def updateObjectTable(diaObjects, diaSources):
"""Update the diaObject table with the new diaSource records.

Parameters
Expand Down
Loading