Ingest CATWISE #629
Ingest CATWISE #629
Conversation
| try: | ||
| filtered_results = results[(results["ab_flags"] == '00') & (results["cc_flags"] == '0000')][0] | ||
| writer.writerow([ | ||
| filtered_results[0], |
There was a problem hiding this comment.
include the db source name in the CSV file. I think it's source["source"]
| source_num+=1 | ||
| print(source_num) | ||
| logger.warning("no source match found") |
There was a problem hiding this comment.
also make a CSV file of the sources with no match.
There was a problem hiding this comment.
also, add counters. Count how many sources have multiple matches, 1 match, and no match.
| filtered_results[1], | ||
| filtered_results[2], | ||
| filtered_results[3], | ||
| filtered_results[4], | ||
| filtered_results[7], | ||
| filtered_results[8], | ||
| filtered_results[9], | ||
| filtered_results[10], | ||
| filtered_results[11], | ||
| filtered_results[12] |
There was a problem hiding this comment.
I'm sure there's a better way to convert this list into a comma delimited string.
| results = Irsa.query_region(coordinates=coord, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec") | ||
|
|
||
| try: | ||
| filtered_results = results[(results["ab_flags"] == '00') & (results["cc_flags"] == '0000')][0] |
There was a problem hiding this comment.
there is no particular motivation for picking the first one. If there are multiple matches which pass these flags, you should keep them. Might as well save them to the CSV file. I would save all matches that pass these filters. If there are sources with multiple matches, save them to a DIFFERENT CSV file and we can look at them individually.
| sources_table = select(db.Sources) | ||
| with db.engine.connect() as conn: | ||
| sources = conn.execute(sources_table).mappings().all() | ||
|
|
There was a problem hiding this comment.
| coord_vector = SkyCoord(ra = sources["ra"], dec = sources["dec"], unit = "deg", frame = "icrs") | |
| results = Irsa.query_region(coordinates=coord_vector, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec") |
| for source in sources: | ||
| #create skycoord object because one of the parameters for query region for position | ||
| coord = SkyCoord(ra = source["ra"], dec = source["dec"], unit = "deg", frame = "icrs") | ||
|
|
||
| # generates a list of objects from the catwise2020 catalogs that are within this radius of a certain position/coordinate | ||
| results = Irsa.query_region(coordinates=coord, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec") |
There was a problem hiding this comment.
Loop over results instead of sources.
Short description: Include what type of data being ingested and appropriate references.
Link to relevant issue: Closes #614
For data ingests: