Skip to content

Commit 55acb7b

Browse files
shenmingfushenmingfu
authored
Merge branch 'main' into tickets/SP-2381
2 parents 53cc3b1 + 8a78ec1 commit 55acb7b

File tree

39 files changed

+4323
-229
lines changed

39 files changed

+4323
-229
lines changed

‎DP1/100_How_to_Use_RSP_Tools/102_Catalog_access/102_1_Get_started_with_TAP.ipynb‎

Lines changed: 244 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
"Data Release: <a href=\"https://dp1.lsst.io\">Data Preview 1</a> <br>\n",
2525
"Container Size: large <br>\n",
2626
"LSST Science Pipelines version: r29.2.0 <br>\n",
27-
"Last verified to run: 2025-09-02 <br>\n",
27+
"Last verified to run: 2025-09-17 <br>\n",
2828
"Repository: <a href=\"https://github.com/lsst/tutorial-notebooks\">github.com/lsst/tutorial-notebooks</a> <br>"
2929
]
3030
},
@@ -420,11 +420,16 @@
420420
"print('Job phase is', job.phase)\n",
421421
"```\n",
422422
"\n",
423-
"Finally, display the error message if the query was not successful, or retrieve the results if the query was completed.\n",
423+
"Display the error message if the query was not successful.\n",
424424
"\n",
425425
"```\n",
426426
"if job.phase == 'ERROR':\n",
427427
" job.raise_if_error()\n",
428+
"```\n",
429+
"\n",
430+
"If the query was completed, retrieve (fetch) the results.\n",
431+
"\n",
432+
"```\n",
428433
"assert job.phase == 'COMPLETED'\n",
429434
"results = job.fetch_result()\n",
430435
"```"
@@ -509,12 +514,40 @@
509514
"job.wait(phases=['COMPLETED', 'ERROR'])\n",
510515
"print('Job phase is', job.phase)\n",
511516
"if job.phase == 'ERROR':\n",
512-
" job.raise_if_error()\n",
517+
" job.raise_if_error()"
518+
]
519+
},
520+
{
521+
"cell_type": "code",
522+
"execution_count": null,
523+
"id": "c8d8433f-e4d2-4d4b-9ff9-c95f4e9f6be7",
524+
"metadata": {},
525+
"outputs": [],
526+
"source": [
513527
"assert job.phase == 'COMPLETED'\n",
514528
"results = job.fetch_result()\n",
515529
"print(len(results))"
516530
]
517531
},
532+
{
533+
"cell_type": "markdown",
534+
"id": "435e67e7-36ae-47ff-ab14-bd0f02d50cac",
535+
"metadata": {},
536+
"source": [
537+
"Clean up."
538+
]
539+
},
540+
{
541+
"cell_type": "code",
542+
"execution_count": null,
543+
"id": "f55a5b05-86b1-4d1d-9140-5b2748048d08",
544+
"metadata": {},
545+
"outputs": [],
546+
"source": [
547+
"job.delete()\n",
548+
"del query, results"
549+
]
550+
},
518551
{
519552
"cell_type": "markdown",
520553
"id": "1075c36f-5784-4a13-a08b-37bad4192993",
@@ -525,14 +558,89 @@
525558
"For a polygon search, replace `CIRCLE` with `POLYGON(ra1, dec1, ra2, dec2, ra3, dec3, ...)` where each `ra1, dec` is a vertex of the polygon."
526559
]
527560
},
561+
{
562+
"cell_type": "markdown",
563+
"id": "8383b976-7ef1-404f-a00d-5a0b02153d0f",
564+
"metadata": {},
565+
"source": [
566+
"### 4.3. Use column constraints\n",
567+
"\n",
568+
"A simple column constraint is to put an upper or lower limit on the values in a column, such as a limiting magnitude.\n",
569+
"\n",
570+
"Use the same cone search as above, but extend the radius to $0.3$ degrees and only return `Objects` with a $g$-band `cModel` magnitude brighter than 20 mag."
571+
]
572+
},
573+
{
574+
"cell_type": "code",
575+
"execution_count": null,
576+
"id": "64578296-3410-48e1-a331-8290757251ef",
577+
"metadata": {},
578+
"outputs": [],
579+
"source": [
580+
"query = \"\"\"SELECT coord_ra, coord_dec, g_cModelMag\n",
581+
" FROM dp1.Object\n",
582+
" WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec),\n",
583+
" CIRCLE('ICRS', 53, -28, 0.3)) = 1\n",
584+
" AND g_cModelMag < 20\"\"\"\n",
585+
"print(query)"
586+
]
587+
},
588+
{
589+
"cell_type": "code",
590+
"execution_count": null,
591+
"id": "2713c73b-0751-4dce-a85e-a066f83fd65a",
592+
"metadata": {},
593+
"outputs": [],
594+
"source": [
595+
"job = service.submit_job(query)\n",
596+
"job.run()\n",
597+
"job.wait(phases=['COMPLETED', 'ERROR'])\n",
598+
"print('Job phase is', job.phase)\n",
599+
"if job.phase == 'ERROR':\n",
600+
" job.raise_if_error()"
601+
]
602+
},
603+
{
604+
"cell_type": "code",
605+
"execution_count": null,
606+
"id": "06ca9cf4-b1bb-445b-8600-6d59bd533b70",
607+
"metadata": {},
608+
"outputs": [],
609+
"source": [
610+
"assert job.phase == 'COMPLETED'\n",
611+
"results = job.fetch_result()\n",
612+
"print(len(results))"
613+
]
614+
},
615+
{
616+
"cell_type": "markdown",
617+
"id": "254d6449-7479-4942-9831-0d88ba7483ed",
618+
"metadata": {},
619+
"source": [
620+
"Clean up."
621+
]
622+
},
623+
{
624+
"cell_type": "code",
625+
"execution_count": null,
626+
"id": "07074ba6-1055-4d6a-89a3-b0fb52a1da16",
627+
"metadata": {},
628+
"outputs": [],
629+
"source": [
630+
"job.delete()\n",
631+
"del query, results"
632+
]
633+
},
528634
{
529635
"cell_type": "markdown",
530636
"id": "56bd367d-9ff8-4826-984f-81562cbfc09a",
531637
"metadata": {},
532638
"source": [
533-
"### 4.3. Sort results with ORDER BY\n",
639+
"### 4.4. Sort results with ORDER BY\n",
534640
"\n",
535-
"Add an `ORDER BY` statement to the TAP query to sort the results in ascending (ASC) or descending (DESC) order.\n",
641+
"Add an `ORDER BY` statement to the TAP query above to sort the results in ascending (ASC) or descending (DESC) order.\n",
642+
"\n",
643+
"> **Warning:** Adding `ORDER BY` statements to TAP queries requires more processing than unordered queries, and consumes more of the shared resource that is the TAP service. Consider whether sorting is necessary, or sorting after the query is returned (as in Section 4.4.1).\n",
536644
"\n",
537645
"Order the results by ascending `coord_ra` values."
538646
]
@@ -544,27 +652,119 @@
544652
"metadata": {},
545653
"outputs": [],
546654
"source": [
547-
"query = \"\"\"SELECT coord_ra, coord_dec\n",
655+
"query = \"\"\"SELECT coord_ra, coord_dec, g_cModelMag\n",
548656
" FROM dp1.Object\n",
549657
" WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec),\n",
550-
" CIRCLE('ICRS', 53, -28, 0.01)) = 1\n",
658+
" CIRCLE('ICRS', 53, -28, 0.3)) = 1\n",
659+
" AND g_cModelMag < 20\n",
551660
" ORDER BY coord_ra ASC\"\"\"\n",
552661
"print(query)"
553662
]
554663
},
664+
{
665+
"cell_type": "code",
666+
"execution_count": null,
667+
"id": "7c01c299-08ed-409a-9f72-84f8d542f529",
668+
"metadata": {},
669+
"outputs": [],
670+
"source": [
671+
"job = service.submit_job(query)\n",
672+
"job.run()\n",
673+
"job.wait(phases=['COMPLETED', 'ERROR'])\n",
674+
"print('Job phase is', job.phase)\n",
675+
"if job.phase == 'ERROR':\n",
676+
" job.raise_if_error()"
677+
]
678+
},
679+
{
680+
"cell_type": "markdown",
681+
"id": "661f83ba-c5b4-487f-886d-358baed0227a",
682+
"metadata": {},
683+
"source": [
684+
"Return the results as an `astropy` formatted table with `to_table()`, in order to use the sorting function of `astropy` tables."
685+
]
686+
},
687+
{
688+
"cell_type": "code",
689+
"execution_count": null,
690+
"id": "dad64abd-f701-4ecd-a24b-f81ecfc03a8a",
691+
"metadata": {},
692+
"outputs": [],
693+
"source": [
694+
"assert job.phase == 'COMPLETED'\n",
695+
"results = job.fetch_result().to_table()"
696+
]
697+
},
698+
{
699+
"cell_type": "markdown",
700+
"id": "4478a975-37b5-4439-a70b-62f40b7ca9e7",
701+
"metadata": {},
702+
"source": [
703+
"#### 4.4.1. Sort an astropy table\n",
704+
"\n",
705+
"Sort the `astropy` table by `g_cModelMag` instead, in descending order (i.e., use `reverse=True`; the default is `False`, for ascending order). Sort by multiple columns with, e.g., `results.sort(['column_1', 'column_2'])`."
706+
]
707+
},
708+
{
709+
"cell_type": "code",
710+
"execution_count": null,
711+
"id": "b68f368b-fab5-455a-8555-e574b9f2812c",
712+
"metadata": {},
713+
"outputs": [],
714+
"source": [
715+
"results.sort('g_cModelMag', reverse=True)"
716+
]
717+
},
718+
{
719+
"cell_type": "markdown",
720+
"id": "519bcdbe-681e-4075-bc5c-2ef63ac1c61d",
721+
"metadata": {},
722+
"source": [
723+
"Option to display the results, now sorted by $g$-band magnitude."
724+
]
725+
},
726+
{
727+
"cell_type": "code",
728+
"execution_count": null,
729+
"id": "74af010a-da4e-41ad-b725-34f49636ba69",
730+
"metadata": {},
731+
"outputs": [],
732+
"source": [
733+
"# results"
734+
]
735+
},
736+
{
737+
"cell_type": "markdown",
738+
"id": "d3358d02-e0f1-4d15-a75e-46638f45cd04",
739+
"metadata": {},
740+
"source": [
741+
"Clean up."
742+
]
743+
},
744+
{
745+
"cell_type": "code",
746+
"execution_count": null,
747+
"id": "e5a6961c-9244-4350-ab63-98c7df9b34e2",
748+
"metadata": {},
749+
"outputs": [],
750+
"source": [
751+
"job.delete()\n",
752+
"del query, results"
753+
]
754+
},
555755
{
556756
"cell_type": "markdown",
557757
"id": "eb343134-f23f-4dc1-976a-0ca45f27a50b",
558758
"metadata": {},
559759
"source": [
560-
"### 4.4. Limit rows with TOP\n",
760+
"### 4.5. Limit rows with TOP\n",
561761
"\n",
562-
"For debugging and testing queries, **the recommended way to restrict the number of rows returned is to use very small spatial regions**, not to use TOP.\n",
762+
"**Very small spatial regions (not use of TOP) is the recommended way to restrict the number of rows returned.**\n",
563763
"\n",
564-
"The TAP service first applies `WHERE` constraints, then `ORDER BY`, and then `TOP`.\n",
764+
"> **Warning:** The TAP service first applies `WHERE` constraints, then `ORDER BY`, and then `TOP`.\n",
565765
"If the query is not well constrained, i.e., if thousands or more objects meet the `WHERE`\n",
566766
"constraints, then they all must first be sorted before the top number are returned. \n",
567-
"This is a waste of time and compute resources.\n",
767+
"**This would be a waste of time and compute resources.**\n",
568768
"\n",
569769
"However, it can be useful to only retrieve a subset of the rows which meet the query constraints. To do this, add `TOP N` after `SELECT` to return only the top `N` number of rows. "
570770
]
@@ -589,7 +789,7 @@
589789
"id": "b1bbb142-a805-41ff-9982-06369b585600",
590790
"metadata": {},
591791
"source": [
592-
"Execute the query and confirm that only 10 rows are returned."
792+
"Execute the query. "
593793
]
594794
},
595795
{
@@ -604,12 +804,42 @@
604804
"job.wait(phases=['COMPLETED', 'ERROR'])\n",
605805
"print('Job phase is', job.phase)\n",
606806
"if job.phase == 'ERROR':\n",
607-
" job.raise_if_error()\n",
807+
" job.raise_if_error()"
808+
]
809+
},
810+
{
811+
"cell_type": "markdown",
812+
"id": "9e83878e-3ecf-4812-a366-73e3ee063a45",
813+
"metadata": {},
814+
"source": [
815+
"Confirm that only 10 rows are returned."
816+
]
817+
},
818+
{
819+
"cell_type": "code",
820+
"execution_count": null,
821+
"id": "639289be-59ca-4442-b8c5-e1c2a63f785a",
822+
"metadata": {},
823+
"outputs": [],
824+
"source": [
608825
"assert job.phase == 'COMPLETED'\n",
609826
"results = job.fetch_result()\n",
827+
"\n",
828+
"assert len(results) == 10\n",
610829
"print(len(results))"
611830
]
612831
},
832+
{
833+
"cell_type": "code",
834+
"execution_count": null,
835+
"id": "4dac97b2-d2fd-44c5-bb17-570851d6a9a6",
836+
"metadata": {},
837+
"outputs": [],
838+
"source": [
839+
"job.delete()\n",
840+
"del query, results"
841+
]
842+
},
613843
{
614844
"cell_type": "markdown",
615845
"id": "f912d6a9-b510-4ca7-bf73-a612d030d787",
@@ -623,7 +853,7 @@
623853
"id": "f1f5c1cb-0fc4-4a41-a736-6233b3a1087b",
624854
"metadata": {},
625855
"source": [
626-
"#### 4.4.1. Use caution with TOP and ORDER BY\n",
856+
"### 4.6. Use caution with TOP and ORDER BY\n",
627857
"\n",
628858
"Combined use of TOP and ORDER BY in ADQL queries can be dangerous: it may take an unexpectedly long\n",
629859
"time because the database is trying to first sort, and *then* extract the top N elements.\n",

0 commit comments

Comments
 (0)