Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Local Group Galaxy Database: An Open, Multi-Source, Curated Catalog\n",
    "\n",
    "This notebook will guide users on how to use the basic API we've designed.\n",
    "\n",
    "There are two ways to start the database, by providing a directory of JSON files to load or by suppling a MongoDB connection string. Here we will use the directory method. All the same method exists regardless of how you start the database, though the MongoDB use will have more extensive query language options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from galcat.core import *\n",
    "\n",
    "db = Database(directory='galcat/tests/test_data', references_file='galcat/tests/test_references.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The database is a NoSQL document store that organizes the data as a collection of JSON/dictionary objects. Each galaxy will be it's own JSON document containing all relevant parameters. For example, here is the structure for a test galaxy, Gal 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = {\n",
    "    \"name\": \"Gal 2\",\n",
    "    \"ra\": [\n",
    "        {\n",
    "            \"value\": 9.14542,\n",
    "            \"best\": 1,\n",
    "            \"reference\": \"\",\n",
    "            \"unit\": \"deg\"\n",
    "        }\n",
    "    ],\n",
    "    \"dec\": [\n",
    "        {\n",
    "            \"value\": 49.64667,\n",
    "            \"best\": 1,\n",
    "            \"reference\": \"\",\n",
    "            \"unit\": \"deg\"\n",
    "        }\n",
    "    ],\n",
    "    \"ebv\": [\n",
    "        {\n",
    "            \"value\": 0.166,\n",
    "            \"best\": 1,\n",
    "            \"reference\": \"Bellazzini_2006_1\"\n",
    "        }\n",
    "    ],\n",
    "    \"v_mag\": [\n",
    "        {\n",
    "            \"value\": 16.2,\n",
    "            \"error_upper\": 0.3,\n",
    "            \"error_lower\": 0.3,\n",
    "            \"best\": 1,\n",
    "            \"unit\": \"mag\",\n",
    "            \"reference\": \"\"\n",
    "        }\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data can be queried from the database using MongoDB's query language, which also uses a JSON/dictionary like structure to parse queries. Here are some example queries you can make with the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'name': 'Gal 2', 'ra': array([{'value': 10.4, 'best': 1, 'reference': '', 'unit': 'deg'}],\n",
      "      dtype=object), 'dec': array([{'value': -32.4, 'best': 1, 'reference': '', 'unit': 'deg'}],\n",
      "      dtype=object), 'ebv': array([{'value': 0.2, 'best': 1, 'reference': 'Bellazzini_2006_1'}],\n",
      "      dtype=object), 'v_mag': array([{'value': 20.2, 'error_upper': 0.5, 'error_lower': 0.5, 'best': 1, 'unit': 'mag', 'reference': ''}],\n",
      "      dtype=object)}]\n"
     ]
    }
   ],
   "source": [
    "# Query on a single field, such as a name\n",
    "doc = db.query_db({'name': 'Gal 2'})\n",
    "print(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<i>QTable masked=True length=2</i>\n",
       "<table id=\"table4876225112\" class=\"table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>dec</th><th>ebv</th><th>half-light_radius</th><th>name</th><th>ra</th><th>radial_velocity</th><th>v_mag</th></tr></thead>\n",
       "<thead><tr><th>object</th><th>float64</th><th>object</th><th>str5</th><th>object</th><th>float64</th><th>object</th></tr></thead>\n",
       "<tr><td>-32.4 deg</td><td>0.2</td><td>--</td><td>Gal 2</td><td>10.4 deg</td><td>--</td><td>20.2 mag</td></tr>\n",
       "<tr><td>49.64667 deg</td><td>0.166</td><td>1.35 arcmin</td><td>Gal 1</td><td>9.14542 deg</td><td>-139.8</td><td>16.2 mag</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<QTable masked=True length=2>\n",
       "    dec        ebv   half-light_radius ...      ra     radial_velocity  v_mag  \n",
       "   object    float64       object      ...    object       float64      object \n",
       "------------ ------- ----------------- ... ----------- --------------- --------\n",
       "   -32.4 deg     0.2                -- ...    10.4 deg              -- 20.2 mag\n",
       "49.64667 deg   0.166       1.35 arcmin ... 9.14542 deg          -139.8 16.2 mag"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Tabular results of query\n",
    "db.query_table({})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A variety of query operators are implemented, including:\n",
    "\n",
    " - `$gt` greater than\n",
    " - `$gte` greater than or equal to\n",
    " - `$lt` less than\n",
    " - `$lte` less than or equal to\n",
    " - `$or` OR operator (AND is implicit in the query)\n",
    " \n",
    " More operators are accessible when using MongoDB rather than the local database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<i>QTable length=1</i>\n",
       "<table id=\"table4876857752\" class=\"table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>name</th><th>v_mag</th></tr></thead>\n",
       "<thead><tr><th>str5</th><th>object</th></tr></thead>\n",
       "<tr><td>Gal 1</td><td>16.2 mag</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<QTable length=1>\n",
       " name  v_mag  \n",
       " str5  object \n",
       "----- --------\n",
       "Gal 1 16.2 mag"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example query with operators ($)\n",
    "query = {'v_mag.value': {'$lt': 17}}\n",
    "df = db.query_table(query=query)\n",
    "df[['name', 'v_mag']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<i>QTable length=1</i>\n",
       "<table id=\"table4876860160\" class=\"table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>name</th><th>v_mag</th><th>radial_velocity</th></tr></thead>\n",
       "<thead><tr><th>str5</th><th>object</th><th>float64</th></tr></thead>\n",
       "<tr><td>Gal 1</td><td>16.2 mag</td><td>-139.8</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<QTable length=1>\n",
       " name  v_mag   radial_velocity\n",
       " str5  object      float64    \n",
       "----- -------- ---------------\n",
       "Gal 1 16.2 mag          -139.8"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example AND query\n",
    "query = {'v_mag.value': {'$gt': 10}, 'radial_velocity.value': {'$lte': -100}}\n",
    "db.query_table(query=query)[['name', 'v_mag', 'radial_velocity']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<i>QTable masked=True length=2</i>\n",
       "<table id=\"table4601221752\" class=\"table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>name</th><th>ra</th><th>dec</th><th>v_mag</th></tr></thead>\n",
       "<thead><tr><th>str5</th><th>object</th><th>object</th><th>object</th></tr></thead>\n",
       "<tr><td>Gal 2</td><td>10.4 deg</td><td>-32.4 deg</td><td>20.2 mag</td></tr>\n",
       "<tr><td>Gal 1</td><td>9.14542 deg</td><td>49.64667 deg</td><td>16.2 mag</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<QTable masked=True length=2>\n",
       " name      ra         dec       v_mag  \n",
       " str5    object      object     object \n",
       "----- ----------- ------------ --------\n",
       "Gal 2    10.4 deg    -32.4 deg 20.2 mag\n",
       "Gal 1 9.14542 deg 49.64667 deg 16.2 mag"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example OR query\n",
    "query = {'$or': [{'v_mag.value': 16.2}, {'dec.value': -32.4}]}\n",
    "df = db.query_table(query=query)\n",
    "df[['name', 'ra', 'dec', 'v_mag']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<i>QTable masked=True length=2</i>\n",
       "<table id=\"table4876932432\" class=\"table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>name</th><th>ra</th><th>v_mag</th></tr></thead>\n",
       "<thead><tr><th>str5</th><th>object</th><th>object</th></tr></thead>\n",
       "<tr><td>Gal 2</td><td>10.4 deg</td><td>20.2 mag</td></tr>\n",
       "<tr><td>Gal 1</td><td>9.14542 deg</td><td>16.2 mag</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<QTable masked=True length=2>\n",
       " name      ra      v_mag  \n",
       " str5    object    object \n",
       "----- ----------- --------\n",
       "Gal 2    10.4 deg 20.2 mag\n",
       "Gal 1 9.14542 deg 16.2 mag"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example AND and OR query\n",
    "query = {'ra.value': {'$gt': 10}, '$or': [{'v_mag.value': {'$lte': 21}},\n",
    "                                          {'v_mag.value': {'$gte': 16}}]}\n",
    "db.query_table(query=query)[['name', 'ra', 'v_mag']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nature of a document is that multiple values can be stored for each parameter. For example, consider Gal 1's values of RA:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'value': 9.14542, 'best': 1, 'reference': '', 'unit': 'deg'}\n",
      " {'value': 999.14542, 'best': 0, 'reference': 'FakeRef2019', 'unit': 'deg'}]\n"
     ]
    }
   ],
   "source": [
    "doc = db.query_db({'name': 'Gal 1'})[0]['ra']\n",
    "print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `selection` parameter in the `db.query_table` method can be used to control which reference to use in the table result, otherwise the 'best' value will be used. (This is anticipated to change)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " name      ra         dec     \n",
      "----- ----------- ------------\n",
      "Gal 1 9.14542 deg 49.64667 deg\n",
      " name       ra          dec     \n",
      "----- ------------- ------------\n",
      "Gal 1 999.14542 deg 49.64667 deg\n"
     ]
    }
   ],
   "source": [
    "query = {'name': 'Gal 1'}\n",
    "tab1 = db.query_table(query=query)[['name', 'ra', 'dec']]\n",
    "tab2 = db.query_table(query=query, selection={'ra': 'FakeRef2019'})[['name', 'ra', 'dec']]\n",
    "print(tab1) # default (best) results\n",
    "print(tab2) # user-selected results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "A reference JSON file can be provided which lists the basic author, journal, title, etc information for any reference used in the database. This data can be queried in a similar fashion and results can be embedded in the data query results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"key\": \"Martin_2005_1\",\n",
      "    \"id\": 2,\n",
      "    \"year\": 2005,\n",
      "    \"doi\": \"10.1111/j.1365-2966.2005.09339.x\",\n",
      "    \"bibcode\": \"2005MNRAS.362..906M\",\n",
      "    \"authors\": [\n",
      "        \"Martin, N. F.\",\n",
      "        \"Ibata, R. A.\",\n",
      "        \"Conn, B. C.\",\n",
      "        \"Lewis, G. F.\",\n",
      "        \"Bellazzini, M.\",\n",
      "        \"Irwin, M. J.\"\n",
      "    ],\n",
      "    \"journal\": \"MNRAS\",\n",
      "    \"title\": \"A radial velocity survey of low Galactic latitude structures - I. Kinematics of the Canis Major dwarf galaxy\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Query references information\n",
    "doc = db.query_reference({'key': 'Martin_2005_1'})[0]\n",
    "print(json.dumps(doc, indent=4, sort_keys=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"value\": 0.166,\n",
      "    \"best\": 1,\n",
      "    \"reference\": {\n",
      "        \"key\": \"Bellazzini_2006_1\",\n",
      "        \"id\": 1,\n",
      "        \"year\": 2006,\n",
      "        \"doi\": \"10.1111/j.1365-2966.2005.09973.x\",\n",
      "        \"bibcode\": \"2006MNRAS.366..865B\",\n",
      "        \"authors\": [\n",
      "            \"Bellazzini, M.\",\n",
      "            \"Ibata, R.\",\n",
      "            \"Martin, N.\",\n",
      "            \"Lewis, G. F.\",\n",
      "            \"Conn, B.\",\n",
      "            \"Irwin, M. J.\"\n",
      "        ],\n",
      "        \"journal\": \"MNRAS\",\n",
      "        \"title\": \"The core of the Canis Major galaxy as traced by red clump stars\"\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Query with references embedded\n",
    "doc = db.query_db({'name': 'Gal 1'}, embed_ref=True)[0]['ebv'][0]\n",
    "print(json.dumps(doc, indent=4, sort_keys=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data\n",
    "\n",
    "New objects (eg, galaxies) can be loaded to the database using the `db.load_file_to_db` method. This is also useful to reset data in memory to that from the disk. \n",
    "\n",
    "To add new parameters to existing objects, use `db.add_data`. Any new data added to the database can be explicitly written to disk with `db.save_all(out_dir)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create some extra data that we want to load to Gal 1. We'll use `db.add_data` to add this to the database, but will not use `db.save_all` as we don't want to permanently store this. Note that `db.add_data` can either take a filename or a dict-like object to load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data for Gal 1 has been updated. Consider running save_all() to update JSON on disk.\n"
     ]
    }
   ],
   "source": [
    "doc = {\n",
    "  \"name\": \"Gal 1\",\n",
    "  \"ra\": [\n",
    "    {\n",
    "      \"value\": 42,\n",
    "      \"reference\": \"Penguin_2020_1\",\n",
    "      \"unit\": \"deg\"\n",
    "    }\n",
    "  ],\n",
    "  \"ebv\": [\n",
    "    {\n",
    "      \"value\": 99,\n",
    "      \"reference\": \"Penguin_2020_1\"\n",
    "    }\n",
    "  ],\n",
    "  \"fake_quantity\": [\n",
    "    {\n",
    "      \"value\": 27,\n",
    "      \"error\": 2,\n",
    "      \"reference\": \"Penguin_2020_1\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "db.add_data(doc, validate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'value': 9.14542, 'best': 1, 'reference': '', 'unit': 'deg'}\n",
      " {'value': 999.14542, 'best': 0, 'reference': 'FakeRef2019', 'unit': 'deg'}\n",
      " {'value': 42, 'reference': 'Penguin_2020_1', 'unit': 'deg'}]\n",
      "[{'value': 0.166, 'best': 1, 'reference': 'Bellazzini_2006_1'}\n",
      " {'value': 99, 'reference': 'Penguin_2020_1'}]\n",
      "[{'value': 27, 'error': 2, 'reference': 'Penguin_2020_1'}]\n"
     ]
    }
   ],
   "source": [
    "doc = db.query_db({'name': 'Gal 1'})[0]\n",
    "print(doc['ra'])\n",
    "print(doc['ebv'])\n",
    "print(doc['fake_quantity'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Re-loading Gal 1\n",
    "db.load_file_to_db('galcat/tests/test_data/Gal_1.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation\n",
    "\n",
    "A validator exists to ensure that JSON added to the database meet some minimum criteria. This can be run against a JSON file itself, against a dict-like object, or against the full database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking Gal 2\n",
      "ERROR: ra has missing references or it does not exist: {'value': 10.4, 'best': 1, 'reference': '', 'unit': 'deg'}\n",
      "ERROR: dec has missing references or it does not exist: {'value': -32.4, 'best': 1, 'reference': '', 'unit': 'deg'}\n",
      "ERROR: v_mag has missing references or it does not exist: {'value': 20.2, 'error_upper': 0.5, 'error_lower': 0.5, 'best': 1, 'unit': 'mag', 'reference': ''}\n",
      "Checking Gal 1\n",
      "ERROR: ra has missing references or it does not exist: {'value': 9.14542, 'best': 1, 'reference': '', 'unit': 'deg'}\n",
      "ERROR: ra has missing references or it does not exist: {'value': 999.14542, 'best': 0, 'reference': 'FakeRef2019', 'unit': 'deg'}\n",
      "ERROR: dec has missing references or it does not exist: {'value': 49.64667, 'best': 1, 'reference': '', 'unit': 'deg'}\n",
      "ERROR: radial_velocity has missing references or it does not exist: {'value': -139.8, 'error_upper': 6.0, 'error_lower': 6.6, 'best': 1, 'reference': ''}\n",
      "ERROR: v_mag has missing references or it does not exist: {'value': 16.2, 'error_upper': 0.3, 'error_lower': 0.3, 'best': 1, 'unit': 'mag', 'reference': ''}\n",
      "ERROR: half-light_radius has missing references or it does not exist: {'value': 1.35, 'error_upper': 0.12, 'error_lower': 0.18, 'best': 1, 'unit': 'arcmin', 'reference': ''}\n",
      "Validation complete.\n"
     ]
    }
   ],
   "source": [
    "from galcat.validator import Validator\n",
    "\n",
    "# Validate full database, including checking that references are set\n",
    "Validator(database=db, is_data=True, ref_check=True, verbose=True).run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: This JSON represents a new/unmatched object: Gal 3. Use load_file_to_db() to load new objects.\n",
      "ERROR: ra has missing references or it does not exist: {'value': 9.14542, 'best': 1, 'reference': '', 'unit': 'deg'}\n",
      "ERROR: dec has missing references or it does not exist: {'value': 49.64667, 'best': 1, 'reference': '', 'unit': 'FAKE UNIT'}\n",
      "ERROR: dec has invalid units: FAKE UNIT\n",
      "ERROR: ebv has missing values/distribution: {'error_upper': 0.1, 'best': 1, 'reference': 'Bellazzini_2006_1'}\n",
      "Validation complete.\n"
     ]
    }
   ],
   "source": [
    "# Validate against dict-like object. For example, this may be a new galaxy you intend to add.\n",
    "doc = {\"name\": \"Gal 3\",\n",
    "       \"ra\": [{\"value\": 9.14542, \"best\": 1, \"reference\": \"\", \"unit\": \"deg\"}],\n",
    "       \"dec\": [{\"value\": 49.64667, \"best\": 1, \"reference\": \"\", \"unit\": \"FAKE UNIT\"}],\n",
    "       \"ebv\": [{\"error_upper\": 0.1, \"best\": 1, \"reference\": \"Bellazzini_2006_1\"}]}\n",
    "#print(json.dumps(doc, indent=4, sort_keys=False))\n",
    "\n",
    "Validator(database=db, db_object=doc, is_data=True, ref_check=True).run()  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Support for distributions is included with the use of the astropy.uncertainty module. Here's an example of how one can load a distribution and examine table results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       ebv          name\n",
      "------------------ -----\n",
      "0.9822537872413795 Gal 9\n"
     ]
    }
   ],
   "source": [
    "# Load a new galaxy with a distribution of E(B-V) values\n",
    "doc = {'name': 'Gal 9',\n",
    "           'ebv': [{'distribution': [1.20881063, 0.93766121, 1.20136033, 1.11122468, 0.88140548,\n",
    "           0.98529047, 0.83750181, 0.95603778, 0.90262727, 0.76719971,\n",
    "           0.96954131, 0.83957612, 1.05208742, 0.9203976 , 0.5388856 ,\n",
    "           0.82028187, 0.99002746, 0.99821842, 1.08264829, 0.88236597,\n",
    "           1.07393172, 0.68800062, 0.95087714, 0.95349601, 1.20331926,\n",
    "           1.1427941 , 1.13346843, 1.12862014, 1.32770298], 'reference': 'Fake'}]}\n",
    "db.load_file_to_db(doc)\n",
    "\n",
    "# Query the database for this galaxy\n",
    "df = db.query_table({'name': 'Gal 9'})\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `get_values_from_distribution` function is used to get the mean and standard deviation values of the distribution. If the distribution already contains units (for example, a numpy array of Quantity objects or similar), these are also stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'value': 0.9822537872413795, 'error': 0.16710914110088962}\n"
     ]
    }
   ],
   "source": [
    "doc = db.query({'name': 'Gal 9'})\n",
    "distr = doc[0]['ebv'][0]['distribution']\n",
    "print(get_values_from_distribution(distr))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'value': 0.9822537872413795, 'error': 0.16710914110088962, 'unit': 'kpc'}\n"
     ]
    }
   ],
   "source": [
    "doc = db.query({'name': 'Gal 9'})\n",
    "distr = np.array(doc[0]['ebv'][0]['distribution'])*u.kpc\n",
    "print(get_values_from_distribution(distr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using MongoDB\n",
    "\n",
    "If you have MongoDB, either locally or through a server, you can connect to it and perform similar queries as to what has been described above. Note that MongoDB adds a key, `_id`, to internally track its documents. If you need to remove it, you do so with `del doc['_id']`, where doc is the single-document output of a query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from galcat.core import *\n",
    "db = Database(conn_string='localhost', mongo_db_name='GalaxyCat', collection_name='galaxies_test')\n",
    "\n",
    "# If no database exists, can create it from the directory (will also update existing documents with those from disk)\n",
    "db.load_all('galcat/tests/test_data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<i>QTable masked=True length=2</i>\n",
       "<table id=\"table4876273480\" class=\"table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>_id</th><th>dec</th><th>ebv</th><th>half-light_radius</th><th>name</th><th>ra</th><th>radial_velocity</th><th>v_mag</th></tr></thead>\n",
       "<thead><tr><th>object</th><th>object</th><th>float64</th><th>object</th><th>str5</th><th>object</th><th>float64</th><th>object</th></tr></thead>\n",
       "<tr><td>5df3e16177219208dbc92a0e</td><td>-32.4 deg</td><td>0.2</td><td>--</td><td>Gal 2</td><td>10.4 deg</td><td>--</td><td>20.2 mag</td></tr>\n",
       "<tr><td>5df3e16177219208dbc92a10</td><td>49.64667 deg</td><td>0.166</td><td>1.35 arcmin</td><td>Gal 1</td><td>9.14542 deg</td><td>-139.8</td><td>16.2 mag</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<QTable masked=True length=2>\n",
       "          _id                dec        ebv   ... radial_velocity  v_mag  \n",
       "         object             object    float64 ...     float64      object \n",
       "------------------------ ------------ ------- ... --------------- --------\n",
       "5df3e16177219208dbc92a0e    -32.4 deg     0.2 ...              -- 20.2 mag\n",
       "5df3e16177219208dbc92a10 49.64667 deg   0.166 ...          -139.8 16.2 mag"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.query_table({})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'Gal 2', 'ra': array([{'value': 10.4, 'best': 1, 'reference': '', 'unit': 'deg'}],\n",
      "      dtype=object), 'dec': array([{'value': -32.4, 'best': 1, 'reference': '', 'unit': 'deg'}],\n",
      "      dtype=object), 'ebv': array([{'value': 0.2, 'best': 1, 'reference': 'Bellazzini_2006_1'}],\n",
      "      dtype=object), 'v_mag': array([{'value': 20.2, 'error_upper': 0.5, 'error_lower': 0.5, 'best': 1, 'unit': 'mag', 'reference': ''}],\n",
      "      dtype=object)}\n"
     ]
    }
   ],
   "source": [
    "doc = db.query_db({'name': 'Gal 2'})[0]\n",
    "del doc['_id']\n",
    "print(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}