diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..dbcaea95 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,29 @@ +# Notebooks 0.8.0 (27 June 2019) + +## New Features +- Each notebook is linked to in README.md +- PR #178 Adding KMeans notebook +- PR #18 Added cuDF example notebooks + +## Improvements +- Updated SSSP notebook +- Regression notebooks show of `train_test_split()` functon + +## Bug Fixes + + +# Notebooks 0.7.0 (10 May 2019) + +## New Features +- PR #133 Adding cuGraph notebooks + +## Improvements +- PR #144 Added top level CHANGELOG. Added a README to cugraph + +## Bug Fixes + + +# Notebooks 0.6.0 (22 Mar 2019) + + + diff --git a/README.md b/README.md index 0a63a16e..9639000f 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,70 @@ # RAPIDS Notebooks and Utilities -* `cuml`: contains four example notebooks showing the usage of different machine learning algorithms included in cuML: `knn`, `dbscan`, `pca` and `tsvd`. It also includes a small subset of the Mortgage Dataset used in the notebooks. -* `mortgage`: contains the notebook which runs ETL + ML on the Mortgage Dataset derived from [Fannie Mae’s Single-Family Loan Performance Data](http://www.fanniemae.com/portal/funding-the-market/data/loan-performance-data.html) ... download the mortgage dataset for use with the notebook [here](https://rapidsai.github.io/demos/datasets/mortgage-data) +## XGBoost Notebook +| Folder | Notebook Title | Description | +|-----------|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| XGBoost | [XGBoost Demo](xgboost/XGBoost_Demo.ipynb) | This notebook shows the acceleration one can gain by using GPUs with XGBoost in RAPIDS. | +## CuML Notebooks + The cuML notebooks showcase how to use the machine learning algorithms implemented in cuML along with the advantages of using cuML over scikit-learn. These notebooks compare the time required and the performance of the algorithms. Below are a list of such algorithms: + +| Folder | Notebook Title | Description | +|-----------|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| cuML | [dbscan_demo](cuml/dbscan_demo.ipynb) | This notebook showcases density-based spatial clustering of applications with noise (dbscan) algorithm using the `fit` and `predict` functions | +| cuML | [knn_demo](cuml/knn_demo.ipynb) | This notebook showcases k-nearest neighbors (knn) algorithm using the `fit` and `kneighbors` functions | +| cuML | [Linear Regression Demo](cuml/linear_regression_demo.ipynb) | This notebook includes code example for linear regression algorithm and it showcases the `fit` and `predict` functions. | +| cuML | [Ridge Regression Demo](cuml/ridge_regression_demo.ipynb) | This notebook includes code examples of ridge regression and it showcases the `fit` and `predict` functions. | +| cuML | [Coordinate Descent](cuml/coordinate_descent_demo.ipynb) | This notebook includes code examples of lasso and elastic net models. These models are placed together so a comparison between the two can also be made in addition to their sklearn equivalent. | +| cuML | [pca_demo](cuml/pca_demo.ipynb) | This notebook showcases principal component analysis (PCA) algorithm where the model can be used for prediction (using `fit_transform`) as well as converting the transformed data into the original dataset (using `inverse_transform`). | +| cuML | [tsvd_demo](cuml/tsvd_demo.ipynb ) | This notebook showcases truncated singular value decomposition (tsvd) algorithm which like PCA performs both prediction and transformation of the converted dataset into the original data using `fit_transform` and `inverse_transform` functions respectively | +| cuML | [sgd_demo](cuml/sgd_demo.ipynb) | The stochastic gradient descent algorithm is demostrated in the notebook using `fit` and `predict` functions | +| cuML | [umap_demo](cuml/umap_demo.ipynb) | The uniform manifold approximation & projection algorithm is compared with the original author's equivalent non-GPU \Python implementation using `fit` and `transform` functions | +| cuML | [umap_demo_graphed](cuml/umap_demo_graphed.ipynb) | Demonstration of cuML uniform manifold approximation & projection algorithm's supervised approach against mortgage dataset and comparison of results against the original author's equivalent non-GPU \Python implementation. | +| cuML | [umap_demo_supervised](cuml/umap_supervised_demo.ipynb) | Demostration of UMAP supervised training. Uses a set of labels to perform supervised dimensionality reduction. UMAP can also be trained on datasets with incomplete labels, by using a label of "-1" for unlabeled samples. | +| cuML | [random forest](cuml/rf_demo.ipynb) | This notebook includes code examples of Random Forest and it showcases the `fit` and `predict` functions. | + +## CuDF Notebooks +| Folder | Notebook Title | Description | +|-----------|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| cuDF | [notebooks_Apply_Operations_in_cuDF](cudf/notebooks_Apply_Operations_in_cuDF.ipynb) | This notebook showcases two special methods where cuDF goes beyond the Pandas library: apply_rows and apply_chunk functions. They utilized the Numba library to accelerate the data transformation via GPU in parallel. | +| cuDF | [notebooks_numba_cuDF_integration](cudf/notebooks_numba_cuDF_integration.ipynb) | This notebook showcases how to use Numba CUDA to accelerate cuDF data transformation and how to step by step accelerate it using CUDA programming tricks | + +## CuGraph Notebooks +| Folder | Notebook Title | Description | +|-----------|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| cuGraph | [Louvain](cugraph/Louvain.ipynb) | Demonstration of using cuGraph to identify clusters in a test graph using the Louvain algorithm | +| cuGraph | [Vertex-Similarity](cugraph/Vertex-Similarity.ipynb) | Demonstration of using cuGraph to compute vertex similarity using both the Jaccard Similarity and the Overlap Coefficient. | +| cuGraph | [Weighted-Jaccard](cugraph/Weighted-Jaccard.ipynb) | Demonstration of using cuGraph to compute the Weighted Jaccard Similarity metric on our training dataset. | +| cuGraph | [Renumber](cugraph/Renumber.ipynb) | Demonstrate of using the renumbering features to assigned new vertex IDs to the test graph. This is useful for when the data sets is non-contiguous or not integer values | +| cuGraph | [BFS](cugraph/BFS.ipynb) | Demonstration of using cuGraph to computer the Bredth First Search space from a given vertex to all other in our training graph | +| cuGraph | [SSSP](cugraph/SSSP.ipynb) | Demonstration of using cuGraph to computer the The Shortest Path from a given vertex to all other in our training graph | +| cuGraph | [Spectral-Clustering](cugraph/Spectral-Clustering.ipynb) | Demonstration of using cuGraph to identify clusters in a test graph using Spectral Clustering using both the (A) Balance Cut and (B) the Modularity Maximization quality metrics | +| cuGraph | [Pagerank](cugraph/Pagerank.ipynb) | Demonstration of using both NetworkX and cuGraph to compute the PageRank of each vertex in our test dataset | +| cuGraph | [Triangle Counting](cugraph/Triangle-Counting.ipynb) | Demonstration of using both NetworkX and cuGraph to compute the the number of Triangles in our test dataset | + +## Tutorial with an End to End workflow + +| Folder | Notebook Title | Description | +|-----------|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Tutorials | [DBSCAN_demo_full](tutorials/DBSCAN_Demo_Full.ipynb) | Demonstration of how to use DBSCAN - a popular clustering algorithm - and how to use the GPU accelerated implementation of this algorithm in RAPIDS. | + +## Utils Scripts +| Folder | Script Title | Description | +|-----------|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Utils | start-jupyter.sh | starts a JupyterLab environment for interacting with, and running, notebooks | +| Utils | stop-jupyter.sh | identifies all process IDs associated with Jupyter and kills them | +| Utils | dask-cluster.py | launches a configured Dask cluster (a set of nodes) for use within a notebook | +| Utils | dask-setup.sh | a low-level script for constructing a set of Dask workers on a single node | +| Utils | split-data-mortgage.sh | splits mortgage data files into smaller parts, and saves them for use with the mortgage notebook | + +## Documentation (WIP) +| Folder | Document Title | Description | +|-----------|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Docs | ngc-readme | | +| Docs | dockerhub-readme | | + +## Additional Information +* The `cuml` folder also includes a small subset of the Mortgage Dataset used in the notebooks and the full image set from the Fashion MNIST dataset. + * `utils`: contains a set of useful scripts for interacting with RAPIDS + +* For additional, community driven notebooks, which will include our blogs, tutorials, workflows, and more intricate examples, please see the [Notebooks Extended Repo](https://github.com/rapidsai/notebooks-extended) diff --git a/blogs/Regression Blog 1 Notebook.ipynb b/blogs/Regression Blog 1 Notebook.ipynb deleted file mode 100644 index ffed1d03..00000000 --- a/blogs/Regression Blog 1 Notebook.ipynb +++ /dev/null @@ -1,304 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import pandas as pd\n", - "import numpy as np\n", - "import cudf \n", - "import cuml\n", - "import os" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This data description, and the data that this example uses are available at \n", - "[the UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset). " - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "RAW_DATA = os.path.expanduser('~/Data/try_this.csv')" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "# Data Description: \n", - "# \n", - "# - instant: record index\n", - "# - dteday : date\n", - "# - season : season (1:springer, 2:summer, 3:fall, 4:winter)\n", - "# - yr : year (0: 2011, 1:2012)\n", - "# - mnth : month ( 1 to 12)\n", - "# - hr : hour (0 to 23)\n", - "# - holiday : weather day is holiday or not (extracted from http://dchr.dc.gov/page/holiday-schedule)\n", - "# - weekday : day of the week\n", - "# - workingday : if day is neither weekend nor holiday is 1, otherwise is 0.\n", - "# + weathersit : \n", - "# - 1: Clear, Few clouds, Partly cloudy, Partly cloudy\n", - "# - 2: Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist\n", - "# - 3: Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds\n", - "# - 4: Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog\n", - "# - temp : Normalized temperature in Celsius. The values are divided to 41 (max)\n", - "# - atemp: Normalized feeling temperature in Celsius. The values are divided to 50 (max)\n", - "# - hum: Normalized humidity. The values are divided to 100 (max)\n", - "# - This is a good example of variables we might not have a good theoretical intuation around\n", - "# - windspeed: Normalized wind speed. The values are divided to 67 (max)\n", - "# - casual: count of casual users\n", - "# - registered: count of registered users\n", - "# - cnt: count of total rental bikes including both casual and registered" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "gdf = cudf.read_csv(RAW_DATA)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We drop the index, the timestamp (because they have broken it down for us), \n", - "and the individual counts that make up our target variable." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "\n", - "drop_list = ['instant', 'dteday', 'casual', 'registered']\n", - "gdf = gdf.drop(drop_list)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We're going to create one-hot encoded variables, also known as dummy variables, for each of the time variables as well as the weather situation. We're going to drop one of each of these dummy variables so we don't create colinearity. \n", - "\n", - "The next data munging step we take is to convert all of our data into the same type, because that is what the cuML algorithms are expecting. \n", - "\n", - "Last, we split our data into test and train sets, training on 2011 data, and testing on 2012. " - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [], - "source": [ - "dummies_list = ['season','yr', 'mnth', 'hr', 'weekday', 'weathersit']\n", - "\n", - "for item in dummies_list:\n", - " codes = gdf[item].unique()\n", - " gdf = gdf.one_hot_encoding(item, '{}_dummy'.format(item), codes)\n", - " gdf = gdf.drop('{}_dummy_1'.format(item))\n", - "\n", - "#cuML current requires all data be of the same type, so this loop converts all values into floats\n", - "for col in gdf.columns.tolist():\n", - " gdf[col] = gdf[col].astype('float32')\n", - " \n", - "test = gdf.query('yr == 1').drop(dummies_list)\n", - "train = gdf.query('yr == 0').drop(dummies_list)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "I am going to test out how well a small variable does against all the variables available. I select \"weathersit_dummy_4\" because as we see above, that seems like the worst weather for bike riding. Also, based on personal experience, bike riding in high wind is not fun either. I add workday because I'm sure it has some impact, but I'm not sure what. " - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [], - "source": [ - "y_train = train['cnt']\n", - "y_test = test['cnt']\n", - "\n", - "#Some of the variables, chosen by theory\n", - "X_train_1 = train[['weathersit_dummy_4', 'windspeed', 'workingday']]\n", - "X_test_1 = test[['weathersit_dummy_4', 'windspeed', 'workingday']]\n", - "\n", - "#all of the varibles.\n", - "X_train_2 = train.drop('cnt')\n", - "X_test_2 = test.drop('cnt')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here, I run two regressions. The first is based on small set of variable I think will be most impactful to bike ridership. The second regression inclueds all the variables availale." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [], - "source": [ - "OLS_1 = cuml.LinearRegression()\n", - "fit_1 = OLS_1.fit(X_train_1, y_train)\n", - "y_hat_1 = fit_1.predict(X_test_1)\n", - "MSE_1 = ((y_test - y_hat_1)**2).sum()" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [], - "source": [ - "OLS_2 = cuml.LinearRegression()\n", - "fit_2 = OLS_2.fit(X_train_2, y_train)\n", - "y_hat_2 = fit_2.predict(X_test_2)\n", - "MSE_2 = ((y_test - y_hat_2)**2).sum()" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [], - "source": [ - "output = {'MSE_OLS_THEORY':MSE_1,\n", - " 'MSE_OLS_ALL': MSE_2}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It looks like I was wrong, and the model with everything performs better on the out-of-sample data. " - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "MSE_OLS_THEORY: 449879008.0\n", - "MSE_OLS_EVERYTHING: 266687056.0\n" - ] - } - ], - "source": [ - "print('MSE_OLS_THEORY: {}'.format(MSE_1))\n", - "print('MSE_OLS_EVERYTHING: {}'.format(MSE_2))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The \"everything\" model outperformed the small model, but let's see if we can do better by doing a Ridge regression. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We're going to do a small hyperparameter search for alpha, checking 100 different values. This is fast to do with RAPIDS. Also notice that I am appending the results of each Ridge model onto the dictionary containing our earlier results, so I can more easily see which model is the best at the end. " - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [], - "source": [ - "for alpha in np.arange(0.0, 1, 0.01):\n", - " \n", - " Ridge = cuml.Ridge(alpha=alpha, fit_intercept=True)\n", - " fit_3 = Ridge.fit(X_train_2, y_train)\n", - " y_hat_3 = fit_3.predict(X_test_2)\n", - " MSE_3 = ((y_test - y_hat_3)**2).sum()\n", - " output['MSE_RIDGE_{}'.format(alpha)] = MSE_3" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here we see that our regulaized model does better than the rest, include OLS with all the variables. " - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Min MSE: MSE_RIDGE_0.1\n" - ] - } - ], - "source": [ - "print('Min MSE: {}'.format(min(output, key=output.get)))" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [], - "source": [ - "#You can uncomment the code below to see all 100 model MSEs\n", - "#Notice in particular that MSE for OLS with everything and Ridge with alpha = 0 are essentially the same. \n", - "\n", - "#print(output)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.1" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/cudf/notebooks_Apply_Operations_in_cuDF.ipynb b/cudf/notebooks_Apply_Operations_in_cuDF.ipynb new file mode 100755 index 00000000..504aafb0 --- /dev/null +++ b/cudf/notebooks_Apply_Operations_in_cuDF.ipynb @@ -0,0 +1,302 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Background\n", + "\n", + "The RAPIDS cuDF library is a GPU DataFrame manipulation library based on Apache Arrow that accelerates loading, filtering, and manipulation of data for model training data preparation. It provides a pandas-like API that will be familiar to data scientists. Pandas lib provides a lot of special methods that covers most of the use cases for data scientists. However, it cannot cover all the cases, sometimes it is nice to have a way to accelerate the customized data transformation. Luckily, in cuDF, there are two special methods that serve this particular purpose: apply_rows and apply_chunk functions. They utilized the Numba library to accelerate the data transformation via GPU in parallel.\n", + "\n", + "In this tutorial, I am going to show a few examples of how to use it." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Difference between `apply_rows` and `apply_chunks`\n", + "\n", + "`apply_rows` is a special case of `apply_chunks`, which processes each of the rows of the Dataframe independently in parallel. Under the hood, the `apply_rows` method will optimally divide the long columns into chunks, and assign chunks into different GPU blocks to compute. Here is one example, I am using `apply_rows` to double the input array and also print out the GPU block/grid allocation information." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "999000.0\n", + "999000.0\n" + ] + } + ], + "source": [ + "import cudf\n", + "import numpy as np\n", + "from numba import cuda\n", + " \n", + "df = cudf.dataframe.DataFrame()\n", + "df['in1'] = np.arange(1000, dtype=np.float64)\n", + " \n", + "def kernel(in1, out):\n", + " for i, x in enumerate(in1):\n", + " print('tid:', cuda.threadIdx.x, 'bid:', cuda.blockIdx.x,\n", + " 'array size:', in1.size, 'block threads:', cuda.blockDim.x)\n", + " out[i] = x * 2.0\n", + " \n", + "outdf = df.apply_rows(kernel,\n", + " incols=['in1'],\n", + " outcols=dict(out=np.float64),\n", + " kwargs=dict())\n", + " \n", + "print(outdf['in1'].sum()*2.0)\n", + "print(outdf['out'].sum())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "From the output.txt, we can see that the for-loop in the kernel function is unrolled by the compiler automatically. It uses 14 CUDA blocks. Each CUDA block uses 64 threads to do the computation. Each of the thread in the block most of the time deals one element in the input array and sometimes deals with two elements. The order of processing row element is not defined. \n", + "\n", + "We implement the same array double logic with the apply_chunks method." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "9900.0\n", + "9900.0\n" + ] + } + ], + "source": [ + "import cudf\n", + "import numpy as np\n", + "from numba import cuda\n", + " \n", + " \n", + "df = cudf.dataframe.DataFrame()\n", + "df['in1'] = np.arange(100, dtype=np.float64)\n", + " \n", + " \n", + "def kernel(in1, out):\n", + " print('tid:', cuda.threadIdx.x, 'bid:', cuda.blockIdx.x,\n", + " 'array size:', in1.size, 'block threads:', cuda.blockDim.x)\n", + " for i in range(cuda.threadIdx.x, in1.size, cuda.blockDim.x):\n", + " out[i] = in1[i] * 2.0\n", + " \n", + "outdf = df.apply_chunks(kernel,\n", + " incols=['in1'],\n", + " outcols=dict(out=np.float64),\n", + " kwargs=dict(),\n", + " chunks=16,\n", + " tpb=8)\n", + " \n", + "print(outdf['in1'].sum()*2.0)\n", + "print(outdf['out'].sum())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "From the output.txt, we can see apply_chunks has more control than the apply_rows method. It can specify how to divide the long array into chunks, map each of the array chunks to different GPU blocks to process (chunks argument) and assign the number of thread in the block (tpb argument). The for-loop is no longer automatically unrolled in the kernel function as apply_rows method but stays as the for-loop for that GPU thread. Each kernel corresponds to each thread in one block and it has full access to all the elements in that chunk of the array. In this example, the chunk size is 16, and it uniformly cute the 100 elements into 7 chunks (except the last one) and assign them to 7 blocks. Each block has 8 thread to process this length 16 subarray (or length 4 for the last block). \n", + "\n", + "## Performance benchmark compare\n", + "\n", + "Here we compare the benefits of using cuDF apply_rows vs pandas apply method by the following python code:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import cudf\n", + "import pandas as pd\n", + "import numpy as np\n", + "import time\n", + " \n", + " \n", + "data_length = 1e6\n", + "df = cudf.dataframe.DataFrame()\n", + "df['in1'] = np.arange(data_length, dtype=np.float64)\n", + " \n", + " \n", + "def kernel(in1, out):\n", + " for i, x in enumerate(in1):\n", + " out[i] = x * 2.0\n", + " \n", + "start = time.time()\n", + "df = df.apply_rows(kernel,\n", + " incols=['in1'],\n", + " outcols=dict(out=np.float64),\n", + " kwargs=dict())\n", + "end = time.time()\n", + "print('cuDF time', end-start)\n", + "assert(np.isclose(df['in1'].sum()*2.0, df['out'].sum()))\n", + " \n", + " \n", + "df = pd.DataFrame()\n", + "df['in1'] = np.arange(data_length, dtype=np.float64)\n", + "start = time.time()\n", + "df['out'] = df.in1.apply(lambda x: x*2)\n", + "end = time.time()\n", + "print('pandas time', end-start)\n", + "assert(np.isclose(df['in1'].sum()*2.0, df['out'].sum()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We change the data_length from 1e4 to 1e7, here is the computation time spent in cuDF and pandas.\n", + "\n", + "| data length | 1e3 | 1e4 | 1e5 | 1e6 | 1e7 | 1e8 |\n", + "| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n", + "| cuDF Time(s)| 0.1750 | 0.1840 | 0.1750 | 0.1720 | 0.1770 | 0.2490 |\n", + "| Pandas Time(s)| 0.0006 | 0.0022 | 0.0180 | 0.2500 | 2.1300 | 21.400 |\n", + "| Speed Up | 0.003x | 0.011x | 0.103x | 1.453x | **12.034x** | **85.944x** |\n", + "\n", + "As we can see, the cuDF has an overhead of launching GPU kernels(mostly the kernel compilation time), the computation time remains relatively constant in this test due to the massive number of cores in P100 card. While the CPU computation scales linearly with the length of the array due to the series computation nature of the \"apply\" function. cuDF has the advantage in computation once the array size is larger than one million. \n", + "\n", + "## Realistic application\n", + "\n", + "In the financial service industry, data scientists usually need to compute features from time series data. The most popular method to process the time series data is to compute moving average. In this example, I am going to show how to utilize apply_chunks to speed up moving average computation for a long array." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "cuDF time 1.2836346626281738\n", + "pandas time 6.339383363723755\n" + ] + } + ], + "source": [ + "import cudf\n", + "import numpy as np\n", + "import pandas as pd\n", + "from numba import cuda\n", + "import time\n", + " \n", + "data_length = int(1e9)\n", + "average_window = 4\n", + "df = cudf.dataframe.DataFrame()\n", + "threads_per_block = 128\n", + "trunk_size = 10240\n", + "df['in1'] = np.arange(data_length, dtype=np.float64)\n", + " \n", + " \n", + "def kernel1(in1, out, average_length):\n", + " for i in range(cuda.threadIdx.x,\n", + " average_length-1, cuda.blockDim.x):\n", + " out[i] = np.inf\n", + " for i in range(cuda.threadIdx.x + average_length - 1,\n", + " in1.size, cuda.blockDim.x):\n", + " summ = 0.0\n", + " for j in range(i - average_length + 1,\n", + " i + 1):\n", + " summ += in1[j]\n", + " out[i] = summ / np.float64(average_length)\n", + " \n", + "def kernel2(in1, out, average_length):\n", + " if in1.size - average_length + cuda.threadIdx.x - average_length + 1 < 0 :\n", + " return\n", + " for i in range(in1.size - average_length + cuda.threadIdx.x,\n", + " in1.size, cuda.blockDim.x):\n", + " summ = 0.0\n", + " for j in range(i - average_length + 1,\n", + " i + 1):\n", + " #print(i,j, in1.size)\n", + " summ += in1[j]\n", + " out[i] = summ / np.float64(average_length)\n", + " \n", + " \n", + "start = time.time()\n", + "df = df.apply_chunks(kernel1,\n", + " incols=['in1'],\n", + " outcols=dict(out=np.float64),\n", + " kwargs=dict(average_length=average_window),\n", + " chunks=list(range(0, data_length,\n", + " trunk_size))+ [data_length],\n", + " tpb=threads_per_block)\n", + " \n", + "df = df.apply_chunks(kernel2,\n", + " incols=['in1', 'out'],\n", + " outcols=dict(),\n", + " kwargs=dict(average_length=average_window),\n", + " chunks=[0]+list(range(average_window, data_length,\n", + " trunk_size))+ [data_length],\n", + " tpb=threads_per_block)\n", + "end = time.time()\n", + "print('cuDF time', end-start)\n", + " \n", + "pdf = pd.DataFrame()\n", + "pdf['in1'] = np.arange(data_length, dtype=np.float64)\n", + "start = time.time()\n", + "pdf['out'] = pdf.rolling(average_window).mean()\n", + "end = time.time()\n", + "print('pandas time', end-start)\n", + " \n", + "assert(np.isclose(pdf.out.as_matrix()[average_window:].mean(),\n", + " df.out.to_array()[average_window:].mean()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the above code, we divide the array into subarrays of size \"trunk_size\", and send those subarrays to GPU blocks to compute moving average. However, there is no history for the elements at the beginning of the subarray. To fix this, we shift the chunk division by an offset of \"average_window\". Then we call the kernel2 to compute the moving average of those missing records only.. Note, in kernel2, we didn't define outcols as it will create a new GPU memory buffer and overwrite the old \"out\" values. Instead, we reuse out array as input. For an array of 1e9 length, cuDF uses 1.387s to do the computation while pandas use 7.58s. \n", + "\n", + "This code is not optimized in performance. There are a few things we can do to make it faster. First, we can use shared memory to load the array and reduce the IO when doing the summation. Secondly, there is a lot of redundant summation done by the threads. We can maintain an accumulation summation array to help reduce the redundancy. This is outside the scope of this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.5.6" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cudf/notebooks_numba_cuDF_integration.ipynb b/cudf/notebooks_numba_cuDF_integration.ipynb new file mode 100644 index 00000000..9e6669f5 --- /dev/null +++ b/cudf/notebooks_numba_cuDF_integration.ipynb @@ -0,0 +1,459 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Objective\n", + "\n", + "In my previous tutorial, I showed how to use `apply_rows` and `apply_chunks` methods in cuDF to implement customized data transformations. Under the hood, they are all using [Numba library](https://numba.pydata.org/) to compile the normal python code into GPU kernels. Numba is an excellent python library that accelerates the numerical computations. Most importantly, Numba has direct CUDA programming support. For detailed information, please check out this [Numba CUDA document](https://numba.pydata.org/numba-doc/dev/cuda/index.html). As we know, the underlying data structure of cuDF is a GPU version of Apache Arrow. We can directly pass the GPU array around without the copying operation. Once we have the nice Numba library and standard GPU array, the sky is the limit. In this tutorial, I will show how to use Numba CUDA to accelerate cuDF data transformation and how to step by step accelerate it using CUDA programming tricks. \n", + "\n", + "The following experiments are performed at DGX V100 node." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## A simple example\n", + "As usual, I am going to start with a simple example of doubling the numbers in an array:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "import cudf\n", + "import numpy as np\n", + "from numba import cuda\n", + " \n", + "array_len = 1000\n", + "number_of_threads = 128\n", + "number_of_blocks = (array_len + (number_of_threads - 1)) // number_of_threads\n", + "df = cudf.dataframe.DataFrame()\n", + "df['in'] = np.arange(array_len, dtype=np.float64)\n", + " \n", + " \n", + "@cuda.jit\n", + "def double_kernel(result, array_len):\n", + " \"\"\"\n", + " double each element of the array\n", + " \"\"\"\n", + " i = cuda.grid(1)\n", + " if i < array_len:\n", + " result[i] = result[i] * 2.0\n", + " \n", + " \n", + "before = df['in'].sum()\n", + "gpu_array = df['in'].to_gpu_array()\n", + "print(type(gpu_array))\n", + "double_kernel[(number_of_blocks,), (number_of_threads,)](gpu_array, array_len)\n", + "after = df['in'].sum()\n", + "assert(np.isclose(before * 2.0, after))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "From the output of this code, it shows the underlying GPU array is of type `numba.cuda.cudadrv.devicearray.DeviceNDArray`. We can directly pass it to the kernel function that is compiled by the `cuda.jit`. Because we passed in the reference, the effect of number transformation will automatically show up in the original cuDF Dataframe. Note we have to manually enter the block size and grid size, which gives us the maximum of GPU programming control. The `cuda.grid` is a convenient method to compute the absolute position for the threads. It is equivalent to the normal `block_id * block_dim + thread_id` formula." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Practical example\n", + "\n", + "### Baseline\n", + "\n", + "We will work on the moving average problem as the last time. Because we have the full control of the grid and block size allocation, the vanilla moving average implementation code is much simpler compared to the `apply_chunks` implementation. " + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "%reset -s -f" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Numba with comipile time 2.067620038986206\n", + "Numba without comipile time 1.9229750633239746\n", + "pandas time 5.2932703495025635\n" + ] + } + ], + "source": [ + "import cudf\n", + "import numpy as np\n", + "import pandas as pd\n", + "from numba import cuda\n", + "import numba\n", + "import time\n", + " \n", + "array_len = int(5e8)\n", + "average_window = 3000\n", + "number_of_threads = 128\n", + "number_of_blocks = (array_len + (number_of_threads - 1)) // number_of_threads\n", + "df = cudf.dataframe.DataFrame()\n", + "df['in'] = np.arange(array_len, dtype=np.float64)\n", + "df['out'] = np.arange(array_len, dtype=np.float64)\n", + " \n", + " \n", + "@cuda.jit\n", + "def kernel1(in_arr, out_arr, average_length, arr_len):\n", + " s = numba.cuda.local.array(1, numba.float64)\n", + " s[0] = 0.0\n", + " i = cuda.grid(1)\n", + " if i < arr_len:\n", + " if i < average_length-1:\n", + " out_arr[i] = np.inf\n", + " else:\n", + " for j in range(0, average_length):\n", + " s[0] += in_arr[i-j]\n", + " out_arr[i] = s[0] / np.float64(average_length)\n", + " \n", + " \n", + "gpu_in = df['in'].to_gpu_array()\n", + "gpu_out = df['out'].to_gpu_array()\n", + "start = time.time()\n", + "kernel1[(number_of_blocks,), (number_of_threads,)](gpu_in, gpu_out,\n", + " average_window, array_len)\n", + "cuda.synchronize()\n", + "end = time.time()\n", + "print('Numba with comipile time', end-start)\n", + " \n", + "start = time.time()\n", + "kernel1[(number_of_blocks,), (number_of_threads,)](gpu_in, gpu_out,\n", + " average_window, array_len)\n", + "cuda.synchronize()\n", + "end = time.time()\n", + "print('Numba without comipile time', end-start)\n", + " \n", + "pdf = pd.DataFrame()\n", + "pdf['in'] = np.arange(array_len, dtype=np.float64)\n", + "start = time.time()\n", + "pdf['out'] = pdf.rolling(average_window).mean()\n", + "end = time.time()\n", + "print('pandas time', end-start)\n", + " \n", + "assert(np.isclose(pdf.out.values[average_window:].mean(),\n", + " df.out.to_array()[average_window:].mean()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note, in order to compare the computation time accurately, I launch the kernel twice. The first time kernel launching will include the kernel compilation time. In this example, it takes 1.9s for the kernel to run without compilation. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Use shared memory\n", + "\n", + "In the baseline code, each thread is reading the numbers from the global memory. When doing the moving average, the same number is read multiple times by different threads. GPU global memory IO, in this case, is the speed bottleneck. To mitigate it, we load the data into shared memory for each of the computation blocks. Then the threads are doing summation from the numbers in the cache. To do the moving average for the elements at the beginning of the array, we make sure to load the `average_window` more data in the shared_memory. " + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "%reset -s -f" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Numba with comipile time 1.3115026950836182\n", + "Numba without comipile time 1.085998773574829\n", + "pandas time 5.594487428665161\n" + ] + } + ], + "source": [ + "import cudf\n", + "import numpy as np\n", + "import pandas as pd\n", + "from numba import cuda\n", + "import numba\n", + "import time\n", + " \n", + "array_len = int(5e8)\n", + "average_window = 3000\n", + "number_of_threads = 128\n", + "number_of_blocks = (array_len + (number_of_threads - 1)) // number_of_threads\n", + "shared_buffer_size = number_of_threads + average_window - 1\n", + "df = cudf.dataframe.DataFrame()\n", + "df['in'] = np.arange(array_len, dtype=np.float64)\n", + "df['out'] = np.arange(array_len, dtype=np.float64)\n", + " \n", + " \n", + "@cuda.jit\n", + "def kernel1(in_arr, out_arr, average_length, arr_len):\n", + " block_size = cuda.blockDim.x\n", + " shared = cuda.shared.array(shape=(shared_buffer_size),\n", + " dtype=numba.float64)\n", + " i = cuda.grid(1)\n", + " tx = cuda.threadIdx.x\n", + " # Block id in a 1D grid\n", + " bid = cuda.blockIdx.x\n", + " starting_id = bid * block_size\n", + " \n", + " shared[tx + average_length - 1] = in_arr[i]\n", + " cuda.syncthreads()\n", + " for j in range(0, average_length - 1, block_size):\n", + " if (tx + j) < average_length - 1:\n", + " shared[tx + j] = in_arr[starting_id -\n", + " average_length + 1 +\n", + " tx + j]\n", + " cuda.syncthreads()\n", + " \n", + " s = numba.cuda.local.array(1, numba.float64)\n", + " s[0] = 0.0\n", + " if i < arr_len:\n", + " if i < average_length-1:\n", + " out_arr[i] = np.inf\n", + " else:\n", + " for j in range(0, average_length):\n", + " s[0] += shared[tx + average_length - 1 - j]\n", + " out_arr[i] = s[0] / np.float64(average_length)\n", + " \n", + " \n", + "gpu_in = df['in'].to_gpu_array()\n", + "gpu_out = df['out'].to_gpu_array()\n", + "start = time.time()\n", + "kernel1[(number_of_blocks,), (number_of_threads,)](gpu_in, gpu_out,\n", + " average_window, array_len)\n", + "cuda.synchronize()\n", + "end = time.time()\n", + " \n", + "print('Numba with comipile time', end-start)\n", + " \n", + "start = time.time()\n", + "kernel1[(number_of_blocks,), (number_of_threads,)](gpu_in, gpu_out,\n", + " average_window, array_len)\n", + "cuda.synchronize()\n", + "end = time.time()\n", + "print('Numba without comipile time', end-start)\n", + " \n", + "pdf = pd.DataFrame()\n", + "pdf['in'] = np.arange(array_len, dtype=np.float64)\n", + "start = time.time()\n", + "pdf['out'] = pdf.rolling(average_window).mean()\n", + "end = time.time()\n", + "print('pandas time', end-start)\n", + " \n", + "assert(np.isclose(pdf.out.values[average_window:].mean(),\n", + " df.out.to_array()[average_window:].mean()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Running this, the computation time is reduced to 1.09s without kernel compilation time. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Reduced redundant summations\n", + "\n", + "Each thread in the above code is doing one moving average in a for-loop. It is easy to see that there are a lot of redundant summation operations done by different threads. To reduce the redundancy, the following code is changed to let each thread to compute a consecutive number of moving averages. The later moving average step is able to reuse the sum of the previous steps. This eliminated `thread_tile` number of for-loops. " + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "%reset -s -f" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Numba with comipile time 0.6331000328063965\n", + "Numba without comipile time 0.30219364166259766\n", + "pandas time 6.03054666519165\n" + ] + } + ], + "source": [ + "import cudf\n", + "import numpy as np\n", + "import pandas as pd\n", + "from numba import cuda\n", + "import numba\n", + "import time\n", + " \n", + "array_len = int(5e8)\n", + "average_window = 3000\n", + "number_of_threads = 64\n", + "thread_tile = 48\n", + "number_of_blocks = (array_len + (number_of_threads * thread_tile - 1)) // (number_of_threads * thread_tile)\n", + "shared_buffer_size = number_of_threads * thread_tile + average_window - 1\n", + "df = cudf.dataframe.DataFrame()\n", + "df['in'] = np.arange(array_len, dtype=np.float64)\n", + "df['out'] = np.arange(array_len, dtype=np.float64)\n", + " \n", + " \n", + "@cuda.jit\n", + "def kernel1(in_arr, out_arr, average_length, arr_len):\n", + " block_size = cuda.blockDim.x\n", + " shared = cuda.shared.array(shape=(shared_buffer_size),\n", + " dtype=numba.float64)\n", + " tx = cuda.threadIdx.x\n", + " # Block id in a 1D grid\n", + " bid = cuda.blockIdx.x\n", + " starting_id = bid * block_size * thread_tile\n", + " \n", + " for j in range(thread_tile):\n", + " shared[tx + j * block_size + average_length - 1] = in_arr[starting_id\n", + " + tx +\n", + " j * block_size]\n", + " cuda.syncthreads()\n", + " for j in range(0, average_length - 1, block_size):\n", + " if (tx + j) < average_length - 1:\n", + " shared[tx + j] = in_arr[starting_id -\n", + " average_length + 1 +\n", + " tx + j]\n", + " cuda.syncthreads()\n", + " \n", + " s = numba.cuda.local.array(1, numba.float64)\n", + " first = False\n", + " s[0] = 0.0\n", + " for k in range(thread_tile):\n", + " i = starting_id + tx * thread_tile + k\n", + " if i < arr_len:\n", + " if i < average_length-1:\n", + " out_arr[i] = np.inf\n", + " else:\n", + " if not first:\n", + " for j in range(0, average_length):\n", + " s[0] += shared[tx * thread_tile + k + average_length - 1 - j]\n", + " s[0] = s[0] / np.float64(average_length)\n", + " out_arr[i] = s[0]\n", + " first = True\n", + " else:\n", + " s[0] = s[0] + (shared[tx * thread_tile + k + average_length - 1]\n", + " - shared[tx * thread_tile + k + average_length - 1 - average_length]) / np.float64(average_length)\n", + " \n", + " out_arr[i] = s[0]\n", + " \n", + " \n", + "gpu_in = df['in'].to_gpu_array()\n", + "gpu_out = df['out'].to_gpu_array()\n", + "start = time.time()\n", + "kernel1[(number_of_blocks,), (number_of_threads,)](gpu_in, gpu_out,\n", + " average_window, array_len)\n", + "cuda.synchronize()\n", + "end = time.time()\n", + "print('Numba with comipile time', end-start)\n", + " \n", + "start = time.time()\n", + "kernel1[(number_of_blocks,), (number_of_threads,)](gpu_in, gpu_out,\n", + " average_window, array_len)\n", + "cuda.synchronize()\n", + "end = time.time()\n", + "print('Numba without comipile time', end-start)\n", + " \n", + "pdf = pd.DataFrame()\n", + "pdf['in'] = np.arange(array_len, dtype=np.float64)\n", + "start = time.time()\n", + "pdf['out'] = pdf.rolling(average_window).mean()\n", + "end = time.time()\n", + "print('pandas time', end-start)\n", + " \n", + "assert(np.isclose(pdf.out.values[average_window:].mean(),\n", + " df.out.to_array()[average_window:].mean()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After this change, the computation time is reduced to 0.3s without kernel compilation time, we achieved a total of 6x speedup compared with the baseline." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion\n", + "\n", + "In this tutorial, we take advantage of CUDA programming model in the Numba library to do moving average computation. We show by using a few CUDA programming tricks, we can achieve **6x** speed up in moving average computations for long arrays.\n", + "\n", + "cuDF is a powerful tool for data scientists to use. It provides the high-level API that covers most of the use cases. However, it also exposes its low-level components. Those components including gpu_array and Numba integration make the cuDF library to be very flexible to process data in a customized way. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.8" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/BFS.ipynb b/cugraph/BFS.ipynb new file mode 100644 index 00000000..ccfffcba --- /dev/null +++ b/cugraph/BFS.ipynb @@ -0,0 +1,489 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Breadth First Search (BFS) \n", + "In this notebook, we will use cuGraph to compute the Breadth First Search path from a starting vertex to everyother vertex in our training dataset.\n", + "\n", + "Notebook Credits\n", + "* Original Authors: Bradley Rees and James Wyles\n", + "* Last Edit: 05/01/2019\n", + "\n", + "RAPIDS Versions: 0.7.0 \n", + "\n", + "Test Hardware\n", + "\n", + "* GP100 32G, CUDA 9,2\n", + "\n", + "\n", + "\n", + "## Introduction\n", + "\n", + "As the name implies, BFS traverses the given graph in a breadth first manner. Starting at a specified vertex, the algorithms iteratively searches neighboring vertices. \n", + "\n", + "\n", + "@see https://en.wikipedia.org/wiki/Breadth-first_search\n", + "\n", + "\n", + "To compute BFS in cuGraph use: __bfs(G, start_id)__\n", + "\n", + "* __G__: A cugraph.Graph object\n", + "* __start_id__ : the starting vertex ID\n", + "\n", + "Returns:\n", + "\n", + "* __df__: cudf.DataFrame with three names columns:\n", + " * df[\"vertex\"]: The vertex id.\n", + " * df[\"distance\"]: The distance to the starting vertex\n", + " * df[\"predecessor\"]: The vertex ID of the vertex that was used to reach this vertex\n", + "\n", + "\n", + "## cuGraph 0.7 Notice \n", + "cuGraph version 0.7 has some limitations:\n", + "* Only Int32 Vertex ID are supported\n", + "* Only float (FP32) edge data is supported\n", + "* Vertex numbering is assumed to start at zero\n", + "\n", + "These limitations are being addressed and will be fixed future versions. \n", + "These example notebooks will illustrate how to manipulate the data so that it comforms to the current limitations \n", + " \n", + "\n", + "\n", + "___" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Data\n", + "We will be using the Zachary Karate club dataset \n", + "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n", + "Anthropological Research 33, 452-473 (1977).*\n", + "\n", + "\n", + "![Karate Club](./img/zachary_black_lines.png)\n", + "\n", + "Our test data is small so that results can be visually verified" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "ename": "ModuleNotFoundError", + "evalue": "No module named 'libgdf_cffi'", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# First step is to import the needed libraries\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mcugraph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mcudf\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mcollections\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32mcugraph/jaccard/jaccard_wrapper.pyx\u001b[0m in \u001b[0;36minit cugraph\u001b[0;34m()\u001b[0m\n", + "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'libgdf_cffi'" + ] + } + ], + "source": [ + "# First step is to import the needed libraries\n", + "import cugraph\n", + "import cudf\n", + "from collections import OrderedDict" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# define a print path function that take the dataframe and a vertex ID\n", + "\n", + "def print_path(df, id):\n", + " \n", + " # Use the BFS predecessors and distance to trace the path \n", + " # from vertex id back to the starting vertex ( vertex 1 in this example)\n", + " dist = df['distance'][id]\n", + " lastVert = id\n", + " for i in range(dist):\n", + " nextVert = df['predecessor'][lastVert]\n", + " d = df['distance'][lastVert]\n", + " print(\"Vertex: \" + str(lastVert) + \" was reached from vertex \" + str(nextVert) + \n", + " \" and distance to start is \" + str(d) )\n", + " lastVert = nextVert" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Read the data using cuDF" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Read the data file\n", + "datafile='./data/karate-data.csv'\n", + "\n", + "cols = [\"src\", \"dst\"]\n", + "\n", + "dtypes = OrderedDict([\n", + " (\"src\", \"int32\"), \n", + " (\"dst\", \"int32\")\n", + " ])\n", + "\n", + "gdf = cudf.read_csv(datafile, names=cols, delimiter='\\t', dtype=list(dtypes.values()) )\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Let's look at the DataFrame. There should be two columns and 156 records\n", + "gdf" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
srcdst
012
113
214
315
416
\n", + "
" + ], + "text/plain": [ + " src dst\n", + "0 1 2\n", + "1 1 3\n", + "2 1 4\n", + "3 1 5\n", + "4 1 6" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Look at the first few data records - the output should be two colums src and dst\n", + "gdf.head().to_pandas()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As you can see in the output, the starting vertex ID is 1. For the BFS algorithm that is okay. \n", + "cuGraph will add an isolated vertex with an ID of zero. It will not be reachable from the test dataset " + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "# create a Graph \n", + "G = cugraph.Graph()\n", + "G.add_edge_list(gdf[\"src\"], gdf[\"dst\"])" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "# Call BFS on the graph starting from vertex 1\n", + "df = cugraph.bfs(G,1)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "vertex int32\n", + "distance int32\n", + "predecessor int32\n", + "dtype: object" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Let's take a looks at the structure of the returned dataframe\n", + "df.dtypes" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Vertex: 22 was reached from vertex 1 and distance to start is 1\n" + ] + } + ], + "source": [ + "print_path(df, 22)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Vertex: 30 was reached from vertex 33 and distance to start is 3\n", + "Vertex: 33 was reached from vertex 3 and distance to start is 2\n", + "Vertex: 3 was reached from vertex 1 and distance to start is 1\n" + ] + } + ], + "source": [ + "print_path(df, 30)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Since we can see in graph illustraion above that vertex 17 is at the edge of the graph, let's run BFS with that as the startring vertex" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "# Call BFS on the graph starting from vertex 17\n", + "df2 = cugraph.bfs(G,17)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "2147483647" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Print the max distance\n", + "df2[\"distance\"].max()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Notice that max returned an unexpected value. That is becouse the isoluated vertex, 0, is unreachable.\n", + "Whenever a graph contains disjointed components, the distance to the unconnected vertices will always be max_int" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "2147483647" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df2[\"distance\"][0]" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "# drop all large distances \n", + "exp=\"distance < 100\"\n", + "df3 = df2.query(exp)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "5" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Print the max distance\n", + "df3[\"distance\"].max()" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Vertex: 30 was reached from vertex 33 and distance to start is 5\n", + "Vertex: 33 was reached from vertex 3 and distance to start is 4\n", + "Vertex: 3 was reached from vertex 1 and distance to start is 3\n", + "Vertex: 1 was reached from vertex 6 and distance to start is 2\n", + "Vertex: 6 was reached from vertex 17 and distance to start is 1\n" + ] + } + ], + "source": [ + "# Print path to vertex 30\n", + "print_path(df2, 30)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/Louvain.ipynb b/cugraph/Louvain.ipynb new file mode 100644 index 00000000..99beac27 --- /dev/null +++ b/cugraph/Louvain.ipynb @@ -0,0 +1,340 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Louvain Community Detection\n", + "\n", + "\n", + "In this notebook, we will use cuGraph to identify the cluster in a test graph using the Louvain algorithm \n", + "\n", + "Notebook Credits\n", + "* Original Authors: Bradley Rees and James Wyles\n", + "* Last Edit: 04/30/2019\n", + "\n", + "RAPIDS Versions: 0.7.0\n", + "\n", + "Test Hardware\n", + "* GP100 32G, CUDA 9,2\n", + "\n", + "\n", + "\n", + "## Introduction\n", + "\n", + "The Louvain method of community detection is a greedy heirarical clsutering algorithm which seeks to optimize Modularity as it progresses. Louvain starts with each vertex in its own clusters and iteratively merges groups using graph contraction. \n", + "\n", + "For a detailed description of the algorithm see: https://en.wikipedia.org/wiki/Louvain_Modularity\n", + "\n", + "It takes as input a cugraph.Graph object and returns as output a \n", + "cudf.Dataframe object with the id and assigned partition for each \n", + "vertex as well as the final modularity score\n", + "\n", + "\n", + "To compute the Louvain cluster in cuGraph use:
\n", + "\n", + "**nvLouvain(G)**\n", + "* __G__: A cugraph.Graph object\n", + "\n", + "Returns:\n", + "\n", + "* tupal __lovain dataframe__ and __modularity__\n", + "\n", + "\n", + "* __louvain__: cudf.DataFrame with two names columns:\n", + " * louvain[\"vertex\"]: The vertex id.\n", + " * louvain[\"partition\"]: The assigned partition.\n", + " \n", + "* __modularity__ : the overall modularity of the graph\n", + "\n", + "All vertices with the same partition ID are in the same cluster\n", + "\n", + "\n", + "## cuGraph 0.7 Notice \n", + "cuGraph version 0.7 has some limitations:\n", + "* Only Int32 Vertex ID are supported\n", + "* Only float (FP32) edge data is supported\n", + "* Vertex numbering is assumed to start at zero\n", + "\n", + "These limitations are being addressed and will be fixed future versions. \n", + "These example notebooks will illustrate how to manipulate the data so that it comforms to the current limitations \n", + "\n", + "A new renumbering feature is being worked and will be reflected in updated notebooks for the next release.\n", + "\n", + "## References\n", + "\n", + "* Blondel, V. D., Guillaume, J.-L., Lambiotte, R., and Lefebvre, E. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment 2008, 10 (2008), P10008.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Data\n", + "We will be using the Zachary Karate club dataset \n", + "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n", + "Anthropological Research 33, 452-473 (1977).*\n", + "\n", + "\n", + "![Karate Club](./img/zachary_black_lines.png)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Prep" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "# Import needed libraries\n", + "import cugraph\n", + "import cudf\n", + "import numpy as np\n", + "from collections import OrderedDict" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Read data using cuDF" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# Test file \n", + "datafile='./data//karate-data.csv'\n", + "\n", + "# Read the data file\n", + "cols = [\"src\", \"dst\"]\n", + "\n", + "dtypes = OrderedDict([\n", + " (\"src\", \"int32\"), \n", + " (\"dst\", \"int32\")\n", + " ])\n", + "\n", + "gdf = cudf.read_csv(datafile, names=cols, delimiter='\\t', dtype=list(dtypes.values()) )" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Louvain is dependent on vertex ID starting at zero\n", + "gdf[\"src_0\"] = gdf[\"src\"] - 1\n", + "gdf[\"dst_0\"] = gdf[\"dst\"] - 1" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# The algorithm also requires that there are vertex weights. Just use 1.0 \n", + "gdf[\"data\"] = 1.0" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "src int32\n", + "dst int32\n", + "src_0 int32\n", + "dst_0 int32\n", + "data float64\n", + "dtype: object" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# just for fun, let's look at the data types in the dataframe\n", + "gdf.dtypes" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "# create a Graph \n", + "G = cugraph.Graph()\n", + "G.add_edge_list(gdf[\"src_0\"], gdf[\"dst_0\"], gdf[\"data\"])" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "# Call Louvain on the graph\n", + "df, mod = cugraph.nvLouvain(G) " + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Modularity was 0.4027777777777778\n", + "\n" + ] + } + ], + "source": [ + "# Print the modularity score\n", + "print('Modularity was {}'.format(mod))\n", + "print()" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "vertex int32\n", + "partition int32\n", + "dtype: object" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.dtypes" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "# How many partitions where found\n", + "part_ids = df[\"partition\"].unique()" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Found 4 partition\n" + ] + } + ], + "source": [ + "print(str(len(part_ids)) + \" partition detected\")" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Partition 0:\n", + "[1, 2, 3, 4, 8, 10, 12, 13, 14, 18, 20, 22]\n", + "Partition 1:\n", + "[5, 6, 7, 11, 17]\n", + "Partition 2:\n", + "[9, 15, 16, 19, 21, 23, 27, 29, 30, 31, 32, 33, 34]\n", + "Partition 3:\n", + "[24, 25, 26, 28]\n" + ] + } + ], + "source": [ + "for p in range(len(part_ids)):\n", + " part = []\n", + " for i in range(len(df)):\n", + " if (df['partition'][i] == p):\n", + " part.append(df['vertex'][i] +1)\n", + " print(\"Partition \" + str(p) + \":\")\n", + " print(part)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/Pagerank.ipynb b/cugraph/Pagerank.ipynb new file mode 100644 index 00000000..c60c33cb --- /dev/null +++ b/cugraph/Pagerank.ipynb @@ -0,0 +1,434 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# PageRank\n", + "\n", + "In this notebook, we will use both NetworkX and cuGraph to compute the PageRank of each vertex in our test dataset. The NetworkX and cuGraph processes will be interleaved so that each step can be compared.\n", + "\n", + "Notebook Credits\n", + "* Original Authors: Bradley Rees and James Wyles\n", + "* Last Edit: 04/30/2019\n", + "\n", + "RAPIDS Versions: 0.7.0 \n", + "\n", + "Test Hardware\n", + "\n", + "* GP100 32G, CUDA 9.2\n", + "\n", + "\n", + "## Introduction\n", + "Pagerank is measure of the relative importance, also called centrality, of a vertex based on the relative importance of it's neighbors. PageRank was developed by Google and is (was) used to rank it's search results. PageRank uses the connectivity information of a graph to rank the importance of each vertex. \n", + "\n", + "See [Wikipedia](https://en.wikipedia.org/wiki/PageRank) for more details on the algorithm.\n", + "\n", + "To compute the Pagerank scores for a graph in cuGraph we use:
\n", + "\n", + "**cugraph.pagerank(G,alpha=0.85, max_iter=100, tol=1.0e-5)**\n", + "* __G__: cugraph.Graph object\n", + "* __alpha__: float, The damping factor represents the probability to follow an outgoing edge. default is 0.85\n", + "* __max_iter__: int, The maximum number of iterations before an answer is returned. This can be used to limit the execution time and do an early exit before the solver reaches the convergence tolerance. If this value is lower or equal to 0 cuGraph will use the default value, which is 100\n", + "* __tol__: float, Set the tolerance the approximation, this parameter should be a small magnitude value. The lower the tolerance the better the approximation. If this value is 0.0f, cuGraph will use the default value which is 0.00001. Setting too small a tolerance can lead to non-convergence due to numerical roundoff. Usually values between 0.01 and 0.00001 are acceptable.\n", + "\n", + "Returns:\n", + "* __df__: a cudf.DataFrame object with two columns:\n", + " * df['vertex']: The vertex identifier for the vertex\n", + " * df['pagerank']: The pagerank score for the vertex\n", + " \n", + "

\n", + "## cuGraph 0.7 Notice \n", + "cuGraph version 0.7 has some limitations:\n", + "* Only Int32 Vertex ID are supported\n", + "* Only float (FP32) edge data is supported\n", + "* Vertex numbering is assumed to start at zero\n", + "\n", + "These limitations are being addressed and will be fixed future versions. \n", + "These example notebooks will illustrate how to manipulate the data so that it comforms to the current limitations " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Data\n", + "We will be using the Zachary Karate club dataset \n", + "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n", + "Anthropological Research 33, 452-473 (1977).*\n", + "\n", + "\n", + "![Karate Club](./img/zachary_black_lines.png)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Prep" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# The notebook compares cuGraph to NetworkX, \n", + "# therefore there some additional non-RAPIDS python libraries need to be installed. \n", + "# Please run this cell if you need the additional libraries\n", + "!pip install networkx\n", + "!pip install scipy" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Import needed libraries\n", + "import cugraph\n", + "import cudf\n", + "from collections import OrderedDict" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# NetworkX libraries\n", + "import networkx as nx\n", + "from scipy.io import mmread" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Some Prep" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# define the parameters \n", + "max_iter = 100 # The maximum number of iterations\n", + "tol = 0.00001 # tolerance\n", + "alpha = 0.85 # alpha" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Define the path to the test data \n", + "datafile='./data/karate-data.csv'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "# NetworkX" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Read the data, this also created a NetworkX Graph \n", + "file = open(datafile, 'rb')\n", + "Gnx = nx.read_edgelist(file)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pr_nx = nx.pagerank(Gnx, alpha=alpha, max_iter=max_iter, tol=tol)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pr_nx" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Running NetworkX is that easy. \n", + "Let's seet how that compares to cuGraph\n", + "\n", + "----" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# cuGraph" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Read in the data - GPU\n", + "cuGraph depends on cuDF for data loading and the initial Dataframe creation\n", + "\n", + "The data file contains an edge list, which represents the connection of a vertex to another. The `source` to `destination` pairs is in what is known as Coordinate Format (COO). In this test case, the data is just two columns. However a third, `weight`, column is also possible" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Test file \n", + "datafile='./data/karate-data.csv'\n", + "\n", + "# Read the data file\n", + "cols = [\"src\", \"dst\"]\n", + "\n", + "dtypes = OrderedDict([\n", + " (\"src\", \"int32\"), \n", + " (\"dst\", \"int32\")\n", + " ])\n", + "\n", + "gdf = cudf.read_csv(datafile, names=cols, delimiter='\\t', dtype=list(dtypes.values()) )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Create a Graph " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# create a Graph using the source (src) and destination (dst) vertex pairs from the Dataframe \n", + "G = cugraph.Graph()\n", + "G.add_edge_list(gdf[\"src\"], gdf[\"dst\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Call the PageRank algorithm" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Call cugraph.pagerank to get the pagerank scores\n", + "gdf_page = cugraph.pagerank(G)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "_It was that easy!_ \n", + "Compared to NetworkX, the cuGraph data loading might have been more steps, but using cuDF allows for a wider range of data to be loaded. \n", + "\n", + "\n", + "----\n", + "\n", + "Let's now look at the results" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Find the most important vertex using the scores\n", + "# This methods should only be used for small graph\n", + "bestScore = gdf_page['pagerank'][0]\n", + "bestVert = gdf_page['vertex'][0]\n", + "\n", + "for i in range(len(gdf_page)):\n", + " if gdf_page['pagerank'][i] > bestScore:\n", + " bestScore = gdf_page['pagerank'][i]\n", + " bestVert = gdf_page['vertex'][i]\n", + " \n", + "print(\"Best vertex is \" + str(bestVert) + \" with score of \" + str(bestScore))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The top PageRank vertex and socre match what was found by NetworkX" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# A better way to do that would be to find the max and then use that values in a query\n", + "pr_max = gdf_page['pagerank'].max()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def print_pagerank_threshold(_df, t=0) :\n", + " filtered = _df.query('pagerank >= @t')\n", + " \n", + " for i in range(len(filtered)):\n", + " print(\"Best vertex is \" + str(filtered['vertex'][i]) + \n", + " \" with score of \" + str(filtered['pagerank'][i])) " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print_pagerank_threshold(gdf_page, pr_max)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "----\n", + "\n", + "a PageRank score of _0.10047_ is very low, which can be an indication that there is no more central vertex than any other. Rather than just looking at the top score, let's look at the top three vertices and see if there are any insights that can be inferred. \n", + "\n", + "Since this is a very small graph, let's just sort and get the first three records" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "sort_pr = gdf_page.sort_values('pagerank', ascending=False)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "sort_pr.head(3).to_pandas()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Going back and looking at the graph with the top three vertices highlighted (illustration below) it is easy to see that the top scoring vertices also appear to be the vertices with the most connections. \n", + "Let's look at sorted list of degrees" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "d = G.degree()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# divide the degree by two since this is an undirected graph\n", + "d['degree'] = d['degree'] / 2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "d.sort_values('degree', ascending=False).head(3).to_pandas()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/README.md b/cugraph/README.md new file mode 100644 index 00000000..74564c3f --- /dev/null +++ b/cugraph/README.md @@ -0,0 +1,73 @@ + + +# cuGraph Notebooks + +![GraphAnalyticsFigure](img/GraphAnalyticsFigure.jpg) + +This repository contains a collection of Jupyter Notebooks that outline how to run various cuGraph analytics. The notebooks do not address a complete data science problem. The notebooks are simply examples of how to run the graph analytics. Manipulation of the data before or after the graph analytic is not covered here. Extended, more problem focused, notebooks are being created and available https://github.com/rapidsai/notebooks-extended + + + + + +## Summary + +| Notebook | Description | +| ------------------- | ------------------------------------------------------------ | +| BFS | Compute the Breadth First Search path from a starting vertex to every other vertex in a graph | +| SSSP | Single Source Shortest Path - compute the shortest path from a starting vertex to every other vertex | +| Vertex Similarity | Compute vertex similarity score using both:
Jaccard Similarity
Overlap Coefficient | +| Spectral-Clustering | Identify clusters in a graph using Spectral Clustering with both
Balanced Cut
Modularity | +| Louvain | Identify clusters in a graph using the Louvain algorithm | +| Weighted Jaccard | Computer the Jaccard Similarity using vertex weights | +| Triangle Counting | Count the number of Triangle in a graph | +| Pagerank | Compute the PageRank of every vertex in a graph | +| Renumbering | Renumber the vertex IDs in a graph | + + + + + +## Requirements + +Running the example in these notebooks requires: + +* The latest version of RAPIDS with cuGraph. + * Download via Docker, Conda, or PIP +* cuGraph is dependent on the latest version of cuDF. Please install all components of RAPIDS +* Python 3.6+ +* A system with an NVIDIA GPU: Pascal architecture or better +* CUDA 9.2+ +* NVIDIA driver 396.44+ + + + +#### Notebook Credits + +- Original Authors: Bradley Rees +- Last Edit: 05/10/2019 + +RAPIDS Versions: 0.7.0 + +Test Hardware + +- GV100 32G, CUDA 9,2 + + + +##### Copyright + +Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. + + + + + +![RAPIDS](img/rapids_logo.png) + diff --git a/cugraph/Renumber.ipynb b/cugraph/Renumber.ipynb new file mode 100644 index 00000000..6628e68c --- /dev/null +++ b/cugraph/Renumber.ipynb @@ -0,0 +1,583 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Renumbering Test\n", + "\n", + "Demonstrate creating a graph with renumbering.\n", + "\n", + "Most cugraph algorithms operate on a CSR representation of a graph. A CSR representation requires an indices array that is as long as the number of edges and an offsets array that is as 1 more than the largest vertex id. This makes the memory utilization entirely dependent on the size of the largest vertex id. For data sets that have a sparse range of vertex ids, the size of the CSR can be unnecessarily large. It is easy to construct an example where the amount of memory required for the offsets array will exceed the amount of memory in the GPU (not to mention the performance cost of having a large number of offsets that are empty but still have to be read to be skipped).\n", + "\n", + "The cugraph renumbering feature allows us to take two columns of any integer type and translate them into a densely packed contiguous array numbered from 0 to (num_unique_values - 1). These renumbered vertices can be used to create a graph much more efficiently.\n", + "\n", + "Another of the features of the renumbering function is that it can take vertex ids that are 64-bit values and map them down into a range that fits into 32-bit integers. The current cugraph algorithms are limited to 32-bit signed integers as vertex ids. and the renumbering feature will allow the caller to translate ids that are 64-bit into a densly packed 32-bit array of ids that can be used in cugraph algorithms. Note that if there are more than 2^31 - 1 unique vertex ids then the renumber method will fail with an error indicating that there are too many vertices to renumber into a 32-bit signed integer.\n", + "\n", + "Note that this version (0.7) is limited to integer types. The intention is to extend the renumbering function to be able to handle strings and other types." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First step is to import the needed libraries" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import cugraph\n", + "import cudf\n", + "import socket\n", + "import struct\n", + "import pandas as pd\n", + "import numpy as np\n", + "import networkx as nx\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Create some test data\n", + "\n", + "This creates a small circle using some ipv4 addresses, storing the columns in a GPU data frame.\n", + "\n", + "The current version of renumbering operates only on integer types, so we translate the ipv4 strings into 64 bit integers." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "sources came from: ['192.168.1.1', '172.217.5.238', '216.228.121.209', '192.16.31.23']\n", + " sources as int = [3232235777, 2899903982, 3638852049, 3222282007]\n", + "destinations came from: ['172.217.5.238', '216.228.121.209', '192.16.31.23', '192.168.1.1']\n", + " destinations as int = [2899903982, 3638852049, 3222282007, 3232235777]\n" + ] + } + ], + "source": [ + "source_list = [ '192.168.1.1', '172.217.5.238', '216.228.121.209', '192.16.31.23' ]\n", + "dest_list = [ '172.217.5.238', '216.228.121.209', '192.16.31.23', '192.168.1.1' ]\n", + "source_as_int = [ struct.unpack('!L', socket.inet_aton(x))[0] for x in source_list ]\n", + "dest_as_int = [ struct.unpack('!L', socket.inet_aton(x))[0] for x in dest_list ]\n", + "\n", + "\n", + "print(\"sources came from: \" + str([ socket.inet_ntoa(struct.pack('!L', x)) for x in source_as_int ]))\n", + "print(\" sources as int = \" + str(source_as_int))\n", + "print(\"destinations came from: \" + str([ socket.inet_ntoa(struct.pack('!L', x)) for x in dest_as_int ]))\n", + "print(\" destinations as int = \" + str(dest_as_int))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Create our GPU data frame" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
source_as_intdest_as_int
032322357772899903982
128999039823638852049
236388520493222282007
332222820073232235777
\n", + "
" + ], + "text/plain": [ + " source_as_int dest_as_int\n", + "0 3232235777 2899903982\n", + "1 2899903982 3638852049\n", + "2 3638852049 3222282007\n", + "3 3222282007 3232235777" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = pd.DataFrame({\n", + " 'source_list': source_list,\n", + " 'dest_list': dest_list,\n", + " 'source_as_int': source_as_int,\n", + " 'dest_as_int': dest_as_int\n", + " })\n", + "\n", + "gdf = cudf.DataFrame.from_pandas(df[['source_as_int', 'dest_as_int']])\n", + "\n", + "gdf.to_pandas()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Run renumbering\n", + "\n", + "The current version of renumbering takes a column of source vertex ids and a column of dest vertex ids. As mentioned above, these must be integer columns.\n", + "\n", + "Output from renumbering is 3 cudf.Series structures representing the renumbered sources, the renumbered destinations and the numbering map which maps the new ids back to the original ids.\n", + "\n", + "In this case,\n", + " * gdf['source_as_int'] is a column of type int64\n", + " * gdf['dest_as_int'] is a column of type int64\n", + " * src_r will be a series of type int32 (we translate back to 32-bit integers)\n", + " * dst_r will be a series of type int32\n", + " * numbering will be a series of type int64 that translates the elements of src and dst back to their original 64-bit values\n", + " \n", + "Note that because the renumbering translates us to 32-bit integers, if there are more than 2^31 - 1 unique 64-bit values in the source/dest passed into renumbering this would exceed the size of the 32-bit integers so you will get an error from the renumber call. " + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
source_as_intdest_as_intoriginal idsrc_renumbereddst_renumbered
032322357772899903982363885204912
128999039823638852049323223577720
236388520493222282007289990398203
332222820073232235777322228200731
\n", + "
" + ], + "text/plain": [ + " source_as_int dest_as_int original id src_renumbered dst_renumbered\n", + "0 3232235777 2899903982 3638852049 1 2\n", + "1 2899903982 3638852049 3232235777 2 0\n", + "2 3638852049 3222282007 2899903982 0 3\n", + "3 3222282007 3232235777 3222282007 3 1" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "G = cugraph.Graph()\n", + "\n", + "src_r, dst_r, numbering = G.renumber(gdf['source_as_int'], gdf['dest_as_int'])\n", + "\n", + "gdf.add_column(\"original id\", numbering)\n", + "gdf.add_column(\"src_renumbered\", src_r)\n", + "gdf.add_column(\"dst_renumbered\", dst_r)\n", + "\n", + "gdf.to_pandas()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Data types\n", + "\n", + "Just to confirm, the data types of the renumbered columns should be int32, the original data should be int64, the numbering map needs to be int64 since the values it contains map to the original int64 types." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "source_as_int int64\n", + "dest_as_int int64\n", + "original id int64\n", + "src_renumbered int32\n", + "dst_renumbered int32\n", + "dtype: object" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "gdf.dtypes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Quick verification\n", + "\n", + "To understand the renumbering, here's a block of verification logic. In the renumbered series we created a new id for each unique value in the original series. The numbering map identifies that mapping. For any vertex id X in the new numbering, numbering[X] should refer to the original value." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " 0: (3232235777,2899903982), renumbered: (1,2), translate back: (3232235777,2899903982)\n", + " 1: (2899903982,3638852049), renumbered: (2,0), translate back: (2899903982,3638852049)\n", + " 2: (3638852049,3222282007), renumbered: (0,3), translate back: (3638852049,3222282007)\n", + " 3: (3222282007,3232235777), renumbered: (3,1), translate back: (3222282007,3232235777)\n" + ] + } + ], + "source": [ + "for i in range(len(src_r)):\n", + " print(\" \" + str(i) +\n", + " \": (\" + str(source_as_int[i]) + \",\" + str(dest_as_int[i]) +\")\"\n", + " \", renumbered: (\" + str(src_r[i]) + \",\" + str(dst_r[i]) +\")\"\n", + " \", translate back: (\" + str(numbering[src_r[i]]) + \",\" + str(numbering[dst_r[i]]) +\")\"\n", + " )\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Now let's do some graph things...\n", + "\n", + "To start, let's run page rank. Not particularly interesting on our circle, since everything should have an equal rank." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
vertexpagerankoriginal id
000.253638852049
110.253232235777
220.252899903982
330.253222282007
\n", + "
" + ], + "text/plain": [ + " vertex pagerank original id\n", + "0 0 0.25 3638852049\n", + "1 1 0.25 3232235777\n", + "2 2 0.25 2899903982\n", + "3 3 0.25 3222282007" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "G.add_edge_list(src_r, dst_r)\n", + "\n", + "pr = cugraph.pagerank(G)\n", + "\n", + "pr.add_column(\"original id\", numbering)\n", + "pr.to_pandas()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Try to run jaccard\n", + "\n", + "Not at all an interesting result, but it demonstrates a more complicated case. Jaccard returns a coefficient for each edge. In order to show the original ids we need to add columns to the data frame for each column that contains one of renumbered vertices. In this case, the columns source and destination contain renumbered vertex ids." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
sourcedestinationjaccard_coefforiginal_sourceoriginal_destination
0030.0216.228.121.209192.16.31.23
1120.0192.168.1.1172.217.5.238
2200.0172.217.5.238216.228.121.209
3310.0192.16.31.23192.168.1.1
\n", + "
" + ], + "text/plain": [ + " source destination jaccard_coeff original_source original_destination\n", + "0 0 3 0.0 216.228.121.209 192.16.31.23\n", + "1 1 2 0.0 192.168.1.1 172.217.5.238\n", + "2 2 0 0.0 172.217.5.238 216.228.121.209\n", + "3 3 1 0.0 192.16.31.23 192.168.1.1" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "jac = cugraph.jaccard(G)\n", + "\n", + "jac.add_column(\"original_source\",\n", + " [ socket.inet_ntoa(struct.pack('!L', numbering[x])) for x in jac['source'] ])\n", + "\n", + "jac.add_column(\"original_destination\",\n", + " [ socket.inet_ntoa(struct.pack('!L', numbering[x])) for x in jac['destination'] ])\n", + "\n", + "jac.to_pandas()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/SSSP.ipynb b/cugraph/SSSP.ipynb new file mode 100644 index 00000000..08fced8a --- /dev/null +++ b/cugraph/SSSP.ipynb @@ -0,0 +1,347 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Single Source Shortest Path (SSSP)\n", + "\n", + "In this notebook, we will use cuGraph to compute the shortest path from a starting vertex to everyother vertex in our training dataset.\n", + "\n", + "Notebook Credits\n", + "* Original Authors: Bradley Rees and James Wyles\n", + "* Last Edit: 06/25/2019\n", + "\n", + "RAPIDS Versions: 0.8.0 \n", + "\n", + "Test Hardware\n", + "\n", + "* GP100 32G, CUDA 9,2\n", + "\n", + "\n", + "\n", + "\n", + "## Introduction\n", + "\n", + "Single source shortest path computes the shortest paths from the given starting vertex to all other reachable vertices. \n", + "\n", + "To compute SSSP for a graph in cuGraph we use:\n", + "**cugraph.sssp(G, source)**\n", + "\n", + "Input\n", + "* __G__: cugraph.Graph object\n", + "* __source__: int, Index of the source vertex\n", + "\n", + "Returns \n", + "* __df__: a cudf.DataFrame object with two columns:\n", + " * df['vertex']: The vertex identifier for the vertex\n", + " * df['distance']: The computed distance from the source vertex to this vertex\n", + " * df['predecessor']: The predecessor vertex along this paths. Allows paths to be recreated\n", + "\n", + "\n", + "## cuGraph 0.7 Notice \n", + "cuGraph version 0.7 has some limitations:\n", + "* Only Int32 Vertex ID are supported\n", + "* Only float (FP32) edge data is supported\n", + "* Vertex numbering is assumed to start at zero\n", + "\n", + "These limitations are being addressed and will be fixed future versions. \n", + "These example notebooks will illustrate how to manipulate the data so that it comforms to the current limitations \n", + " \n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Data\n", + "We will be using the Zachary Karate club dataset \n", + "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n", + "Anthropological Research 33, 452-473 (1977).*\n", + "\n", + "\n", + "![Karate Club](./img/zachary_black_lines.png)\n", + "\n", + "This is a small graph which allows for easy visual inspection to validate results. \n", + "__Note__: The Karate dataset starts with vertex ID 1 which the cuGraph analytics assume a zero-based starting ID. " + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "# Import needed libraries\n", + "import cudf\n", + "import cugraph" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Read the data and adjust the vertex IDs" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# Test file - using the clasic Karate club dataset. \n", + "datafile='./data/karate-data.csv'" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Read the data file\n", + "gdf = cudf.read_csv(datafile, names=[\"src\", \"dst\"], delimiter='\\t', dtype=[\"int32\", \"int32\"] )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since the data starts with vertex ID of 1, the code will add a disconnected vertex (ID 0) that \n", + "is unreachable (no path from vertices in data to vertex 0). In earlier version of cuGraph, we shifted the data\n", + "so that it started to avoid that case. Starting with release 0.8, a new feature to filter unreachable vertices from the results has been added" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
srcdst
012
113
214
315
416
\n", + "
" + ], + "text/plain": [ + " src dst\n", + "0 1 2\n", + "1 1 3\n", + "2 1 4\n", + "3 1 5\n", + "4 1 6" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "gdf.head().to_pandas()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Create a Graph and call SSSP" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "# create a Graph \n", + "G = cugraph.Graph()\n", + "G.add_edge_list(gdf[\"src\"], gdf[\"dst\"])" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "# Call cugraph.sssp to get the distances from vertex 1:\n", + "df = cugraph.sssp(G, 1)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "# Removed the unreachable vertices\n", + "clean_df = cugraph.filter_unreachable(df)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Farthest distance from source is 3.0\n", + "and there are 8 vertices of that distance\n" + ] + } + ], + "source": [ + "# Find the farthest vertex from the source using the distances:\n", + "# __note__ the vertex ID is shifted back to 1-based so that it can be seen on picture above\n", + "longest_distance = clean_df['distance'].max()\n", + "ldf = df.query('distance == @longest_distance')\n", + "\n", + "print(\"Farthest distance from source is \" + str(longest_distance) )\n", + "print(\"and there are \" + str(len(ldf)) + \" vertices of that distance\")" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "# Sort the data to make allow rows to be referenced by vertex IDs\n", + "df = df.sort_values(by='vertex')" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(0) path: [1, 32, 34, 15]\n", + "(1) path: [1, 32, 34, 16]\n", + "(2) path: [1, 32, 34, 19]\n", + "(3) path: [1, 32, 34, 21]\n", + "(4) path: [1, 32, 34, 23]\n", + "(5) path: [1, 32, 34, 24]\n", + "(6) path: [1, 32, 34, 27]\n", + "(7) path: [1, 32, 34, 30]\n" + ] + } + ], + "source": [ + "# Print the paths\n", + "# Not using the filterred dataframe to ensure that vertex IDs match row IDs\n", + "for i in range(len(ldf)) :\n", + " v = ldf['vertex'][i] \n", + " d = int(df['distance'][v])\n", + " \n", + " path = [None] * ( int(longest_distance) + 1)\n", + " path[d] = v\n", + " \n", + " while d > 0 :\n", + " v = df['predecessor'][v]\n", + " d = int(df['distance'][v])\n", + " path[d] = v\n", + " \n", + " print( \"(\" + str(i) + \") path: \" + str(path))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are a number of vertices with the same distance of 3" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/Spectral-Clustering.ipynb b/cugraph/Spectral-Clustering.ipynb new file mode 100644 index 00000000..1c9ab16d --- /dev/null +++ b/cugraph/Spectral-Clustering.ipynb @@ -0,0 +1,532 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Spectral Clustering \n", + "\n", + "In this notebook, we will use cuGraph to identify the cluster in a test graph using Spectral Clustering with both the (A) Balance Cut metric, and (B) the Modularity Maximization metric\n", + "\n", + "\n", + "Notebook Credits\n", + "* Original Authors: Bradley Rees and James Wyles\n", + "* Last Edit: 05/03/2019\n", + "\n", + "RAPIDS Versions: 0.7.0\n", + "\n", + "Test Hardware\n", + "* GP100 32G, CUDA 9,2\n", + "\n", + "\n", + "## Introduction\n", + "\n", + "Spectral clustering uses the eigenvectors of a Laplacian of the input graph to find a given number of clusters which satisfy a given quality metric. Balanced Cut and Modularity Maximization are such quality metrics. \n", + "\n", + "@See: https://en.wikipedia.org/wiki/Spectral_clustering\n", + "\n", + "To perform spectral clustering using the balanced cut metric in cugraph use:\n", + "\n", + "__cugraph.spectralBalancedCutClustering(G, num_clusters, num_eigen_vects)__\n", + "
or
\n", + "__cugraph.spectralModularityMaximizationClustering(G, num_clusters, num_eigen_vects)__\n", + "\n", + "\n", + "\n", + "Input\n", + "* __G__: A cugraph.Graph object\n", + "* __num_clusters__: The number of clusters to find\n", + "* __num_eig__: (optional) The number of eigenvectors to use\n", + "\n", + "Returns\n", + "* __df__: cudf.DataFrame with two names columns:\n", + " * df[\"vertex\"]: The vertex id.\n", + " * df[\"cluster\"]: The assigned partition.\n", + "\n", + "\n", + "## cuGraph 0.7 Notice \n", + "cuGraph version 0.7 has some limitations:\n", + "* Only Int32 Vertex ID are supported\n", + "* Only float (FP32) edge data is supported\n", + "* Vertex numbering is assumed to start at zero\n", + "\n", + "These limitations are being addressed and will be fixed future versions. \n", + "These example notebooks will illustrate how to manipulate the data so that it comforms to the current limitations \n", + "\n", + "\n", + "----" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Data\n", + "We will be using the Zachary Karate club dataset \n", + "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n", + "Anthropological Research 33, 452-473 (1977).*\n", + "\n", + "\n", + "![Karate Club](./img/zachary_black_lines.png)\n", + "\n", + "Zachary used a min-cut flow model to partition the graph into two clusters, shown by the circles and squares. Zarchary wanted just two cluster based on a conflict that caused the Karate club to break into two separate clubs. Many social network clustering methods identify more that two social groups in the data." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "# Import needed libraries\n", + "import cugraph\n", + "import cudf\n", + "import numpy as np\n", + "from collections import OrderedDict" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Read the CSV datafile using cuDF" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# Test file \n", + "datafile='./data/karate-data.csv'\n", + "\n", + "# Read the data file\n", + "cols = [\"src\", \"dst\"]\n", + "\n", + "dtypes = OrderedDict([\n", + " (\"src\", \"int32\"), \n", + " (\"dst\", \"int32\")\n", + " ])\n", + "\n", + "gdf = cudf.read_csv(datafile, names=cols, delimiter='\\t', dtype=list(dtypes.values()) )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Adjusting the vertex ID\n", + "Let's adjust all the vertex IDs to be zero based. We are going to do this by adding two new columns with the adjusted IDs" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "gdf[\"src\"] = gdf[\"src\"] - 1\n", + "gdf[\"dst\"] = gdf[\"dst\"] - 1" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# The algorithm requires that there are edge weights. In this case all the weights are being ste to 1\n", + "gdf[\"data\"] = cudf.Series(np.ones(len(gdf), dtype=np.float32))" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
srcdstdata
0011.0
1021.0
2031.0
3041.0
4051.0
\n", + "
" + ], + "text/plain": [ + " src dst data\n", + "0 0 1 1.0\n", + "1 0 2 1.0\n", + "2 0 3 1.0\n", + "3 0 4 1.0\n", + "4 0 5 1.0" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Look at the first few data records - the output should be two colums src and dst\n", + "gdf.head().to_pandas()" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "src int32\n", + "dst int32\n", + "data float32\n", + "dtype: object" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# verify data type\n", + "gdf.dtypes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Everything looks good, we can now create a graph" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "# create a CuGraph \n", + "G = cugraph.Graph()\n", + "G.add_edge_list(gdf[\"src\"], gdf[\"dst\"], gdf[\"data\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "----\n", + "#### Define and print function, but adjust vertex ID so that they match the illustration" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "def print_cluster(_df, id):\n", + " \n", + " _f = _df.query('cluster == @id')\n", + " \n", + " part = []\n", + " for i in range(len(_f)):\n", + " part.append(_f['vertex'][i] + 1)\n", + " print(part)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "----\n", + "#### Using Balanced Cut" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "# Call spectralBalancedCutClustering on the graph for 3 clusters\n", + "# using 3 eigenvectors:\n", + "bc_gdf = cugraph.spectralBalancedCutClustering(G, 3, num_eigen_vects=3)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "19.0" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check the edge cut score for the produced clustering\n", + "score = cugraph.analyzeClustering_edge_cut(G, 3, bc_gdf['cluster'])\n", + "score" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[15, 16, 19, 20, 21, 23, 27]\n" + ] + } + ], + "source": [ + "# See which nodes are in cluster 0:\n", + "print_cluster(bc_gdf, 0)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[25, 26]\n" + ] + } + ], + "source": [ + "# See which nodes are in cluster 1:\n", + "print_cluster(bc_gdf, 1)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 22, 24, 28, 29, 30, 31, 32, 33, 34]\n" + ] + } + ], + "source": [ + "# See which nodes are in cluster 2:\n", + "print_cluster(bc_gdf, 2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "----\n", + "#### Modularity Maximization\n", + "Let's now look at the clustering using the modularity maximization metric" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "# Call spectralModularityMaximizationClustering on the graph for 3 clusters\n", + "# using 3 eigenvectors:\n", + "mm_gdf = cugraph.spectralModularityMaximizationClustering(G, 3, num_eigen_vects=3)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.3579881489276886" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check the modularity score for the produced clustering\n", + "score = cugraph.analyzeClustering_modularity(G, 3, mm_gdf['cluster'])\n", + "score" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[9, 10, 15, 16, 19, 21, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34]\n" + ] + } + ], + "source": [ + "# See which nodes are in cluster 0:\n", + "print_cluster(mm_gdf, 0)" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[17]\n" + ] + } + ], + "source": [ + "print_cluster(mm_gdf, 1)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 18, 20, 22]\n" + ] + } + ], + "source": [ + "print_cluster(mm_gdf, 2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Notice that the two metrics produce different results" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/Triangle-Counting.ipynb b/cugraph/Triangle-Counting.ipynb new file mode 100644 index 00000000..744a7a8b --- /dev/null +++ b/cugraph/Triangle-Counting.ipynb @@ -0,0 +1,312 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Triangle Counting\n", + "\n", + "In this notebook, we will count the numner of trianges in our test dataset. The NetworkX and cuGraph processes will be interleaved so that each step can be compared.\n", + "\n", + "Notebook Credits\n", + "* Original Authors: Bradley Rees\n", + "* Last Edit: 05/03/2019\n", + "\n", + "RAPIDS Versions: 0.7.0 \n", + "\n", + "Test Hardware\n", + "\n", + "* GP100 32G, CUDA 9.2\n", + "\n", + "\n", + "## Introduction\n", + "Triancle Counting, as the name implies, finds the number of triangles in a graph. Triangles are important in computing the clustering Coefficient and can be used for clustering. \n", + "\n", + "\n", + "To compute the Pagerank scores for a graph in cuGraph we use:
\n", + "\n", + "**cugraph.triangles(G)**\n", + "* __G__: cugraph.Graph object\n", + "\n", + "\n", + "Returns:\n", + "* int64 count\n", + " \n", + "

\n", + "## cuGraph 0.7 Notice \n", + "cuGraph version 0.7 has some limitations:\n", + "* Only Int32 Vertex ID are supported\n", + "* Only float (FP32) edge data is supported\n", + "* Vertex numbering is assumed to start at zero\n", + "\n", + "These limitations are being addressed and will be fixed future versions. \n", + "These example notebooks will illustrate how to manipulate the data so that it comforms to the current limitations " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Data\n", + "We will be using the Zachary Karate club dataset \n", + "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n", + "Anthropological Research 33, 452-473 (1977).*\n", + "\n", + "\n", + "![Karate Club](./img/zachary_black_lines.png)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Prep" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# The notebook compares cuGraph to NetworkX, \n", + "# therefore there some additional non-RAPIDS python libraries need to be installed. \n", + "# Please run this cell if you need the additional libraries\n", + "!pip install networkx\n", + "!pip install scipy\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Import needed libraries\n", + "import cugraph\n", + "import cudf\n", + "from collections import OrderedDict" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# NetworkX libraries\n", + "import networkx as nx\n", + "from scipy.io import mmread" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Some Prep" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Define the path to the test data \n", + "datafile='./data/karate-data.csv'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "# NetworkX" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Read the data, this also created a NetworkX Graph \n", + "file = open(datafile, 'rb')\n", + "Gnx = nx.read_edgelist(file)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "nx_count = nx.triangles(Gnx)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# NetworkX does not give a single count, but list how many triangles each vertex is associated with\n", + "nx_count" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# To get the number of triangles, we would need to loop through the array and add up each count\n", + "count = 0\n", + "for key, value in nx_count.items():\n", + " count = count + value" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "count" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's seet how that compares to cuGraph\n", + "\n", + "----" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# cuGraph" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Read in the data - GPU\n", + "cuGraph depends on cuDF for data loading and the initial Dataframe creation\n", + "\n", + "The data file contains an edge list, which represents the connection of a vertex to another. The `source` to `destination` pairs is in what is known as Coordinate Format (COO). In this test case, the data is just two columns. However a third, `weight`, column is also possible" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Test file \n", + "datafile='./data/karate-data.csv'\n", + "\n", + "# Read the data file\n", + "cols = [\"src\", \"dst\"]\n", + "\n", + "dtypes = OrderedDict([\n", + " (\"src\", \"int32\"), \n", + " (\"dst\", \"int32\")\n", + " ])\n", + "\n", + "gdf = cudf.read_csv(datafile, names=cols, delimiter='\\t', dtype=list(dtypes.values()) )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Create a Graph " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# create a Graph using the source (src) and destination (dst) vertex pairs from the Dataframe \n", + "G = cugraph.Graph()\n", + "G.add_edge_list(gdf[\"src\"], gdf[\"dst\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Call the Triangle Counting " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Call cugraph.pagerank to get the pagerank scores\n", + "cu_count = cugraph.triangles(G)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "cu_count" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "_It was that easy!_ \n", + "\n", + "----\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/Vertex-Similarity.ipynb b/cugraph/Vertex-Similarity.ipynb new file mode 100644 index 00000000..ad44bbdb --- /dev/null +++ b/cugraph/Vertex-Similarity.ipynb @@ -0,0 +1,608 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Vertex Similarity\n", + "----\n", + "\n", + "In this notebook, we will use cuGraph to compute vertex similarity using both the Jaccard Similarity and the Overlap Coefficient. \n", + "\n", + "\n", + "Notebook Credits\n", + "\n", + " Original Authors: Bradley Rees\n", + " Last Edit: 04/24/2019\n", + "\n", + "RAPIDS Versions: 0.7.x\n", + "\n", + "Test Hardware\n", + "\n", + " GP100 32G, CUDA 9,2\n", + "\n", + "\n", + "\n", + "## Introduction\n", + "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Defining a Set\n", + "Both Jaccard and the Overlap Coefficient operate on sets, and in a graph setting, those sets are the list of neighbor vertices.
\n", + "For those that like math: The neighbors of a vertex, _v_, is defined as the set, _U_, of vertices connected by way of an edge to vertex v, or _N(v) = {U} where v ∈ V and ∀ u ∈ U ∃ edge(v,u)∈ E_.\n", + "\n", + "For the rest of this introduction, set A will equate to A = N(i) and set B will quate to B = N(j). That just make the rest of the text more readable." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Jaccard Similarity\n", + "\n", + "The Jaccard similarity between two sets is defined as the ratio of the volume of their intersection divided by the volume of their union. \n", + "\n", + "The Jaccard Similarity can then be defined as\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "For further detail see Wikipedia - https://en.wikipedia.org/wiki/Jaccard_index\n", + "\n", + "To compute the Jaccard similarity between all pairs of vertices connected by an edge in cuGraph use:
\n", + "__jaccard(G)__\n", + "\n", + " G: A cugraph.Graph object\n", + "\n", + "Returns:\n", + "\n", + " df: cudf.DataFrame with three names columns:\n", + " df[\"source\"]: The source vertex id.\n", + " df[\"destination\"]: The destination vertex id.\n", + " df[\"jaccard_coeff\"]: The jaccard coefficient computed between the source and destination vertex.\n", + "\n", + "
\n", + "\n", + "\n", + "__References__\n", + "\n", + " https://research.nvidia.com/publication/2017-11_Parallel-Jaccard-and" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Overlap Coefficient\n", + "\n", + "The Overlap Coefficient between two sets is defined as the ratio of the volume of their intersection divided by the volume of the smaller set.\n", + "The Overlap Coefficient can be defined as\n", + "\n", + "\n", + "\n", + "For further detail see Wikipedia - https://en.wikipedia.org/wiki/Overlap_coefficient\n", + "\n", + "To compute the Overlap Coefficient between all pairs of vertices connected by an edge in cuGraph use:
\n", + "\n", + "__overlap(G)__\n", + "\n", + " G: A cugraph.Graph object\n", + "\n", + "Returns:\n", + "\n", + " df: cudf.DataFrame with three names columns:\n", + " df[\"source\"]: The source vertex id.\n", + " df[\"destination\"]: The destination vertex id.\n", + " df[\"overlap_coeff\"]: The overlap coefficient computed between the source and destination vertex.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### cuGraph 0.7 Notice\n", + "\n", + "cuGraph version 0.7 has some limitations:\n", + "\n", + " Only Int32 Vertex ID are supported\n", + " Only float (FP32) edge data is supported\n", + " Vertex numbering is assumed to start at zero\n", + "\n", + "These limitations are being addressed and will be fixed in future versions.\n", + "These example notebooks will illustrate how to manipulate the data so that it comforms to the current limitations" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Data\n", + "We will be using the Zachary Karate club dataset \n", + "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n", + "Anthropological Research 33, 452-473 (1977).*\n", + "\n", + "\n", + "![Karate Club](./img/zachary_black_lines.png)\n", + "\n", + "This is a small graph which allows for easy visual inspection to validate results. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Import needed libraries\n", + "import cugraph\n", + "import cudf\n", + "from collections import OrderedDict" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "----\n", + "### Define some Print functions\n", + "(the `del` are not needed since going out of scope should free memory)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# define a function for printing the top most similar vertices\n", + "def print_most_similar_jaccard(df):\n", + " \n", + " jmax = df['jaccard_coeff'].max()\n", + " dm = df.query('jaccard_coeff >= @jmax') \n", + " \n", + " #find the best\n", + " for i in range(len(dm)): \n", + " print(\"Vertices \" + str(dm['source'][i]) + \" and \" + \n", + " str(dm['destination'][i]) + \" are most similar with score: \" \n", + " + str(dm['jaccard_coeff'][i]))\n", + " del jmax\n", + " del dm" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# define a function for printing the top most similar vertices\n", + "def print_most_similar_overlap(df):\n", + " \n", + " smax = df['overlap_coeff'].max()\n", + " dm = df.query('overlap_coeff >= @smax') \n", + " \n", + " for i in range(len(dm)):\n", + " print(\"Vertices \" + str(dm['source'][i]) + \" and \" + \n", + " str(dm['destination'][i]) + \" are most similar with score: \" \n", + " + str(dm['overlap_coeff'][i]))\n", + " \n", + " del smax\n", + " del dm" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# define a function for printing jaccard similar vertices based on a threshold\n", + "def print_jaccard_threshold(_d, limit):\n", + " \n", + " filtered = _d.query('jaccard_coeff > @limit')\n", + " \n", + " for i in range(len(filtered)):\n", + " print(\"Vertices \" + str(filtered['source'][i]) + \" and \" + \n", + " str(filtered['destination'][i]) + \" are similar with score: \" + \n", + " str(filtered['jaccard_coeff'][i]))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# define a function for printing similar vertices based on a threshold\n", + "def print_overlap_threshold(_d, limit):\n", + " \n", + " filtered = _d.query('overlap_coeff > @limit')\n", + " \n", + " for i in range(len(filtered)):\n", + " if filtered['source'][i] != filtered['destination'][i] :\n", + " print(\"Vertices \" + str(filtered['source'][i]) + \" and \" + \n", + " str(filtered['destination'][i]) + \" are similar with score: \" + \n", + " str(filtered['overlap_coeff'][i]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Read the CSV datafile using cuDF\n", + "data file is actually _tab_ separated, so we need to set the delimiter" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Test file \n", + "datafile='./data/karate-data.csv'\n", + "\n", + "# define the column names\n", + "cols = [\"src\", \"dst\"]\n", + "\n", + "# define the column data types\n", + "dtypes = OrderedDict([\n", + " (\"src\", \"int32\"), \n", + " (\"dst\", \"int32\")\n", + " ])\n", + "\n", + "gdf = cudf.read_csv(datafile, names=cols, delimiter='\\t', dtype=list(dtypes.values()) )" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Let's look at the DataFrame. There should be two columns and 156 records\n", + "gdf" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Look at the first few data records - the output should be two colums src and dst\n", + "gdf.head().to_pandas()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Create a Graph" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# create a Graph \n", + "G = cugraph.Graph()\n", + "G.add_edge_list(gdf[\"src\"], gdf[\"dst\"])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "G.degree()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# How many vertices are in the graph? Remember that Graph is zero based\n", + "G.number_of_vertices()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "_The test graph has only 34 vertices, so why is the Graph listing 35?_\n", + "\n", + "As mentioned above, cuGraph vertex numbering is zero-based, meaning that the first vertex ID starts at zero. The test dataset is 1-based. Because of that, the Graph object adds an extra isolated vertex with an ID of zero. Hence the difference in vertex count. \n", + "We are working on a renumbering feature to address this issue. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Jaccard " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Call cugraph.nvJaccard \n", + "%time df = cugraph.jaccard(G)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The Most similar shoul be 33 and 34.\n", + "Vertex 33 has 12 neighbors, vertex 34 has 17 neighbors. They share 10 neighbors in common:\n", + "$jaccard = 10 / (10 + (12 -10) + (17-10)) = 10 / 19 = 0.526$" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print_most_similar_jaccard(df)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "### let's look at all similarities over a threshold\n", + "print_jaccard_threshold(df, 0.4)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Since it is a small graph we can print all scores.\n", + "# Notice that only connected vertices are computed\n", + "\n", + "# let's sort the data first. Please note that you may get a warning. Just ignore it. \n", + "## It is just converted into a dataframe so that we could do this function call. \n", + "## If we were going to actually do further work on it, we would leave it as it was :)\n", + "\n", + "g = df.groupby(['jaccard_coeff'], method='cudf', as_index=False)\n", + "df_s = g.as_df()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# The groupby as_df function returns a tuple where the first item is the dataframe\n", + "print_jaccard_threshold(df_s[0], 0.0)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Overlap Coefficient" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Call cugraph.nvJaccard \n", + "do = cugraph.overlap(G)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print_most_similar_overlap(do)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Expanding vertex pairs for similarity scoring" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# get all two-hop vertex pairs\n", + "p = G.get_two_hop_neighbors()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Let's look at the Jaccard score\n", + "j2 = cugraph.jaccard(G, first=p['first'], second=p['second'])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print_most_similar_jaccard(j2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "notice that there are a lot of very similar vertices. For example vertices 15 and 16 share their only two neighbors in common. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "j2o = cugraph.overlap(G, first=p['first'], second=p['second'])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print_most_similar_overlap(j2o)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "the overlap score captures all the same matches that Jaccrd did, but also includes those sets that are exact subsets" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "----\n", + "### Adjusting the vertex ID\n", + "Let's adjust all the vertex IDs to be zero based. We are going to do this by adding two new columns with the adjusted IDs" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "gdf[\"src_0\"] = gdf[\"src\"] - 1\n", + "gdf[\"dst_0\"] = gdf[\"dst\"] - 1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# create a new Graph \n", + "G2 = cugraph.Graph()\n", + "G2.add_edge_list(gdf[\"src_0\"], gdf[\"dst_0\"])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# How many vertices are in the graph? Remember that Graph is zero based while teh data start at vertex 1\n", + "G2.number_of_vertices()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The number of vertices now matches what is in the test graph" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Call cugraph.nvJaccard \n", + "df2 = cugraph.jaccard(G2)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print_most_similar_jaccard(df2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Adjusting the vertices back (e.g adding +1 to vertex IDs) yields 33 and 34 which matches the orginal results.\n", + "For Jaccard, the fact that vertex IDs do not start of 0 is not an issue" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/Weighted-Jaccard.ipynb b/cugraph/Weighted-Jaccard.ipynb new file mode 100644 index 00000000..6e5244e9 --- /dev/null +++ b/cugraph/Weighted-Jaccard.ipynb @@ -0,0 +1,242 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Weighted Jaccard Similarity\n", + "\n", + "In this notebook, we will use cuGraph to compute the Weighted Jaccard Similarity metric on our training dataset. \n", + "\n", + "\n", + "Notebook Credits\n", + "* Original Authors: Bradley Rees and James Wyles\n", + "* Last Edit: 04/30/2019\n", + "\n", + "RAPIDS Versions: 0.7.0 \n", + "\n", + "\n", + "Test Hardware\n", + "* GP100 32G, CUDA 9,2\n", + "\n", + "\n", + "## Introduction\n", + "Weighted Jaccard is similar to the Jaccard Similarity but takes into account vertex weights placed. \n", + "\n", + "\n", + "given:\n", + "The neighbors of a vertex, v, is defined as the set, U, of vertices connected by way of an edge to vertex v, or N(v) = {U} where v ∈V and ∀ u∈U ∃ edge(v,u)∈E.\n", + "and\n", + "wt(i) is the weight on vertex i\n", + " \n", + "we can now define weight summing function as
\n", + "$WT(U) = \\sum_{v \\in U} {wt(v)}$\n", + "\n", + "$WtJaccard(i, j) = \\frac{WT(N(i) \\cap N(j))}{WT(N(i) \\cup N(j))}$\n", + "\n", + "\n", + "To compute the weighted Jaccard similarity between each pair of vertices connected by an edge in cuGraph use:
\n", + "\n", + "**jaccard_w(input_graph, vect_weights_ptr)**\n", + "\n", + "Input\n", + "* input_graph: A cugraph.Graph object\n", + "* vect_weights_ptr: An array of vertex weights\n", + "\n", + "Returns: \n", + "* __df__: cudf.DataFrame with three columns:\n", + " * df['source']: The source vertex id.\n", + " * df['destination']: The destination vertex id.\n", + " * df['jaccard_coeff']: The weighted jaccard coefficient computed between the source and destination vertex.\n", + " \n", + "

\n", + "\n", + "\n", + "__Note:__ For this example we will be using PageRank as the edge weights. Please review the PageRank notebook if you have any questions about running PageRank\n", + "\n", + "\n", + " \n", + "## cuGraph 0.7 Notice \n", + "cuGraph version 0.7 has some limitations:\n", + "* Only Int32 Vertex ID are supported\n", + "* Only float (FP32) edge data is supported\n", + "* Vertex numbering is assumed to start at zero\n", + "\n", + "These limitations are being addressed and will be fixed future versions. \n", + "These example notebooks will illustrate how to manipulate the data so that it comforms to the current limitations " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Data\n", + "We will be using the Zachary Karate club dataset \n", + "*W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of\n", + "Anthropological Research 33, 452-473 (1977).*\n", + "\n", + "\n", + "![Karate Club](./img/zachary_black_lines.png)\n", + "\n", + "This is a small graph which allows for easy visual inspection to validate results. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Import needed libraries\n", + "import cugraph\n", + "import cudf\n", + "from collections import OrderedDict" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Read the data using cuDF" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Test file \n", + "datafile='./data/karate-data.csv'\n", + "\n", + "# Read the data file\n", + "cols = [\"src\", \"dst\"]\n", + "\n", + "dtypes = OrderedDict([\n", + " (\"src\", \"int32\"), \n", + " (\"dst\", \"int32\")\n", + " ])\n", + "\n", + "gdf = cudf.read_csv(datafile, names=cols, delimiter='\\t', dtype=list(dtypes.values()) )" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# create a Graph \n", + "G = cugraph.Graph()\n", + "G.add_edge_list(gdf[\"src\"], gdf[\"dst\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Compute PageRank to use as the vertex weights" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Call Pagerank on the graph to get weights to use:\n", + "pr_df = cugraph.pagerank(G)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pr_df.head().to_pandas()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Now compute the Weighted Jaccard " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Call weighted Jaccard using the Pagerank scores as weights:\n", + "df = cugraph.jaccard_w(G, pr_df['pagerank'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Find the most similar pair of vertices" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "bestEdge = 0\n", + "for i in range(len(df)):\n", + " if df['jaccard_coeff'][i] > df['jaccard_coeff'][bestEdge]:\n", + " bestEdge = i\n", + " \n", + "print(\"Vertices \" + str(df['source'][bestEdge]) + \n", + " \" and \" + str(df['destination'][bestEdge] ) + \n", + " \" are most similar with score: \" + str(df['jaccard_coeff'][bestEdge]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## It is that easy with cuGraph!" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2019, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cugraph/data/karate-data.csv b/cugraph/data/karate-data.csv new file mode 100644 index 00000000..1ee3b2cb --- /dev/null +++ b/cugraph/data/karate-data.csv @@ -0,0 +1,156 @@ +1 2 +1 3 +1 4 +1 5 +1 6 +1 7 +1 8 +1 9 +1 11 +1 12 +1 13 +1 14 +1 18 +1 20 +1 22 +1 32 +2 1 +2 3 +2 4 +2 8 +2 14 +2 18 +2 20 +2 22 +2 31 +3 1 +3 2 +3 4 +3 8 +3 9 +3 10 +3 14 +3 28 +3 29 +3 33 +4 1 +4 2 +4 3 +4 8 +4 13 +4 14 +5 1 +5 7 +5 11 +6 1 +6 7 +6 11 +6 17 +7 1 +7 5 +7 6 +7 17 +8 1 +8 2 +8 3 +8 4 +9 1 +9 3 +9 31 +9 33 +9 34 +10 3 +10 34 +11 1 +11 5 +11 6 +12 1 +13 1 +13 4 +14 1 +14 2 +14 3 +14 4 +14 34 +15 33 +15 34 +16 33 +16 34 +17 6 +17 7 +18 1 +18 2 +19 33 +19 34 +20 1 +20 2 +20 34 +21 33 +21 34 +22 1 +22 2 +23 33 +23 34 +24 26 +24 28 +24 30 +24 33 +24 34 +25 26 +25 28 +25 32 +26 24 +26 25 +26 32 +27 30 +27 34 +28 3 +28 24 +28 25 +28 34 +29 3 +29 32 +29 34 +30 24 +30 27 +30 33 +30 34 +31 2 +31 9 +31 33 +31 34 +32 1 +32 25 +32 26 +32 29 +32 33 +32 34 +33 3 +33 9 +33 15 +33 16 +33 19 +33 21 +33 23 +33 24 +33 30 +33 31 +33 32 +33 34 +34 9 +34 10 +34 14 +34 15 +34 16 +34 19 +34 20 +34 21 +34 23 +34 24 +34 27 +34 28 +34 29 +34 30 +34 31 +34 32 +34 33 diff --git a/cugraph/img/GraphAnalyticsFigure.jpg b/cugraph/img/GraphAnalyticsFigure.jpg new file mode 100644 index 00000000..894931a8 Binary files /dev/null and b/cugraph/img/GraphAnalyticsFigure.jpg differ diff --git a/cugraph/img/rapids_logo.png b/cugraph/img/rapids_logo.png new file mode 100644 index 00000000..40504083 Binary files /dev/null and b/cugraph/img/rapids_logo.png differ diff --git a/cugraph/img/zachary_black_lines.png b/cugraph/img/zachary_black_lines.png new file mode 100644 index 00000000..937137e9 Binary files /dev/null and b/cugraph/img/zachary_black_lines.png differ diff --git a/cugraph/img/zachary_graph_pagerank.png b/cugraph/img/zachary_graph_pagerank.png new file mode 100644 index 00000000..3a9c34d3 Binary files /dev/null and b/cugraph/img/zachary_graph_pagerank.png differ diff --git a/cuml/README.md b/cuml/README.md index b8581161..49b301fa 100644 --- a/cuml/README.md +++ b/cuml/README.md @@ -4,8 +4,16 @@ * `dbscan_demo`: notebook showcasing density-based spatial clustering of applications with noise (dbscan) algorithm comparison between cuML and scikit-learn. * `knn_demo`: notebook showcasing k-nearest neighbors (knn) algorithm comparison between cuML and scikit-learn. +* `kmeans_demo`: notebook showcasing k-means clustering (kmeans) algorithm comparison between cuML and scikit-learn. * `pca_demo`: notebook showcasing principal component analysis (PCA) algorithm comparison between cuML and scikit-learn. * `tsvd_demo`: notebook showcasing truncated singular value decomposition (tsvd) algorithm comparison between cuML and scikit-learn. +* `linear_regression`: notebook showcasing linear regression comparison between cuML and scikit-learn. +* `ridge_regression`: notebook showcasing ridge regression comparison between cuML and scikit-learn. +* `sgd`: notebook showcasing stochastic gradient descent comparison between cuML and scikit-learn. +* `umap`: notebook showcasing and evaluating cuML's UMAP dimension reduction technique. +* `umap_demo_graphed`: Demonstration of cuML uniform manifold approximation & projection algorithm's supervised approach against mortgage dataset and comparison of results against the original author's equivalent non-GPU \Python implementation. +* `umap_supervised`: Demostration of UMAP supervised training. Uses a set of labels to perform supervised dimensionality reduction. UMAP can also be trained on datasets with incomplete labels, by using a label of "-1" for unlabeled samples. +* `coordinate descent`: This notebook includes code examples of lasso and elastic net models. These models are placed together so a comparison between the two can also be made in addition to their sklearn equivalent. ## dbscan_demo @@ -33,8 +41,6 @@ Note that the timing differences depend upon the exact dataset being used. Also, ## knn_demo -The `knn_demo` notebook demonstrates how cuml establishes interoperability between `cudf` and `faiss-gpu`. There is native support for this demo with CUDA 9.2. With CUDA 10.0, the user must build `faiss-gpu` from source with [GPU support](https://github.com/facebookresearch/faiss/blob/master/INSTALL.md). - Typical output of the cells processing knn looks like: - For scikit-learn knn: @@ -58,6 +64,28 @@ compare knn: cuml vs sklearn indexes NOT equal Note that the indexes can differ currently between results, but distances should be equal. +## kmeans_demo + +Typical output of the cells processing kmeans looks like: + +- For scikit-learn kmeans: +``` +CPU times: user 138 ms, sys: 236 ms, total: 374 ms +Wall time: 940 ms +``` + +- For cuML kmeans: +``` +CPU times: user 22.1 ms, sys: 20.3 ms, total: 42.4 ms +Wall time: 40.7 ms +``` + +Final cell of the notebook should output: + +``` +compare kmeans: cuml vs sklearn labels_ are equal +``` + ## pca_demo Typical output of the cells processing pca looks like: @@ -102,3 +130,277 @@ Final cell of the notebook should output: compare tsvd: cuml vs sklearn transformed results equal ``` +## linear_regression_demo + +Typical output of the cells processing linear_regression looks like: + +- For scikit-learn: + +Fit: + +``` +CPU times: user 1min 8s, sys: 21.2 s, total: 1min 30s +Wall time: 6.06 s +``` + +Predict: + +``` +CPU times: user 5.46 s, sys: 312 ms, total: 5.77 s +Wall time: 471 ms +``` + + +- For cuML: + +Fit: + +``` +CPU times: user 504 ms, sys: 347 ms, total: 851 ms +Wall time: 1.08 s +``` + +Predict: + +``` +CPU times: user 144 ms, sys: 7.71 ms, total: 152 ms +Wall time: 145 ms +``` + + +Final cell of the notebook should output: + +``` +SKL MSE(y): +5.6481553e-05 +CUML MSE(y): +7.246567e-07 +``` + +## ridge_regression_demo + +Typical output of the cells processing ridge_regression looks like: + +- For scikit-learn: + +Fit: + +``` +CPU times: user 1min 1s, sys: 1.99 s, total: 1min 3s +Wall time: 5.02 s +``` + +Predict: + +``` +CPU times: user 2.66 s, sys: 75.6 ms, total: 2.74 s +Wall time: 180 ms +``` + + +- For cuML: + +Fit: + +``` +CPU times: user 518 ms, sys: 309 ms, total: 827 ms +Wall time: 831 ms +``` + +Predict: + +``` +CPU times: user 146 ms, sys: 7.31 ms, total: 154 ms +Wall time: 149 ms +``` + +Final cell of the notebook should output: + +``` +SKL MSE(y): +0.0204307456949534 +CUML MSE(y): +0.00012496959 +``` + +## sgd_demo + +Typical output of the cells processing sgd looks like: + +- For scikit-learn: + +Fit: + +``` +CPU times: user 10min 22s, sys: 417 ms, total: 10min 22s +Wall time: 10min 20s +``` + +Predict: + +``` +CPU times: user 146 ms, sys: 63 ms, total: 209 ms +Wall time: 166 ms +``` + + +- For cuML: + +Fit: + +``` +CPU times: user 2min 13s, sys: 8.1 s, total: 2min 21s +Wall time: 2min 18s +``` + +Predict: + +``` +CPU times: user 139 ms, sys: 10.9 ms, total: 150 ms +Wall time: 142 ms +``` + +Final cell of the notebook should output: + +``` +SKL MSE(y): +1.144686926876654e-07 +CUML MSE(y): +1.0390148e-07 +``` + +## umap_demo + +This notebook currently performs assertions and does not print any output. It contains two types of tests that evaluate UMAP's ability to preserve local neighborhood structure. + +The first test verifies that when the input contains blobs generated from several different clusters, the UMAP output produces low-dimensional blobs with the same number of clusters. + +The second test demonstrates that the neighborhoods of the low-dimensional embeddings are similar to the neighborhoods of the inputs. A score, known as trustworthiness, and made popular by t-SNE, is used to evaluate the UMAP embeddings for both random and spectral initialization strategies. + +## umap demo graphed + +Outputs of the cells processing umap on the Fashion datasets look like: + +- For cuML: + +Fit_transform: + +``` +8.57 s ± 111 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) +``` + +Compare the CPU and GPU implementations of UMAP: + +``` +Scoring ~97% shows the GPU implementation is comparable to the original CPU implementation and the training time was ~9.5X faster +``` + +## umap suprevised + +Typical output of the cells processing umap looks like: + +- For cuML: + +Supervised fit_transform: + +``` +Took 3.194058 sec. +``` + +Unsupervised fit_transform: + +``` +Took 2.225810 sec +``` + +## Coordinate Descent + +### Lasso +Typical output of the cells processing lasso looks like: + +- For scikit-learn: + +Fit: + +``` +CPU times: user 59.8 s, sys: 19 s, total: 1min 18s +Wall time: 9.47 s +``` + +Predict: + +``` +CPU times: user 5.26 s, sys: 2.42 s, total: 7.68 s +Wall time: 1.24 s +``` + + +- For cuML: + +Fit: + +``` +CPU times: user 8.28 s, sys: 1.06 s, total: 9.34 s +Wall time: 2.11 s +``` + +Predict: + +``` +CPU times: user 392 ms, sys: 24 ms, total: 416 ms +Wall time: 410 ms +``` + +Final cell of the notebook should output: + +``` +SKL MSE(y): +1.2218163805946025e-05 +CUML MSE(y): +1.2218108e-05 +``` + +### Elastic Net +Typical output of the cells processing elastic net looks like: + +- For scikit-learn: + +Fit: + +``` +CPU times: user 59.7 s, sys: 23 s, total: 1min 22s +Wall time: 9.17 s +``` + +Predict: + +``` +CPU times: user 4.43 s, sys: 1.91 s, total: 6.34 s +Wall time: 1.21 s +``` + + +- For cuML: + +Fit: + +``` +CPU times: user 8.02 s, sys: 760 ms, total: 8.78 s +Wall time: 1.44 s +``` + +Predict: + +``` +CPU times: user 8.02 s, sys: 760 ms, total: 8.78 s +Wall time: 1.44 s +``` + +Final cell of the notebook should output: + +``` +SKL MSE(y): +1.2070175934420877e-05 +CUML MSE(y): +1.20697405e-05 +``` diff --git a/cuml/benchmarks.ipynb b/cuml/benchmarks.ipynb deleted file mode 100644 index 2af6db7d..00000000 --- a/cuml/benchmarks.ipynb +++ /dev/null @@ -1,633 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Benchmark and Bounds Tests\n", - "\n", - "The purpose of this notebook is to benchmark all of the single GPU cuML algorithms against their skLearn counterparts, while also providing the ability to find and verify upper bounds. \n", - "\n", - "This benchmark will persist results into a file so that benchmarking may be continued, in the case of failure. \n", - "\n", - "Also supported is the ability to draw charts with the results, which should aid in presentations and transparency to end-users. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Notebook Credits\n", - "### Authorship\n", - "Original Author: Corey Nolet \n", - "\n", - "Last Edit: Corey Nolet, 03/21/2019 \n", - " \n", - "### Test System Specs\n", - "Test System Hardware: DGX-1 \n", - "Test System Software: Ubuntu 16.04 \n", - "RAPIDS Version: 0.6.0 - Source Install \n", - "Driver: 410.79 \n", - "CUDA: 10.0 \n", - "\n", - "### Known Working Systems\n", - "RAPIDS Versions: 0.6" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "import pandas as pd\n", - "import cudf\n", - "import os\n", - "import time\n", - "import pickle\n", - "\n", - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "\n", - "from pylab import rcParams\n", - "rcParams['figure.figsize'] = 25, 10\n", - "rcParams['figure.dpi'] = 100\n", - "\n", - "sns.set_style(\"darkgrid\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Data loading functions" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import gzip\n", - "\n", - "\n", - "import gzip\n", - "def load_data_mortgage_X(nrows, ncols, cached = 'data/mortgage.npy.gz',source='mortgage'):\n", - " if os.path.exists(cached) and source=='mortgage':\n", - " print('use mortgage data')\n", - " with gzip.open(cached) as f:\n", - " X = np.load(f)\n", - " X = X[np.random.randint(0,X.shape[0]-1,nrows),:ncols]\n", - " else:\n", - " print('use random data')\n", - " X = np.random.random((nrows,ncols)).astype('float32')\n", - " df = pd.DataFrame({'fea%d'%i:X[:,i] for i in range(X.shape[1])}).fillna(0)\n", - " return df\n", - "\n", - "def load_data_mortgate_Xy(nrows, ncols, dtype = np.float32):\n", - " \"\"\"\n", - " Generate a dataframe and series based on rows and cols\n", - " \"\"\"\n", - " X = load_data_mortgage_X(nrows, ncols, dtype)\n", - " y = load_data_mortgage_X(nrows, 1, dtype)[\"fea0\"]\n", - " return (X, y)\n", - "\n", - "\n", - "def load_data_X(nrows, ncols, dtype = np.float32):\n", - " \"\"\"\n", - " Generate a single dataframe with specified rows and cols\n", - " \"\"\"\n", - " X = np.random.normal(nrows,ncols)\n", - " df = pd.DataFrame({'fea%d'%i:X[:,i].astype(dtype) for i in range(X.shape[1])})\n", - " return df\n", - "\n", - "def load_data_Xy(nrows, ncols, dtype = np.float32):\n", - " \"\"\"\n", - " Generate a dataframe and series based on rows and cols\n", - " \"\"\"\n", - " X = load_data_X(nrows, ncols, dtype)\n", - " y = load_data_X(nrows, 1, dtype)[\"fea0\"]\n", - " return (X, y)\n", - "\n", - "def load_data_X_npy(nrows, ncols, dtype=np.float32):\n", - " return np.random.uniform(-1, 1,(nrows, ncols))\n", - "\n", - "def load_data_Xy_npy(nrows, ncols, dtype = np.float32):\n", - " X = load_data_X_npy(nrows, ncols, dtype)\n", - " y = load_data_X_npy(nrows, 1, dtype)\n", - " return (X, y)\n", - " " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def pandas_convert(data):\n", - " if isinstance(data, tuple):\n", - " return tuple([pandas_convert(d) for d in data])\n", - " elif isinstance(data, pd.DataFrame):\n", - " return cudf.DataFrame.from_pandas(data)\n", - " elif isinstance(data, pd.Series):\n", - " return cudf.Series.from_pandas(data)\n", - " else:\n", - " raise Exception(\"Unsupported type %s\" % str(type(data)))\n", - " \n", - "def no_convert(data):\n", - " if isinstance(data, tuple):\n", - " return tuple([d for d in data])\n", - " elif isinstance(data, np.ndarray):\n", - " return data\n", - " else:\n", - " raise Exception(\"Unsupported type %s\" % str(type(data)))\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Pluggable benchmark function " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class SpeedupBenchmark(object):\n", - " \n", - " def __init__(self, converter = pandas_convert):\n", - " self.name = \"speedup\"\n", - " self.converter = converter\n", - " \n", - " def __str__(self):\n", - " return \"Speedup\"\n", - " \n", - " def run(self, algo, rows, dims, data):\n", - "\n", - " data2 = self.converter(data)\n", - " cu_start = time.time()\n", - " algo.cuml(data2)\n", - " cu_elapsed = time.time() - cu_start\n", - " \n", - " sk_start = time.time()\n", - " algo.sk(data)\n", - " sk_elapsed = time.time() - float(sk_start)\n", - "\n", - " # Needs to return the calculation and the name given to it.\n", - " return sk_elapsed / float(cu_elapsed)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class BenchmarkRunner(object):\n", - " \n", - " def __init__(self, \n", - " benchmarks = [SpeedupBenchmark()],\n", - " out_filename = \"benchmark.pickle\",\n", - " rerun = False,\n", - " n_runs = 3,\n", - " bench_rows = [2**x for x in range(13, 20)],\n", - " bench_dims = [64, 128, 256, 512]):\n", - "\n", - " self.benchmarks = benchmarks\n", - " self.rerun = rerun\n", - " self.n_runs = n_runs\n", - " self.bench_rows = bench_rows\n", - " self.bench_dims = bench_dims\n", - " self.out_filename = out_filename\n", - " \n", - " \n", - " def load_results(self):\n", - " \n", - " if os.path.exists(self.out_filename):\n", - " print(\"Loaded previous benchmark results from %s\" % (self.out_filename))\n", - " with open(self.out_filename, 'rb') as f:\n", - " return pickle.load(f)\n", - " \n", - " else:\n", - " return {}\n", - " \n", - " def store_results(self, final_results):\n", - " with open(self.out_filename, 'wb') as f:\n", - " pickle.dump(final_results, f)\n", - " \n", - " \n", - " def run(self, algo):\n", - " \n", - " final_results = self.load_results()\n", - " \n", - " for benchmark in self.benchmarks:\n", - " if algo.name in final_results:\n", - " results = final_results[algo.name]\n", - " else:\n", - " results = {}\n", - " final_results[algo.name] = results\n", - "\n", - " for n_rows in self.bench_rows:\n", - " for n_dims in self.bench_dims: \n", - " if (n_rows, n_dims, benchmark.name) not in results or self.rerun:\n", - "\n", - " print(\"Running %s. (nrows=%d, n_dims=%d)\" % (str(algo), n_rows, n_dims))\n", - "\n", - " data = algo.load_data(n_rows, n_dims)\n", - " runs = [benchmark.run(algo, n_rows, n_dims, data) for i in range(self.n_runs)]\n", - " results[(n_rows, n_dims, benchmark.name)] = np.mean(runs)\n", - "\n", - " print(\"Benchmark for %s = %f\" % (str((n_rows, n_dims, benchmark.name)), \n", - " results[(n_rows, n_dims, benchmark.name)]))\n", - " \n", - " self.store_results(final_results)\n", - "\n", - " \n", - " def chart(self, algo, title = \"cuML vs SKLearn\"):\n", - " \n", - " for benchmark in self.benchmarks:\n", - " \n", - " results = self.load_results()[algo.name]\n", - "\n", - " final = {}\n", - "\n", - " plts = []\n", - " for dim in self.bench_dims:\n", - " data = {k: v for (k, v) in results.items() if dim == k[1]}\n", - "\n", - " if len(data) > 0:\n", - " data = [(k[0], v) for k, v in data.items()]\n", - " data.sort(key = lambda x: x[0])\n", - "\n", - " final[dim] = list(map(lambda x: x[1], data))\n", - "\n", - " keys = list(map(lambda x: np.log2(x[0]), data))\n", - " line = plt.plot(keys, final[dim], label = str(dim), linewidth = 3, marker = 'o', markersize = 7)\n", - "\n", - " plts.append(line[0])\n", - " leg = plt.legend(handles = plts, fontsize = 10)\n", - " leg.set_title(\"Dimensions\", prop = {'size':'x-large'}) \n", - " plt.title(\"%s %s: %s\" % (algo, benchmark, title), fontsize = 20)\n", - "\n", - " plt.ylabel(str(benchmark), fontsize = 20)\n", - " plt.xlabel(\"Training Examples (2^x)\", fontsize = 20)\n", - "\n", - " plt.tick_params(axis='both', which='major', labelsize=15)\n", - " plt.tick_params(axis='both', which='minor', labelsize=15)\n", - "\n", - " plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class BaseAlgorithm(object):\n", - " def __init__(self, load_data = load_data_X):\n", - " self.load_data = load_data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Nearest Neighbors" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from sklearn.neighbors import NearestNeighbors\n", - "from cuml.neighbors import NearestNeighbors as cumlNN\n", - "\n", - "class NearestNeighborsAlgo(BaseAlgorithm):\n", - " \n", - " def __init__(self, n_neighbors = 1024, load_data = load_data_X):\n", - " self.n_neighbors = n_neighbors\n", - " self.name = \"nearest_neighbors\"\n", - "\n", - " BaseAlgorithm.__init__(self, load_data)\n", - " \n", - " def __str__(self):\n", - " return \"NearestNeighbors\"\n", - " \n", - " def sk(self, X):\n", - " knn_sk = NearestNeighbors(n_neighbors = self.n_neighbors, algorithm = 'brute')\n", - " knn_sk.fit(X)\n", - " D_sk,I_sk = knn_sk.kneighbors(X[0:100])\n", - " print(\"Done SK.\")\n", - "\n", - " def cuml(self, X):\n", - " print(\"Starting cuml\")\n", - " knn_cuml = cumlNN(n_neighbors = self.n_neighbors, n_gpus=5)\n", - " knn_cuml.fit(X)\n", - " print(\"Done fit.\")\n", - " D_cuml,I_cuml = knn_cuml.kneighbors(X[0:100])\n", - " print(\"Done kneighbors.\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(benchmarks = [SpeedupBenchmark(no_convert)], bench_rows = [2**x for x in range(23, 30)])\n", - "runner.run(NearestNeighborsAlgo(load_data = load_data_X_npy))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner()\n", - "runner.chart(NearestNeighborsAlgo())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### DBSCAN" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from sklearn.cluster import DBSCAN as skDBSCAN\n", - "from cuml import DBSCAN as cumlDBSCAN\n", - "\n", - "class DBSCANAlgo(BaseAlgorithm):\n", - " \n", - " def __init__(self, eps = 3, min_samples = 2):\n", - " self.name = \"dbscan\"\n", - " self.eps = 3\n", - " self.min_samples = 2\n", - " BaseAlgorithm.__init__(self)\n", - " \n", - " def __str__(self):\n", - " return \"DBSCAN\"\n", - "\n", - " def sk(self, X):\n", - " clustering_sk = skDBSCAN(eps = self.eps, min_samples = self.min_samples, algorithm = \"brute\")\n", - " clustering_sk.fit(X)\n", - "\n", - " def cuml(self, X):\n", - " clustering_cuml = cumlDBSCAN(eps = self.eps, min_samples = self.min_samples)\n", - " clustering_cuml.fit(X)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(bench_rows = [2**x for x in range(10, 17)])\n", - "runner.run(DBSCANAlgo())" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(bench_rows = [2**x for x in range(10, 17)])\n", - "runner.chart(DBSCANAlgo())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### UMAP" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from umap import UMAP as skUMAP\n", - "from cuml.manifold.umap import UMAP as cumlUMAP\n", - "\n", - "class UMAPAlgo(BaseAlgorithm):\n", - " \n", - " def __init__(self, n_neighbors = 5, n_epochs = 500):\n", - " self.name = \"umap\"\n", - " self.n_neighbors = n_neighbors\n", - " self.n_epochs = n_epochs\n", - " BaseAlgorithm.__init__(self)\n", - " \n", - " def __str__(self):\n", - " return \"UMAP\"\n", - "\n", - " def sk(self, X):\n", - " clustering_sk = skUMAP(n_neighbors = self.n_neighbors, n_epochs = self.n_epochs)\n", - " clustering_sk.fit(X)\n", - "\n", - " def cuml(self, X):\n", - " clustering_cuml = cumlUMAP(n_neighbors = self.n_neighbors, n_epochs = self.n_epochs)\n", - " clustering_cuml.fit(X)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(rerun = True, bench_rows = [2**x for x in range(10, 20)])\n", - "runner.run(UMAPAlgo())" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(bench_rows = [2**x for x in range(10, 15)])\n", - "runner.chart(UMAPAlgo())" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner().load_results()\n", - "\n", - "l = []\n", - "for k in runner[\"umap\"]:\n", - " if k[0] > 32768:\n", - " l.append(k)\n", - " \n", - "for i in l:\n", - " del runner[\"umap\"][i]\n", - " \n", - " \n", - " \n", - " " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "BenchmarkRunner().store_results(runner)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Linear Regression" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from sklearn.linear_model import LinearRegression as skLR\n", - "from cuml.linear_model import LinearRegression as cumlLR\n", - "\n", - "class LinearRegressionAlgo(BaseAlgorithm):\n", - " def __init__(self):\n", - " BaseAlgorithm.__init__(self, load_data_Xy)\n", - " self.name = \"linear_regression\"\n", - " \n", - " def __str__(self):\n", - " return \"Linear Regression\"\n", - "\n", - " def sk(self, data):\n", - " X, y = data\n", - " clustering_sk = skLR()\n", - " clustering_sk.fit(X, y)\n", - "\n", - " def cuml(self, data):\n", - " X, y = data\n", - " cuml_lr = cumlLR()\n", - " cuml_lr.fit(X, y)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(rerun = True, bench_rows = [2**x for x in range(15, 23)]).run(LinearRegressionAlgo())" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(bench_rows = [2**x for x in range(10, 17)])\n", - "runner.chart(LinearRegressionAlgo())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### PCA / SVD" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from sklearn.decomposition import PCA as skPCA\n", - "from cuml import PCA as cumlPCA\n", - "\n", - "class PCAAlgo(BaseAlgorithm):\n", - " \n", - " def __init__(self, n_components = 10, load_data = load_data_mortgage_X):\n", - " self.n_components = 10\n", - " self.name = \"pca\"\n", - " BaseAlgorithm.__init__(self, load_data = load_data)\n", - " \n", - " def __str__(self):\n", - " return \"PCA\"\n", - "\n", - " def sk(self, X):\n", - " skpca = skPCA(n_components = 10)\n", - " skpca.fit(X)\n", - "\n", - " def cuml(self, X):\n", - " cumlpca = cumlPCA(n_components = 10)\n", - " cumlpca.fit(X)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(bench_rows = [2**x for x in range(18, 23)])\n", - "runner.run(PCAAlgo())" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "runner = BenchmarkRunner(bench_rows = [2**x for x in range(10, 17)])\n", - "runner.chart(LinearRegressionAlgo())" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python (cuml3)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.6" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/cuml/coordinate_descent_demo.ipynb b/cuml/coordinate_descent_demo.ipynb new file mode 100644 index 00000000..3527f313 --- /dev/null +++ b/cuml/coordinate_descent_demo.ipynb @@ -0,0 +1,348 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Coordinate descent\n", + "CuML library can implement lasso and elastic net algorithms. The lasso model extends LinearRegression with L2 regularization and elastic net extends LinearRegression with a combination of L1 and L2 regularizations.\n", + "We see tremendous speed up for datasets with large number of rows and fewer columns. Furthermore, the MSE value for the cuML implementation is much smaller than the scikit-learn implementation for very small datasets." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Select a particular GPU to run the notebook (if needed)\n", + "# %env CUDA_VISIBLE_DEVICES=2\n", + "# Import the required libraries\n", + "import numpy as np\n", + "import pandas as pd\n", + "import cudf\n", + "import os\n", + "from cuml import Lasso as cuLasso\n", + "from sklearn.linear_model import Lasso\n", + "from sklearn.datasets import make_regression\n", + "from sklearn.metrics import mean_squared_error\n", + "from cuml.linear_model import ElasticNet as cuElasticNet\n", + "from sklearn.linear_model import ElasticNet" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Helper Functions" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Check if the mortgage dataset is present and then extract the data from it, else just create a random dataset for regression \n", + "import gzip\n", + "def load_data(nrows, ncols, cached = 'data/mortgage.npy.gz'):\n", + " # Split the dataset in a 80:20 split\n", + " train_rows = int(nrows*0.8)\n", + " if os.path.exists(cached):\n", + " print('use mortgage data')\n", + "\n", + " with gzip.open(cached) as f:\n", + " X = np.load(f)\n", + " # The 4th column is 'adj_remaining_months_to_maturity'\n", + " # used as the label\n", + " X = X[:,[i for i in range(X.shape[1]) if i!=4]]\n", + " y = X[:,4:5]\n", + " rindices = np.random.randint(0,X.shape[0]-1,nrows)\n", + " X = X[rindices,:ncols]\n", + " y = y[rindices]\n", + " df_y_train = pd.DataFrame({'fea%d'%i:y[0:train_rows,i] for i in range(y.shape[1])})\n", + " df_y_test = pd.DataFrame({'fea%d'%i:y[train_rows:,i] for i in range(y.shape[1])})\n", + " else:\n", + " print('use random data')\n", + " X,y = make_regression(n_samples=nrows,n_features=ncols,n_informative=ncols, random_state=0)\n", + " df_y_train = pd.DataFrame({'fea0':y[0:train_rows,]})\n", + " df_y_test = pd.DataFrame({'fea0':y[train_rows:,]})\n", + "\n", + " df_X_train = pd.DataFrame({'fea%d'%i:X[0:train_rows,i] for i in range(X.shape[1])})\n", + " df_X_test = pd.DataFrame({'fea%d'%i:X[train_rows:,i] for i in range(X.shape[1])})\n", + "\n", + " return df_X_train, df_X_test, df_y_train, df_y_test" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Obtain and convert the dataset" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# nrows = number of samples\n", + "# ncols = number of features of each sample \n", + "nrows = 2**21\n", + "ncols = 500\n", + "\n", + "# Split the dataset into training and testing sets, in the ratio of 80:20 respectively\n", + "X_train, X_test, y_train, y_test = load_data(nrows,ncols)\n", + "print('training data',X_train.shape)\n", + "print('training label',y_train.shape)\n", + "print('testing data',X_test.shape)\n", + "print('testing label',y_test.shape)\n", + "print('label',y_test.shape)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Convert the pandas dataframe to cudf format\n", + "X_cudf = cudf.DataFrame.from_pandas(X_train)\n", + "X_cudf_test = cudf.DataFrame.from_pandas(X_test)\n", + "y_cudf = y_train.values\n", + "y_cudf = y_cudf[:,0]\n", + "y_cudf = cudf.Series(y_cudf)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Define the model parameters" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# lr = learning rate\n", + "# algo = algorithm used in the model\n", + "lr = 0.001\n", + "algo = 'cyclic'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Lasso\n", + "The lasso model implemented in cuml allows the user to change the following parameter values:\n", + "1. alpha: regularizing constant that is multiplied with L1 to control the extent of regularization. (default = 1)\n", + "2. normalize: variable decides if the predictors in X will be normalized or not. (default = False)\n", + "3. fit_intercept: if set to True the model tries to center the data. (default = True)\n", + "4. max_iter: maximum number of iterations for training (fitting) the data to the model. (default = 1000)\n", + "5. tol: the tolerance for optimization. (default = 1e-3)\n", + "3. algorithm: the user can set the algorithm value as 'cyclic' or 'random'\n", + "\n", + "The model can take array-like objects, either in host as NumPy arrays or in device (as Numba or _cuda_array_interface_compliant), as well as cuDF DataFrames. In order to convert your dataset to cudf format please read the cudf documentation on https://rapidsai.github.io/projects/cudf/en/latest/. For additional information on the lasso model please refer to the documentation on https://rapidsai.github.io/projects/cuml/en/latest/index.html" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Scikit-learn model for lasso" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Use the sklearn lasso model to fit the training dataset \n", + "skols = Lasso(alpha=np.array([lr]), fit_intercept = True, normalize = False, max_iter = 1000, selection=algo, tol=1e-10)\n", + "skols.fit(X_train, y_train)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Calculate the mean squared error for the sklearn lasso model on the testing dataset\n", + "sk_predict = skols.predict(X_test)\n", + "error_sk = mean_squared_error(y_test,sk_predict)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### CuML model for lasso" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Run the cuml linear regression model to fit the training dataset \n", + "cuols = cuLasso(alpha=np.array([lr]), fit_intercept = True, normalize = False, max_iter = 1000, selection=algo, tol=1e-10)\n", + "cuols.fit(X_cudf, y_cudf)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Calculate the mean squared error of the testing dataset using the cuml linear regression model\n", + "cu_predict = cuols.predict(X_cudf_test).to_array()\n", + "error_cu = mean_squared_error(y_test,cu_predict)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Print the mean squared error of the sklearn and cuml model to compare the two\n", + "print(\"SKL MSE(y):\")\n", + "print(error_sk)\n", + "print(\"CUML MSE(y):\")\n", + "print(error_cu)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Elastic Net\n", + "The elastic net model implemented in cuml contains the same parameters as the lasso model.\n", + "In addition to the variable values that can be altered in lasso, elastic net has another variable who's value can be changed\n", + "- l1_ratio: decides the ratio of amount of L1 and L2 regularization that would be applied to the model. When L1 ratio = 0, the model will have only L2 reqularization shall be applied to the model. (default = 0.5)\n", + "\n", + "The model can take array-like objects, either in host as NumPy arrays or in device (as Numba or _cuda_array_interface_compliant), as well as cuDF DataFrames. In order to convert your dataset to cudf format please read the cudf documentation on https://rapidsai.github.io/projects/cudf/en/latest/. For additional information on the lasso model please refer to the documentation on https://rapidsai.github.io/projects/cuml/en/latest/index.html" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Scikit-learn model for elastic net" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Use the sklearn linear regression model to fit the training dataset \n", + "elastic_sk = ElasticNet(alpha=np.array([lr]), fit_intercept = True, normalize = False, max_iter = 1000, selection=algo, tol=1e-10)\n", + "elastic_sk.fit(X_train, y_train)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Calculate the mean squared error of the sklearn linear regression model on the testing dataset\n", + "sk_predict_elas = elastic_sk.predict(X_test)\n", + "error_sk_elas = mean_squared_error(y_test,sk_predict_elas)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### CuML model for elastic net" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Run the cuml linear regression model to fit the training dataset \n", + "elastic_cu = cuElasticNet(alpha=np.array([lr]), fit_intercept = True, normalize = False, max_iter = 1000, selection=algo, tol=1e-10)\n", + "elastic_cu.fit(X_cudf, y_cudf)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "# Calculate the mean squared error of the testing dataset using the cuml linear regression model\n", + "cu_predict_elas = elastic_cu.predict(X_cudf_test).to_array()\n", + "error_cu_elas = mean_squared_error(y_test,cu_predict_elas)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Print the mean squared error of the sklearn and cuml model to compare the two\n", + "print(\"SKL MSE(y):\")\n", + "print(error_sk_elas)\n", + "print(\"CUML MSE(y):\")\n", + "print(error_cu_elas)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.8" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cuml/data/fashion/t10k-images-idx3-ubyte.gz b/cuml/data/fashion/t10k-images-idx3-ubyte.gz new file mode 100644 index 00000000..667844f1 Binary files /dev/null and b/cuml/data/fashion/t10k-images-idx3-ubyte.gz differ diff --git a/cuml/data/fashion/t10k-labels-idx1-ubyte.gz b/cuml/data/fashion/t10k-labels-idx1-ubyte.gz new file mode 100644 index 00000000..abdddb89 Binary files /dev/null and b/cuml/data/fashion/t10k-labels-idx1-ubyte.gz differ diff --git a/cuml/data/fashion/train-images-idx3-ubyte.gz b/cuml/data/fashion/train-images-idx3-ubyte.gz new file mode 100644 index 00000000..e6ee0e37 Binary files /dev/null and b/cuml/data/fashion/train-images-idx3-ubyte.gz differ diff --git a/cuml/data/fashion/train-labels-idx1-ubyte.gz b/cuml/data/fashion/train-labels-idx1-ubyte.gz new file mode 100644 index 00000000..9c4aae27 Binary files /dev/null and b/cuml/data/fashion/train-labels-idx1-ubyte.gz differ diff --git a/cuml/dbscan_demo.ipynb b/cuml/dbscan_demo.ipynb index 7f9bb700..3389d258 100644 --- a/cuml/dbscan_demo.ipynb +++ b/cuml/dbscan_demo.ipynb @@ -1,5 +1,23 @@ { "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Density-Based Spatial Culstering of Applications with Noise (DBSCAN)\n", + "The DBSCAN algorithm is a clustering algorithm which works really well for datasets in which samples conregate in large groups. cuML’s DBSCAN expects a cuDF DataFrame, and constructs an adjacency graph to compute the distances between close neighbours. The DBSCAN model implemented in the cuML library can accept the following parameters : \n", + "1. eps: maximum distance between 2 sample points\n", + "2. min_samples: minimum number of samples that should be present in a neighborhood for it to be considered as a core points.\n", + "\n", + "The methods that can be used with DBSCAN are: \n", + "1. fit: Perform DBSCAN clustering from features.\n", + "1. fit_predict: Performs clustering on input_gdf and returns cluster labels.\n", + "1. get_params: Sklearn style return parameter state\n", + "1. set_params: Sklearn style set parameter state to dictionary of params.\n", + "\n", + "The model can take array-like objects, either in host as NumPy arrays or in device (as Numba or _cuda_array_interface_compliant), as well as cuDF DataFrames. In order to convert your dataset to cudf format please read the cudf documentation on https://rapidsai.github.io/projects/cudf/en/latest/. For additional information on the DBSCAN model please refer to the documentation on https://rapidsai.github.io/projects/cuml/en/latest/index.html" + ] + }, { "cell_type": "code", "execution_count": null, @@ -27,36 +45,9 @@ "metadata": {}, "outputs": [], "source": [ - "from timeit import default_timer\n", - "\n", - "class Timer(object):\n", - " def __init__(self):\n", - " self._timer = default_timer\n", - " \n", - " def __enter__(self):\n", - " self.start()\n", - " return self\n", - "\n", - " def __exit__(self, *args):\n", - " self.stop()\n", - "\n", - " def start(self):\n", - " \"\"\"Start the timer.\"\"\"\n", - " self.start = self._timer()\n", - "\n", - " def stop(self):\n", - " \"\"\"Stop the timer. Calculate the interval in seconds.\"\"\"\n", - " self.end = self._timer()\n", - " self.interval = self.end - self.start" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ + "# check if the mortgage dataset is present and then extract the data from it, else just create a random dataset for clustering \n", "import gzip\n", + "# change the path of the mortgage dataset if you have saved it in a different directory\n", "def load_data(nrows, ncols, cached = 'data/mortgage.npy.gz'):\n", " if os.path.exists(cached):\n", " print('use mortgage data')\n", @@ -64,6 +55,7 @@ " X = np.load(f)\n", " X = X[np.random.randint(0,X.shape[0]-1,nrows),:ncols]\n", " else:\n", + " # create a random dataset\n", " print('use random data')\n", " X = np.random.rand(nrows,ncols)\n", " df = pd.DataFrame({'fea%d'%i:X[:,i] for i in range(X.shape[1])})\n", @@ -76,6 +68,7 @@ "metadata": {}, "outputs": [], "source": [ + "# this function checks if the results obtained from two different methods is the same\n", "from sklearn.metrics import mean_squared_error\n", "def array_equal(a,b,threshold=5e-3,with_sign=True):\n", " a = to_nparray(a)\n", @@ -85,6 +78,7 @@ " res = mean_squared_error(a,b)\n", + "\n", + "| Parameters: | Description: |\n", + "| ------------ | ---------------- |\n", + "| `n_clusters:int (default = 8)` | The number of centroids or clusters you want. |\n", + "| `max_iter:int (default = 300)` | The more iterations of EM, the more accurate, but slower. |\n", + "| `tol:float (default = 1e-4)` | Stopping criterion when centroid means do not change much. |\n", + "| `verbose:boolean (default = 0)` | If True, prints diagnositc information. |\n", + "| `random_state:int (default = 1)` | If you want results to be the same when you restart Python, select a state. |\n", + "| `precompute_distances:boolean (default = ‘auto’)` | Not supported yet. |\n", + "| `init: (default = 'scalable-k-means++')`
-`'scalable-k-means++' or 'k-meansǀǀ'`
-`'random' or an ndarray` |
- Uses fast and stable scalable kmeans++ intialization.
- Choose 'n_cluster' observations (rows) at random from data for the initial centroids.
If an ndarray is passed, it should be of shape (n_clusters, n_features) and gives the initial centers. |\n", + "| `n_init:int (default = 1)` | Number of times intialization is run. More is slower, but can be better. |\n", + "| `algorithm:“auto”` | Currently uses full EM, but will support others later. |\n", + "\n", + "
\n", + "\n", + "| Methods: | Description: |\n", + "| ------------------| ----------------- |\n", + "|`fit(self, X)` | Compute k-means clustering with X. |\n", + "|`fit_predict(self, X)` | Compute cluster centers and predict cluster index for each sample. |\n", + "|`fit_transform(self, input_gdf)` | Compute clustering and transform input_gdf to cluster-distance space. |\n", + "|`get_params(self[, deep])` | Sklearn style return parameter state |\n", + "|`predict(self, X)` | Predict the closest cluster each sample in X belongs to. |\n", + "|`set_params(self, **params)` | Sklearn style set parameter state to dictionary of params. |\n", + "|`transform(self, X)` | Transform X to a cluster-distance space. |\n", + "\n", + "
\n", + " \n", + "The model can take array-like objects, either in host as NumPy arrays or in device (as Numba or _cuda_array_interface_compliant), as well as cuDF DataFrames as the input. For additional information on the KMeans model please refer to the documentation on https://rapidsai.github.io/projects/cuml/en/latest/index.html\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "from sklearn.cluster import KMeans as skKMeans\n", + "from sklearn import datasets\n", + "from cuml import KMeans as cumlKMeans\n", + "%matplotlib inline\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.metrics import adjusted_rand_score" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Run tests" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "rs = 7\n", + "# create a blobs dataset with 500 samples and 10 features each\n", + "data, labels = datasets.make_blobs(\n", + " n_samples=100000, n_features=2, centers=5, random_state=rs)\n", + "\n", + "kmeans_sk = skKMeans(n_clusters=5, n_jobs=-1, random_state=rs)\n", + "kmeans_cuml = cumlKMeans(n_clusters=5, n_gpu=1, random_state=rs)" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Calling sklearn fit\n", + "CPU times: user 138 ms, sys: 292 ms, total: 430 ms\n", + "Wall time: 967 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "print(\"Calling sklearn fit\")\n", + "kmeans_sk.fit(data)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Calling cuml fit\n", + "CPU times: user 356 ms, sys: 9.18 ms, total: 365 ms\n", + "Wall time: 363 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "print(\"Calling cuml fit\")\n", + "kmeans_cuml.fit(data)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6wAAAJOCAYAAACzwIp5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3hUxdfA8e9sT6H3DoKAgNJBxYKoCIIKduwVy0+xoAj2gr52RcUGKAiKNGmC9N6b9N4DIaSQQsr2ef/YTUzIbrJpJMD5PI+P7N57Z+buLjx7ds6cUVprhBBCCCGEEEKIssZQ2gMQQgghhBBCCCECkYBVCCGEEEIIIUSZJAGrEEIIIYQQQogySQJWIYQQQgghhBBlkgSsQgghhBBCCCHKJAlYhRBCCCGEEEKUSRKwCiGEKDCllFZKNTkL/TyilFpRiOuWKKWeKOixs+1svY6lrazcp1LqdaXUyNIehxBCiNCZSnsAQgghhBD5UUqNBo5prd8sbBta64+Kb0RCCCHOBplhFUIIIYqBUkp+BC7D5P0RQohzkwSsQghxHlFK1VNK/aWUilNKJSilvvM//65Saly28xr60zRN/sdLlFJDlVKrlFKpSqmZSqkqSqnflVIpSqn1SqmGIY7hUaXULqXUaaXUQaXUU9mOdVVKHVNKDVRKxSqlTiilHs12vIpSaoa/z3VA4zz6sSmlxvnvM8k/xhoBzqullNqqlHolSDuP+cebqJSaq5RqkO3YMKVUlH88G5VSV2c79q5SarJ/DCnAI/7nJiqlfvPf/w6lVIcQX7er/H1d53+slVLPKqX2+dv6QCnVWCm12j+eiUopS7breyulNvtfi1VKqcuyHRuslDrgb2enUqpvtmOPKKVWKKU+978Gh5RSPc84ftB/7SGl1P1Bxm/0p9xm9rNRKVUvwHk5UrKzp30rn6/8n41k//vWSinVH7gfGJT5+fSfX1spNcX/eT+klBoQwvszzn888+/Aw0qpo0qpeKXUG9muD1NKjfG/JruUUoOUUsdCeS+FEEIUHwlYhRDiPKGUMgJ/A0eAhkAd4M8CNHEv8KD/usbAauBXoDKwC3gnxHZigd5AeeBR4CulVLtsx2sCFfz9PA4MV0pV8h8bDtiBWsBj/v+CedjfTj2gCvA0kJH9BOULspcC32mtPz+zAaVUH+B14HagGrAcGJ/tlPVAG3yvwR/AJKWULdvx24DJQEXgd/9zt+J73SsCM4Dv8riHzHHc5O/3Dq314myHegDtgcuBQcDP+AK3ekAroJ//+nbAL8BT/tfiJ2CGUsrqb+cAcDW+1+s9YJxSqla2fjoDe4CqwKfAKH/wGAF8A/TUWpcDrgQ2B7mNl/3juRnfe/8YkJ7fvZ+hO3AN0BTf63cPkKC1/hnf6/up1jpSa32LUsoAzAS24PssXQ+86H8tMwV6f850FdDMf/3bSqlL/M+/g+/v0UXAjcADBbwXIYQQxUACViGEOH90AmoDr2qt07TWdq11QQoW/aq1PqC1Tgb+AQ5orRdord3AJKBtKI1orWf529Fa66XAPHzBUiYX8L7W2qW1ng2kAs38AfcdwNv+8W8HxuTRlQtfcNZEa+3RWm/UWqdkO94CWAK84w94AnkK+D+t9S7/fX4EtMmcZdVaj9NaJ2it3VrrLwArvuAm02qt9TSttVdrnRksr9Baz9Zae4CxQOu8XzHuwheI3qy1XnfGsU+01ila6x3AdmCe1vpgtvco8z15EvhJa73W/1qMARz4Al201pO01tH+cU4A9uH7vGQ6orUe4R/zGHw/GGTOVnuBVkqpMK31Cf9YAnkCeFNrvcf/3m/RWifkc+9ncgHlgOaA8r8vJ4Kc2xGoprV+X2vt1FofBEbg++ElU6D350zvaa0ztNZb8AW/me/X3cBHWutErfUxfIG7EEKIs0wCViGEOH/Uwxd4uAt5/clsf84I8DgylEaUUj2VUmuUUqeUUkn4ZtyqZjsl4YwxpvvbroavGGBUtmNH8uhqLDAX+FMpFa2U+lQpZc52/H7gOL4ZtmAaAMP8abRJwClA4ZuxQ/lSl3f501OT8M1QZr+XqFwtQswZ92ZTea+ffBGYqLXeFuBYqO9JA2Bg5n34x1oP3w8YKKUeypYunIRvdjb7fWSNWWudOSsaqbVOwzfL+TRwQik1SynVPMh91MM3k1toWutF+GakhwMnlVI/K6XKBzm9AVD7jHt+nf8CbQj8/pzpzPcr8zWtfcb1obQlhBCimEnAKoQQ548ooH6Q4CgNCM/2uGZJDMCfgjoF+ByoobWuCMzGFwTmJw5w4wt8MtUPdrJ/hvY9rXULfKmqvYGHsp3yLhAP/OGfvQ0kCnhKa10x239hWutVyrde9TV8M22V/PeSfMa96BDuKz93AX2UUi8WoY0o4MMz7iNcaz3eP1s8AngOqOK/j+2E9p6gtZ6rtb4R36zrbn9bwcYQdM1xNnl+FrXW32it2wMt8aUGv5p5KEB/h86453Ja65uzNxfCeII5AdTN9jjXelwhhBAlTwJWIYQ4f6zD9yX7Y6VUhPIVJeriP7YZuEYpVV8pVQEYUkJjsOBLm40D3P7iPd1DudCfjvoX8K5SKlwp1QLfOtWAlFLXKaUu9QejKfjSST3ZTnHhCwYjgLH+NY9n+hEYopRq6W+zglLqLv+xcvgC6DjApJR6G9/azOIWjW/95ACl1LOFbGME8LRSqnPm2lOlVC+lVDl896/x3QfKV+SqVSiNKqVqKKVu9a9ldeBL3/YEOX0k8IFS6mL/GC5TSlUJcN5m4Hb/e9wE3zrmzP46+u/BjC+wtWfr7yS+9aSZ1gEpSqnX/AWSjP4CTR1DubcQTMT32aiklKqDL+AXQghxlknAKoQQ5wl/wHcL0AQ4ChzDl86J1no+MAHYCmzEV5ypJMZwGhiA78t+InAfvsJDoXoOX0pmDDAaX9GnYGriS/dNwVcUaikwLvsJWmsnvoJK1YFfzgxatdZTgU/wpRWn4Jt5zKyQOxffOtG9+FKT7ZRQWqjW+ii+oPU1la2CbgGu34BvHet3+F73/cAj/mM7gS/wFdE6CVwKrAyxaQMwEF9QfQq4FggWVH+J732fh+89GQWEBTjvK8DpH8sYchZDKo8v+E7E95on4Jutx99eC3/677Rsn/c2wCF8s+kj8aVtF4f38f0dOgQswPdZcxRT20IIIUKktC6ObCYhhBBCiPOXUuoZ4F6t9bWlPRYhhLiQyAyrEEIIIcQZlG//3i5KKYNSqhm+meappT0uIYS40ORVtVAIIYQQ4kJlwbefbSMgCd/eut+X6oiEEOICJCnBQgghhBBCCCHKJEkJFkIIIYQQQghRJpWplOCqVavqhg0blvYwhBBCCCGEEEKUgI0bN8ZrrauFen6ZClgbNmzIhg0bSnsYQgghhBBCCCFKgFLqSEHOl5RgIYQQQgghhBBlkgSsQgghhBBCCCHKJAlYhRBCCCGEEEKUSRKwCiGEEEIIIYQokyRgFUIIIYQQQghRJknAKoQQQgghhBCiTJKAVQghhBBCCCFEmSQBqxBCCCGEEEKIMqlYAlal1C9KqVil1PZsz1VWSs1XSu3z/79ScfQlhBBCCCGEEOLCUFwzrKOBHmc8NxhYqLW+GFjofyyEEEIIIYQQQoSkWAJWrfUy4NQZT98GjPH/eQzQpzj6EkIIIYQQQghxYSjJNaw1tNYnAPz/rx7oJKVUf6XUBqXUhri4uBIcjhBCCCGEEEKIc0mpF13SWv+ste6gte5QrVq10h6OEEIIIYQQQogyoiQD1pNKqVoA/v/HlmBfQgghhBBCCCHOMyUZsM4AHvb/+WFgegn2JYQQQgghhBDiPFNc29qMB1YDzZRSx5RSjwMfAzcqpfYBN/ofCyGEEEIIIYQQITEVRyNa635BDl1fHO0LIYQQQgghhLjwlHrRJSGEEEIIIYQQIhAJWIUQQgghhBBClEnFkhIsRFm0c/UeRr3+BztX7cVkMXLNXVfw6NB+VK1dubSHJoQQQgghhAiBzLCK89KmBVsZdMP7bF26E7fLjT3NwcJxy3im3SBOxSSW9vCEEEIIIYQQIZCAVZSajDQ7+zcf4uSRuGJtV2vN10//hCPDmeN5j9tLamIqf348Ld82Uk6dJiXhdLGOSwghhBBCCFEwkhIszjqPx8OoIb8zY/hcjGYjbqeb+pfUZfDY52nQol6R2z95JI5TJ5ICHnO7PCyZuIpnv3404PFty3fx7fMjidodDUDtxjX43zeP0+76S4s8LiGEEEIIIUTByAyrOOuGD/iFGd/PxZHhJD0lA6fdxYHNh3jhqjdJPBk40CwIr8cLSgU/7vYGfH7X2n0M6TmUQ1uP4na6cTvdHN11nLdv/ZgtS3YUeVxCCCGEEEKIgpGAVZxVyfEpzP11MY70nOm6WoPL7mL693OL3EfNRtWJqBAW8JjBaODyW9oHPPbzq7/lGheAI8PJjwPHFHlcQgghhBBCiIKRgFWcVbvX7sNsCZyJ7rS7WDd7U5H7MBgMPPvVo1jDLTmeVwpsEVbuf+OOXNdordmxak/QNg9uPcK/i7bxyvXvcku5B7irxuOMHDKOtJT0Io83mPjoU8z5dTFzRy+WQlFCCCGEEOKCJGtYxVlli7Ch8zgeFmkrln6uvftKTBYTP7/6G7FRCWitaXVVc57/9nFqXVQj4DUGgwGP1xO4Qa15s/f/4bS7ALCnOfjr61msnLae79d/TFhk4BndwtBa8/2LvzJ7xAIMRgNKgdvlpc/zPXnykwdQeaQ7CyGEEEIIcT6RgFWcVa2uao7RGHhi3xZhpdeTN+R63uv1snvtPlISUmncpiHV6lYJqa8ufTpx5W0dSU1Kw2QxERYRPBhWStHp5rasmbkRrXOH1MpoyApWM7kcbuKOxjN7xALueOmWkMYUimnfzuafUYty9Tfj+7nUblKT3v1vLLa+hBBCCCGEKMskJVicVUaTkVdHP4c13JJjptAabqVp+8Zcc9cVOc7fuWYv9zd4hsE3DeXjB4fx8MXP896dn2NPd4TUn1KKcpUi8wxWM/X/7CHCy4dhMPw3LmVQWMMtmMzGgNc4MpzMHb0kpLGEQmvN+P+biiPA/TnSHfzx4ZRi60sIIYQQQoiyTgJWcdZdcUsHvlz6Plfe1pHKtSpR/5K6PPnpA3w8701M5v8m/WOj4hnc/QPij58iI9VOWnIGLodvnevHD3xD9IEYXrr2LXra7qWH5V6ev2IIu9buK/S46l5ci+83fELXe7tgi7Rhi7DSpU8nuj/SNWAxpkwup7vQfZ7JaXeSHB98/9e4qAQ8niBpy0IIIYQQQpxnVKD0x9LSoUMHvWHDhtIehigjRrw2jqnDZgUMCA1GA9rr5cyPr9Fk4IOZQ+h4U5si9+9xe3i910dsWbwdT5CtcFDQ+6kbeeH7/kXuD8DlctM74v6gW++Elw9jetJvxdKXEEIIIYQQZ5tSaqPWukOo58sMqyizti7dEXT20uvJHawCeNxePn3o24DrUEPlcrpwu9zMG7OEHSt2Bw9WATQsm7SG04mphe4vu7HvTiRYSSWTxcjNAdb4CiGEEEIIcb6SokuizKpYvUKhrks5lcqRncdo2LJega7buXoPP7w8hr3r94NSWGxmHBnBU4EzOdId/DNyIXe/eluhxpvVToaDqd/MDhogW2wWHn7vniL1IURJcDjdOF1uIsOtUsVaCCGEEMVKAlZxVuzffIhdq/cSUSGcy2/pQHi5/LeBueHBa1g/59+8ZzgD0eAu4LrSHav28Fr397OtVdXY00Ir7OTIcLJqxvoiB6wxh2JRhuBf9k0WE7Zwa5H6EKI4RZ9M4quRC1m/5QgAlStG8NT9V3PTtS1KeWRCCCGEOF9IwCpKVPrpDN685f/Yu+EgaI3BaMD75I+8+PNTXHlrRxaMXcaGeZuJrBRBj0e7cenVl6CUwulwMe6DyXi9BU/tNZgMNGxVsNnVH14anWdhpfxYiyGQjKwUidsZvKBSZMWIIvchRHE5lZTGk6+N43SaI+vvaWzCaT77aR7pGQ769mhbyiMUQgghxPlAAlZRoj5+8Bt2r92Py5FzT9GvnvyJ4QN+we10Y09zoBQsn7yGq27vzKDRz7Fs0mpiDsWiCxGw9hvSJ0e14fw47U72bTpY4H4y2SKs9Hi0W6Gvz1SlViWatGnI7nX7c63BtYZbuPXZm4rchxDFZcLMjaRlOHP9qGR3uPnx9+X0vv4yzEG2gxJCCCGECJUErKLExEefYsPcLbmCVfAFiU77fzOaWoM9zcGKv9Zyea/2LBq/IuSU3CwK+g64mYfeOXvrPK1hFhq3acjVd3Tm2L4T7FrjS3vu0L01FpuFg1uPEHs0niq1K7F9xS6WTVqD0WKk+0Nd6XpvFyxWc472Bv32PAOueB1HugOn3fe62SKsNG7TiFuekYBVlB1L1+7FHSRdX2vYdziWFhfXOsujEkIIIcT5RgJWUWKO7YnGYjMHDFiDsac5+GvYLCJCSH+1Rdqo0aAazTo2plzlcvQbchtHd0Xz6SPfEX/8FC2ubMYtT3enSq1KebZjsVlo3qkJO1fvzbdPk8WIxWbB6/ESWSmCvgN60fPxbrzT9zM2L9qG0WREGRQej4fylctx+lQqymAg43QGyqCyZoz3rNvPlK//5uvlH+D1eNmydCdKKS67tgW/7Pqav3+cx+qZGwmLtNHz8eu59u4rCjRrLERJMxqCF5nXWmPIYz22EEIIIUSoZB9WUWKi9hznqbav4rKHHrAC1GhQjYffu4dv/jci4CyrMijad2+N1+1l67KdmMxGnHYXXk/O2R6TxYT2auo2q83F7RrRd8DNNG3fOGCfe9bvZ+B17+JID21W1xZhRXs1jS5rQMXq5dk0f2vWjGiozFYTzTo2Ye+GA5gsvmDU7fLw0Dt3cc+gPvle73a5ObrrOBabmToX15LqrOKsGjNpNWOmrMHpyr3uukK5MGaMegajUXZOE0IIIUROsg+ryGHTwm28ev173FOnP//r9BqLxq8o0h6loXK73Ix6/Q/cBZhdzaSMihZXNqVGg2qYLTlnFa3hVvoN6Uv9ZrXZsWp31hrYM4NV8FUK9rg9HNkRxYKxy3jp6rf465tZAfts1rEJXy59jzbdWmEwGlAGlWfFXnuaA0eGk70bD7Dm740FDlYBXA4321fsxml3kZ6SQXpKBs4MJ2Pfn8zC35fnee2M7+dwV40nePHqN3m63SAeavIcW5bsKPAYhCis229uS6UK4ZjOCEqtFhMD+98gwaoQQgghioXMsJ7H/hr2N7+8MT5H9VtbhJWud1/JwFHPlmjfv7zxB399PSvPfUyzp8hmZzQasEZYeeGH/myav5UlE1bicrioXLsSD7x1J9fd24W7az4Z0h6pZzKZjfy65xtqNqwe9BytNTN/mMsPL48p8PY4xaV24xqM2fddwGN//zyfH18ek2s22Bpu4esVQ2nSptHZGKIQJKWkM2L8CuYv24XD6abpRTXof9/VdGzdoLSHJoQQQogyqqAzrBKwnqeS41O4r/7TAWf+rOFWPl/0Ds07XVwifXs8Hm6v8ijpKRl5nmcwGtBaB60EbDAoTBYTnXq144XhT1CxekUA9m48wMvXvl3obWhueaY7A4Y/mec5UXuO89glLxaq/eIy1z0BwxnrBD0eD3fXepKU+NO5zldKcXnv9rw//bWzNUQhhBBCCCEKRFKCBQCrpq/HECQlz2V3smDcshLrOy05PaR1q4HSeHMc92qcdhdrZm5kSM+P8Hq9xB6NY+g9XxZpz9T1czbne07ModhSXRNqDbfkClYBYo/E4wwys6y1ZuuynSU9NCGEEEIIIc4aKTt6nspIteMJsuWE16tJS0ovsb4jyodjMBkhhHTaUPZZdTvdHN11jOEDfmXltLUknkwu0viS41OCj0drXE43MYfjQAEhJiBYwywAWWnKSvm29iisGg2ro7XOCpo3LdzGpM+nc3R3dJ7BusU/DiGEEEIIIc4HErCepy67tgUGY+AZwrBIGx1ualNifRtNRno+dh2zRy7MvxhRiEGh0+5i5o9zQwpw81OuUmSu5zLS7Ix+809mjViAI8OBLcKKQqFDjFhdTjeXXX0JCTGJnE5MI6mIQfXJw3F8+eSPrJ65geS4lJBeJ7PVRPeHrwV8+9wum7yGtbM2You0ceOD13Lp1ZcEnDU+tP0oU7+ZzZEdUdRvXoc+A26mceuGRRq/EOeqU0lpTJ2zmY3bj1KxfBi33diaTm0aShVuIYQQopTIGtbzQGYq6Mpp6wDo0qcTl13TgkE3vs/OVXtyBI1Gk4EqdSrz6+5vsFjNJTYmR4aD17p/wIEtR7Cn2kusn8Lo0KM1/zf7zazHHo+H/3V8jQObjxS5baPFiMeZe5uPwjAYFV5PaH8/LTYzVWpX5vsNn+C0Oxlw5RukxJ8mI9WOUr51y517teP1P17MkWo8d8xivn12JC6nG6/Hi8FowGwx8fRXj9C7/43Fch9CnCv2HjzJ829PwOX2ZG3XY7OZubZTE94ccLMErUIIIUQxkKJLFxiX08WbvT9m5+o9WVVjreFWmne6mLcmvczwAb+yfMoazFYTLoebll2aMXjsAKrUqpRnu+mnM/jz46nMGrGAjNMZ1Gteh2e+fIQ217UKeWxaazYv3s78sUuZP2Zpke6zOFnCLLw98WU692oPwNJJqxh6z1elPKqCMxgUtkgb5auU46ZHutJ3wM1EVIjg9Z4fsmnhNjzunIGzLcLK/4Y9Ro/HugGQGJvMAw2fCTgLbrGZGbPvW6rWqVLgccVGxXNs7wmq1a1MvWZ1CndzQpxlWmvueXYk0bG5syNsVjPvvNiLqzs1KYWRCSGEEOeXggaskhJ8jhv3wWR2rNydY4sXe5qDnWv2MvmLmQwZN4Dnvn2Mk4fjqFSzYr6BKoA93cH/Og3m+L4TWSm4B7cc4dXr3+Ph9+7mgbfuCnqtI8PB4e1R2CJt1G9eh7bdLsVsNbNs0uoiFUoqTs4MJ18//TNeryY1MRVPPsWfyiqvV9Pn+Zt59IN7s55Ljk9h85IduYJV8H0uJn/1d1bAunTCKt9i2wC0hoV/rOCeV28LeTypSWl8dN8wtizZjtlqxu1yU69ZHd6ePJBajWrkHk+6g+WT1xB9IIYaDatz7V2XExYZFnJ/QhSnvYdiSUwJvLbf7nAxZfYmCViFEEKIUiAB6zlMa8304XMC7kfqzHAy8fMZ7N14kM43t6X7w12JqBARUrv/jFpIdLZgNbsx70yk231XU7txzVxjGTd0MhM/nY7BaMDr9lKpZgUGjX6OuaMXl5lgNVP88VOlPYQiC4u00apLM8D3+m9ZsoO1szcFi0EBSIxJyvpzcnxK0IrDLoerQMWttNYM6fkh+/89hNvpzpq1PbjlMC90eZOxB77DGmbNOn/nmr283vNDPB4v9lQ7tkgbP7z0Kx/+PYRWV10Scr9CFJfklAwMhuB/eRJKsFCdEEIIIYKTbW3OYV6Pl/Tk4F+iPC4PG+dt4ZfXx/NI0wGcOHgypHZn/zwfbx7FjX59c3yu537/cAoTPpmOPc1BekoG9nQHJw7G8tpNQ1kwtuS20LlQGU0GqtWrQvvurUk8mcSTl77M27d9wtRvZudZ6Kr+Jf+l6F7c7iLCytkCnhcWaaN5x8Yhj2f3uv0c3n4U9xmVob1ejT3VzuI/V2U9Z0938HrPD0lLTs9a32xPtZOeksHrvT4i/XTe+/eKoklNczBzwVZGT17N8nX7cZ+jGQbF7aIGVXG5Aq8/NxoUlzavfZZHJIQQQgiQgPWcZjQZKV+1fL7n2dMdpCSc5qP7hwEQH32KfZsOkpqUFvD89JS8iyQd2Hw4x2On3cmET6dnraHNzmV34Q6QniqKpl7zOnyx5D0MBgPv9P2UY3tP+LYyCvKFG3xrm+97446sx517taNcpchcs0rKoDCYDKybvYmfXv2NQ9uP5jueXav3BkxDBt8WS5sXbct6vGzSajyewOdqr2bx+BX59icKZ9nafdz2xA8M+2Uxo/5cyQfDZnHXMz9zPNvM+4WqaqVIunRojMVszHXMbDZy760hL7URQgghRDGSgPUcd9fAW7CGW/M9z+vVHNh8mOcuf52HmjzHy13f5q4aj/NG74+wp+cMUC/r2iLPtpwOFw82/h89rPfSM6wfd1Z/PGCwCr5U0VD3MhWhu6pvZypWq8CRXcc4uPVI0GDRYDRgi7RhsZl5dOi9dMy2nZHRZOSrZe9Tv0VdbBFWwsuHYQ23ogwKt9PN/LHL+OvrWTzfeQgjXhub53giKoZjNAdeYWAwGihftVzW4+gDMdhTA39e7GkOovYcz+/2RSGciE3mva9n4XC6sTtcaA3pdhfxp9J4+f3JlKUCfKXljed70qlNIyxmI+FhFsLDLJSLtPHRoD7Ur10567yklHTGTlnDwKGT+Wj4HHbsPVGKoxZCCCHOb7KG9Rx358BbOLQ9iuVT1uD1eHOlZGbncrrYs34faMhMGl03+1/uqPoYXy59n2YdfQVFnvj4fhaOWx70C+zJw3E5HrsdwfsUJWPP+v3s//cQi8Yvx53HrGrVOpV56N27ueLWDpSvXC7X8er1qzFi65fs23SQ6P0xjH1/Ikd3R2etOfZ6vDgynMz4fi5trmtFxx5tA/bTpU8nvvnfyIDHzFYTNz1yXdbjmo1qEBZpIyPAdke2cCt1mtTK895F4UyduxmPN3f6r9aaU8lpbN11nNYt6pbCyEpWeoaT1HQHlSuEYzLlnj3NzmY18/HgPkSfTGLX/hjKR9po27Jejuv2HjzJ8+9MwO324nC6UUqxaOVu+tzUhuce7lrCdyOEEEJceGSG9RxnNBoZ/Nvz/LDxUx4b2i/HGsVcNAFnO512F690e5eEE4kAVKlVmTf+fBGVRwESUbrWz9nMM+0HMenzmUHTgJVSXHJFU2565LqAwWp2F7e7iCbtGnHiUFzAYlv2NAeTv/w76PWRFSMYMPwJrOGWHJ8bW4SVW5/tQePWDbOeu/auy4NWJ8ag6HbfVXmOVRTOwSPxuN2B16tqrYmKTjzLIypZicnpvPHZdHo9Mpx+z42i1yPD+WXCypCqgteuUZHruzSnY+uGOYJVrTWDP55GWroTh4rWcb8AACAASURBVP/HQa01doebaXM3symE9HkhhBBCFIwErOeJ+s3rcNcrtzJw1LNYwyy5judV/RJ8VWFnfj8H8KX8Ht11HINBPh7nMkuYmdsH3AxAWko6w1/4hdsqPER309081uIFlk5aneP8+GOnMFuCJ11sWbKdd+/4jDd6f8TIweOIORyb43jb6y/lqtsvJ7JiBLYIK03aNeLtSQPp/+mDOc4Liwxj6MzBhJWzYYvwpbNbw63YIqy8P21QyNWsRcHUr1MZkzHw32mDMlCzev7r4c8VDoeL/oN/Z+X6A7jcHhxON2kZTv6Yvp7Pf55f6Ha374nmdFrgNf4Op5sp//xb6LaFEEIIEZikBJ9nWlzelBd/fophT/+MwWBAa43H7aFSjYqcPBIX9DqP28uWZbvYv/kQr1z3Lml5VB8W5waLzcL04XMxmY18+shwog/E4PKnb0ftjuazR4cTdyyBO1/qDUDtJjVxOoJXGPa4vaycug6ATQu2Me3bfxg8bgBX9e3MjlV7GNJjKC6nOyst/dieaBaNX0GHm9qgzphRveyaFoyP+onF41dybO9x6jSpRbf7rpJgtQT1uak10+dtCVgVOCLcQrtW9Yu1v627j/Pj2GVs3xuNyWjgms4X8/T9V1OzeoVi7edMWmvGTl1HwqnUXPdqd7iZu3Qnj919JdWq5J11EMippDQMeexdHJdwulBjFkIIIURwqiwV2ujQoYPesGFDaQ/jvGBPd7B+zmbsqXZaXdWcuGMJDO4xFFceW5506duJrUt3cvpU6lkcqShtlapX4J7XbqPvC714s/fH/LtoW55robOzhln4I+pH+l86MCulPDtbhJU3/3yJzr3aF/ewRSHMWrSdL/wzjE6XhzCbGbPJyLfv30PjBtWKrZ8NW4/w2v9NzUqbBTAoRWSEldFfPkz1QgSLwaSlO5i5YBsLV+3G4/aSkJRGYnJ60K25wmxmBj55Az26tixwX1HRiTwycEyO+8pkMhq4rXtrXnri+gK3K4QQQlxIlFIbtdYhl9+XGdbzlC3cytW3d856XLNRdbr06cSSP1cGvSZ6f0zWvpjiwpEYm8yvb01g/+bDDPl9AC93fYfD20Jbi6eU4s+Pp5GeGnjvVHuagxk/zM0zYHW73Kycuo6lE1ehDIqu917Flbd2wJhPgRxRcL26taJT6wbMWbKDuFOpNGtcg+u7NMdmNaO1Jj3DidlsxBKk4nMotNZ89tP8XEGdV2vSMhyMmbyGV5+6sai3AvhmPJ8YNI7k0xkBg8hAlFKYA2xdE4p6tSvRsmlttu0+huuM9cAmk4G7e8sPM0IIIURxk4D1AqGUYsi4AdRsWI0Jn04PWFjnUIhBijj/ONIdLJu0muvvu5q4qPiQr7OnO5j0+QwMQdZGAiTGJAc9ln46g5eufosTB09mVQ1eN2cz9f37zNpC2LJJFEy1KuV48I7Lczw3f/kufvp9OfH+7IrObRvx0uPdCpW+G5twOmhqrMejWbpmb7EFrF+PWkRCUioeT+iZQm6Pl05tGoZ8vterWbF+P9PmbSEpOZ1LLq5JeoaDw1EJKINCKYVS8P7Lt1CnZsUC34PD4SImLoUK5cOoWD68wNcLIYQQ5zsJWC8gBoOBR4f2o8WVzRh6z1c4M5ylPSRRhjgdLn5+dSwZKYFnS/PizaPyat1mwbepGTl4HFF7jmetrQWwp9o5vP0oY96ZwFOfPVTgsYiCmTZvC9+NXow923uwetNBduyNZuzXj1CpgOuKtVfnWrOcXbBU3YJyuz0sX7e/QMEqQMfWDXjurT85HJVAWJiF3t1a8dg9XQgPUKzO69W89/XfrNp4kAz/coqDUfGYTUZe7n8DWmsqlAujU5uGBZ6Vdnu8/DB2KdPnbcGgFC63l9Yt6vL6cz2KNWVaCCGEONdJGdgLyOqZG7i37lN8eK8EqyIADUd2HSu2gCLT3g0HAz7v9XqZN3pJjmA1k9PuYvaIBcU6DpGby+Xhh7FLcwSr4AvU0tKdTPx7Y0jteDzerJTcGtXKU6FcWMDzDAbFlR0uKtqg/RxOd9C9ovOycv0BDhyJx+PVpKY5mDJnM0+//kfAlOKVGw7kCFYB3G4vGXYXP/2+nJ5dW3FVxyaFSqH+ePgcps3dgt3hJt3uwuX28O+OKPq/No50+fdZCCGEyCIB6wVi+4pdfNjvKxJjknCky5chEZjXm/celZYwMxRwe964Ywm5tsABX1DqymPdYXpKRr7jEQWXmJzG0ehTOF1u9h2OJVjM53J7WLx6X75tfTBsFjfcN4wb7xvGXc/8zIIVu3jx8W5Yz9giSSkIs5p59K4rCzVut8fLrv0n2LnvBBkZThat2lOods7kcnmIPpnE/OW7ch2bOndzjmA1u4wMJzv2Rheqz5i4FBat2pMrSPZ4vKSmO5m7dGeh2hVCCCHOR5ISfIEYOeR3CVRFvkxmEx6XO2AQ06BFXR7+4F4+fuCbAs/QZwQo5mUNs1ChajkSTwZe41q9ftUyvxdwYnIaqzcewu3x0rF1A2qV8JYtRXE8JokPv/uHXftjMBkNKKW4vkuzgOvZMxnz2L85Ld3BE4PGEZ+YhsefEn4iNoVPfphH/35XM/TVW/luzBKORSeCUrRtWY8XH+8WdJ1neoaTect2svvASWpULcd1VzZl76E40tIcJJ/OYMLMjXj8P2DYHS4MCtwFTAcOxu5wM2fJDnpff2mO55Py2N5LGRQphSxSt2XnMYxGA7g8AcbiYsX6/fTt0YaDR+M4HpNMzerlubhh9UL1JYQQQpzrJGAtJkd2RrF58Q4sNjNX3NqBitXKzhfXSV/MYMfK4pmNEOc3s8WEUuRK07WGWbjsmkv4v/uH4XUXbNbT5XBRt2nudaxKKfoN6cuo18fjSHfk7C/cyv1v3lHwGziLRk1Yye9T12E0+vY79no111/VnMHP3OQLRsqQpJR0nnxtHKfTHGitcfkDpfnLdwVNAbeYjXS/+pKgbc6Yv5WklIysYDWT3eHmpz+WM2v0//h92GOkZzgxGg25Zlyz23vwJAPenYjb7fUFowbFqAmrMJuNaK1xB/jMFffce6D9adu0rMfBqPiA/btcHpo2KlwQabEY81znqwyKx18dy5FjCRhNBjweL3VqVOTjIX3L9I8iQgghREmQgLWInA4XQ+/5ko3zt4LWGIwGvnt+FA+9ezf3DOpTYv16PB6mDpvNnx9PJeVUKuUqR9JvcF/ueKl3ji9CK6auZczbE0psHOL8kpFqx2IzU6lmRVISTqOAOk1rccP91zBu6JQ89/ENRmuNx+3FnLumDX2ev5kTB2P5+6f5GM0GFAq3y0Of53rQ8/Gyu5/l/OW7GD99PU6XJ8cs2eJVe6hVrTyP3dOlFEeX29Q5W7A7XbnWfNodbgwGhQKyHzEaDVQsH07fnm2znktJtTNl9ibm+YPcjAxn0K1kHE43c5bspM9NrQMWM8rO7fEycOgUUtP++9EiM4h2BZiBLAlKwdWdmuR6/u5e7Zi5YGuugNViNnJVxyZUK2RxpM5tGuUK9DPZrCb2HjxJckoGHq8GfzLDoWMJPPvmeCZ9/yQm2fJJCCHEBUQC1iL68eXRbJy/NVeK5Nj3J9OgRT0uL4F9+bTWvH3rJ6yb82/Wt8yU+NP89MpvrPl7A58veg+A6AMxfPLQtzikgIcoAKfdhUpO56tlH1CvWW0iK0bw0rVv55oFDZXBYMBoCjzjqJTi2a8f5e5Bt7Fp/laUUnS4qTWVahR8e5Cz6ddJq3MVKgJfADjh7408dOcVmMrQLOvKDftxOgMHf2fOsBoMip5dW/DU/ddQPtIGQGJyOo+/OpaklHRfkB6CURNWclv3y/KcSQTYsPUIDkfBfwgpTlpDdExSrudrVq/AF2/dyZufTed0mgO0bz/ZK9pfxBvP9QjSlmbPwZPExKZQp1bFgKm84WEWnn/0Or79dXGOoN9mNVGvViWOxST5gtVsvF7N6VQ7y9btp9uVzYp4x0IIIcS5QwLWIshIszNv9JKA6/kc6Q7++HBKiQSse9bvZ/2czTmnRPy2LNnJ6pkbaN6pCc91HoI9Le8gI6xcGG2ua8nqGRuKfZzi3OVyulnx11qe/OQBAKJ2HStUO0pB2+svxWwx53le1dqV6f5w10L1URqiT+YObjK5XB4Sk9OJCLMQZjPnG7CdDVZr3q9/dhazkWYX1aRShf/2BB0xfgWnktICps0GY3e42LHvBK2a1s7zvJNxKXgKUe23uM1dtpN+t3UkItya497tdhdp6U4U4HR7sFlNbNx6lN+nrSMywkaLi2vRsmktlFIcj0li0Ed/cTL+NEaDwuP1Uq92ZT4Z0jfXVjV9urembs2K/DpxNQePxlOhvI07b25HYlIaY6asDThGu8PNguW7JGAVQghxQZGAtQjijyVgyGMWJWpP4SpI5mfSFzPz3M5h7HuT6HBTazJS899P0+10kRKfQq6cQHFB83q8nE5MzXpcqWZFkuNPF6gNZVCElwvj2a8fLe7hlbrykWGcSkoLeMzt8XL3syPwejXlI208fOfl3NGzbciB64nYZGYu2MqJ2GSaNqpBr26tKB9km5hQ9b7+UvYcOIk9hJlMu8PNiPEr6NG1BWE2Xzrv/OW7ChSsgi9wPxmXkm/AWq92pTLxb4/d4ebhl8fg9WqaNa7BkGdvokL5MF7/dHqOWVDfzLqbXyauxmQyYDIaaVi3Mp8M7sszb4wnMTk9x7/PB4/E8dybfzL+u8dzrW3ucFkDOlzWIMdzk2Ztwmo24XAFTrdetfEgJ2KTZS2rEEKIC0bZyVk7B1WsXgF3HulxlWoW/xeKwzuiWDE18K/vmeKOJbBy2jrcQVIAs3M53Oxcs69MfGEUZYfRbKR+8zpZj+94sTe2CGuB2mjaoTE/bPo0RzvBHNsbzcTPpvPHR3+xZ8OBAo/3bLujZ5ugRYQyixp5PF4Sk9P5cdwyfhi7LKR2Zy3cxgMv/Mof09czf/luRv25kjue/pltu48Xabw3dGlOtSqRIZ+fkmrn6dfHZ6XqOvPYfigYt8fLzAVb8z2vbct6Bd0pKQeDQWHxFwsrKofTjcvtYcfeaPoP+YOJf2/EGSRwBLKKRO05eJLn352A3e7M9WOix6s5GZ/Cz38sD2lN7vVdmuHRef84MHn2JsA30//VyIU8/PJoBrwzgSWr9xb7PspCCCFEaZOAtQjKVYqkfffWmMy5C2DYIqzc8WLvYu/zw35f51ulNSXhNGkpwbdjOFNe21qIC5PH5eHXN8ez8I/lAHR/pCude7XDGp53AZ3sbn3mJmo1qpHnOVprvnl2BE+1eYVf3xzPmHcmMLDrOwzpMRRnKa9rzMt9t3WiZdNahNn+S7W1mI0oRa4tgewON5NnbyIxjy1SwLftzJcjF+JwurOK/NidbjLsLl796K9CFSDyejXT5m7hvgG/EBWdWKBrj51IZI5/P9CmF+X9Pgazeecxjp1IRGtN/KlUEhJzzko7HL57CzabGAqjwcCMkc9QPrJos9DZae0L0qfN3RJSAKg1HD2eSHqQomQer2bC3xt57NXfchSXCqRyxQh6dm0Z9Ljb42X3/hi27DrGwy+PYfq8LRw4Es+m7VEM/fYf3vpihgStQgghzisSsBbRwJFPU71+VcL8xUmU8gWrnXq2o+cToVU5tac7WDV9PYv/XElsVDwASyeuon/rgdxS7gEebPw/pn33D1F7jnPiQEy+7Xk9XlITA6crChEqp93Fl0/8QHJ8CgaDgcFjB9CoVf2gBZSyMxgNdOnTMd/z/hm1iPljl+K0u3C7PHg9XhzpDrYu38nIweOK4zZKhNls5Ot37uaDV27lpmsuoduVTWlQt3LA/WsBTEYDG7cdzbPNmQu2Bq0c6/V6WbXpYIHH+ckPc/luzGJOxAbe6zYvDqeb4b8tISY2mf73XYUlwA9z+XG7vfw6aTW3P/UTdzz1E7f3/5F7/jeSTf7X4vMRC9i0PSro6xaKiHALkRFW+nRvHXDW22BQGPLYTzYYl9tT6H1WA3G7vRw7kcS3oxfne273a1pgCzKDr5SietVyvP3FTDLsrhyp2naHi3X/Hmb5un3FNm4hhBCitMka1iKqWK0CI3d8xfIpa1k7ayNhkTZueOAaWnZpHtKatXm/LeGbZ0diNPn2cnQ7PdS9uBYnDp7E7q/KGnMolpGDf2fFX2tRxtC+eDnSpTKwKAZKsfjPlfR5rieLx6/k8I4oPCHsw9rn+ZuJqBCR9dhpdzL318X8M2ohjgwnnW5uxx0v9mLCp9MCFgZzZrj4Z+RCnvzkgXwLNpUWg0FxedtGXN62EQCDPvqLfYfiAp+sVL4pq9Enk4KuE3W7vcQnFGwN8aGoeBas2B1065lQpGe4ePCl0dSsVgFvIaPKuf5Z2kzHY5J46YPJfDqkLwtX7A656nAw9WpV4u+F27izVzvWbz3Coah4MvwznWE2M5UrRlCnRgXWbTlSpH6Kg8vtYd7yXbz61I15bk3T+pK6REZYsQd470wmAweOxAVdQ53hcDF1zmauvbxpsY1bCCGEKE0SsBYDs8VMt35X0a3fVQW6btvyXXzz7IhcweWh7blnYhzpDnav248jo3BbiwhRGM4MJwc2HwZg+vdz8q06nWn2iPnc/+btlK9cDnu6g5eufouoPdFZW+NEHzjJPyMX5rnlkterSY4/TdXalYt8H2dD92ta8O+OqKxgKTu320PHM4rrnOniRjVYueFgwADTaDRQv06VHM95PF7Wbz1CTGwydWpWpP2lDXLMJC5ftx+3p+j7mGbYXRzyZ34UF4/Hy+c/L8BkMhY5YN22J5p9h+MY9ssihr56KxkZLuYu3cmJuGROp9qJiUsu1AxzSXG5PExfsJU7erQNeo7BoPh4SF9eeGcibo8Xh9ON0aBAKbxezcGjCXn2kXQ6/4J7QgghxLlCAtZSNO6DyQWaCQ20fY4QJW3huGVUrF6epJOhf+l32l3M/XUxdw28lanDZnF09zGcGf8Fcm6nG4/LjdEc/J8gBZSvHHqhoNJ2beeL+X3qOo4cT8gRhNmsZu7v0zHfSr+9r2/Fb1PW5HpeKUXF8mE0u6gG/yzZQVq6gwrlbHw7eil2hxOPR2MwKMpF2Pjq7TtpUNcX2LrdnjK9lrE4g8jM6sdvfDqDid8/wZylOzgcFY/bUzbv//sxS2nfqj4N61YJek7zxjWZ+MOT/L1wG9t2H8dsMrJi/QFc7rwDfJPJQNuW9Yp7yEIIIUSpkTWspejA5kMFOl9rLdV8xVnncrqZ+NkMYo+FPsvm9XjZs95X7Xf2iAU5gtVMWvvOs4blLuRksZm5/sFrsNhCL/JU2sxmI8OH3svtPdoQ4b+n2jUqMLD/DTx695X5Xl+pQgSfvn47EeEWwm1mzCYj4TYzNaqW4/Yebenb/0e+HLGA4WOW8N7XszmVlEZ6hguHvzBT3KnTPPf2hKziTJ3aNsJaRtOpS4rWmh9/X87KDQfKbLAKvh8Tpsz+N9/zKpQL4/4+nfh4cF8qV4wIusY5O7PJyD0lsP+3EEIIUVpkhrUUla9avkB7WxpNRjz5/LouREnwFnAPThRUr18VgPTTwQvXWGxmLru2JVuW7MBpd6K9GlukjUat6vH0Fw8XZcilIjzMwnOPXMczD16L3eEiPMwS8v6r4NviZcaoZ1m54QDxp1JpWLcKkRFWnn9nQr5rUbX2zTQuX7+fblc2o3H9qlzcqBo7957AU4ZnWouTw+lmwfJdIRVxMpkMeDzeIhV8KiyPV3P4WN5pvQCpaQ7+WbKDTduPsv9wXJ7riM0mIxUrhPHBwFupKXu0CiGEOI9IwFqKrurbiYmfzQg5CPV4PJhtZlxBtk4QoqwwmU3c7K+S3aBlXbYt2xXwPIPBwDtTXuHw9qMsm7wGt9NN517taHNdK1KT0pj4www2zN1MRIVwej5+PR1uao3BUHYTQ1LTHHw/dilzlu7E4/FSLsLKg7d35u7e7UMOXK0WE92ubJb1+O0vZuIMYU9l8K033XfoJBu2HmHu0p1orYMGqwaDIsxqJt3uLJWgraSEuibWoBS/f/cE/Z4fddZTp41GRaP6wdOBAY4cS+CZN8fjdHqyUp6DMZuNvPrUjfTs2rJAP5AIIYQQ5wIJWIsgKS6ZWT/PZ8O8rZSrFEGvJ2+gY8+2+X6h9ng8fPLQt6yavj5XsGoNs+ByuvAGSmfTYDQZ8BgNBZ/xEuIsevTDfuzfcpgP7v6CqD3RAc+xhVu5e9BtWKxmmrZvTNP2jbOOHdt3gheufB1HhjNrnfeGeVtof+NlvD1pYJkMWl0uD8+88QfHTiRlrTNMSslgxPgVxMSl8MJj3QrV7v7Dsb7lACGwWU0sWb2Pk/Ep+QZuXq+mbat67D0US2wBMj3OFx6vJiEpjcb1q7LvcJDqziXEZDRyZ892eZ7z+mfTOZ1qz/fHBLPJwMUNq3Pzda2KcYRCCCFE2SEBayEd3hHFS1e/hdPuxOmf8fx34Tbad2+d7xfqad/+w6rp63MVXFIGRdOOjYPORgHYU6VKsCjbDEYDY96egMflCZo9YLaYuP3FXtz3+u0Bj3/Y7ytOn0rLEajZU+1snLeFRX+s4IYHrimRsRfFkjV7iYlLyVUUx+5wM23eFu7v24mqlQpeRKpG1fIcjU4M6Vyny0PUidDOBTidaif9Ai3m5vF4+XrkQq7s0PisBaxmkxGlYNDT3alfJ3D1a4fTzR/T1nH0eGLQYNVgUITZLLhcHlq3qMN7L99SgqMWQgghSpcErIU09J4vSUtOy/GFwp7mYOO8LSz5cyXd7rs66LWTv5gZsDqw9mr2+gvVhEopzqt0PnHu83q8+Va0btCyHp17t8flcOUqrHTi4EmO7joecFbRnuZg6jezylzAqrVm0ao9Abe0ATAZDazfcoSeXVtmPZdyOoOTCaepXb0CEeHWoG3ffUt7tu2JzjMt1GhQeLy6wKmtJpMhq0jThWjvoVj2Hoot8X46tWlA84tqUrlSBNd3aU6lCuEBz0vPcPLMG+OJij6V56x6zWrlGfjkDdSvU5lasl5VCCHEeU4C1kKI2nOcmMOxAQNFe5qDqd/+k2fAeiomKegxVz6FVc4kwao4F+3ffIghPYaivZqH3ruHO1/qnXUsMTYZs8UUNOhNik05W8PM1+FjCXw3ZgnrNh/ON1jMXFkYm3Ca59/+k+Mx/23rUj7SRrlIKzWrVeCuXu3o0qFx1lrEy9s24rYbL2Py7E1B16MWtqiSwWDMt5iTKBqr1cTnb9yZY4/cYH6bvIaj0afy/BHBYFBc2rwOnds2Ks5hCiGEEGWWBKyFkBx/GpPZSLDk3OS4vL9QV65ZkfjjpwIeM5pkfaq4AGhIT8kAYPRbf2ILt1C3aW1W/LUGp92FI0iwqpTKsda1NB09for+g38nI4SiRR6Pl85tG+F0eej33KhcQWJKqp2UVDvHY5LZsfcEN17VnEHPdEcphVKK5x+9jsWr9xKbULxrTc3msrcW+HxiMRu577aOOYJVrTX7D8fhcLpp0rAaNut/Ww/NXLg13xlvs8lIv1s7ltiYhRBCiLJGAtZCqN+8Di5H4FkJg0HRvFOTPK+/65Vb+eWN8TjSc4a8ljALTdtfxPYVu4ttrEKUdY50B8MH/ILRbMKZ4fs7oYwGlEGhz5g5tISZuXdI39IYZi4//r4spGDVZjVxd+/2VKoQzsg/V+Y7o2l3uPh70TaWr9+fVTipXu1KpKRmFNfQs6zacLDY2xT/qVwxggdv75z1eP2WI3z43T+kpTswKF8a9/19O/LInVeglCItwFKR7JSC/vddRZOG1Up66EIIIUSZIQFrIZSvUo6u93ZhyYRVudIWzTYz97zWJ8/r+zzfkz0bDrDyr7W4nG60V2MNt9CgZT12r9tfkkMXokxyuzy4s80sabcXZVAYjAas4b69TLVXM3DUszTrUDZmWNdsOpRvsFqpQhj9+11N7xsuBWDRytB+jNLaV2E4054DJ2W7knOQx+vFYjaRlu7gtylrGT9jfa7U8d+nrsdoMOJ2e3Dnk12jNYz6cyUdWzfgovoStAohhLgwSMBaSC/80B9HuoPVMzZgsvheRqUUg8Y8R+PWDfO81mAwMGTsAA4NOsKKaevwur10urkdf3w4BXc+sy/KoLD4U8g8Hi/a68XjlhRiUTYUZxEw7dVYIy288H1/wsuH0f7Gy3IVaCpNodymw+Fm9JQ1XNH+IqpWjsRkMha+P1mwfs6JS0hl0EdTWPvv4aDrjO0OF79MWInHG9q/4+l2F8N+Wcywd+8uzqEKIYQQZZYqS1+COnTooDds2FDawyiQ2KNx7Fq7n/DyYbS5riVmizn/i4LoV++poGtbASw2M498cC82f0XRjj3bEr0/hvfv+oK0lPTQvkELUYKUUaED7SFc2PaUYlbG70X6e1VSBn88lZUbDuQboBuNBlpeXIvvP+zHlH828dXIRYXuU6qCC/AtPVk0/sWsH0Bi4lLYcyCGchE2LmtRF5NR1iYLIYQou5RSG7XWHUI9X2ZYi6h6/WpUL6bUrEp5FGMCsEXYuPXZm7CG+QLWtJR0Vk5f76ssLF9iRVlQwM+hyWLC7Qr++S1fJbLUg9UMu5PUNAdhNjMr1h9g1/4YMhwuml5UnQ1bj+a53Qz4MiF2HzxJTGwyfW9qy+iJq0lMKdx6VK19AbAnSOqo2WTMtQ+sOD95vBqPw8UH38xm1caDmE1GNGA2GXjv5VvocFmDXNccPBrHnKU7SU2107ZVfbpe3hSzufCz/kIIIcTZIAFrKYs5HEv8sQRqNa7J7S/0YtgzP2NPC1x/OP10BsOeGcFTnz+ELcLKEy1fIj76lASrosw4s0hSXpSCvgNuxmw1MeWrv3PtTWwNt3LXK7cW9xBDlnw6g89/ms8K/97ILrcn1wyn0aioV6sSx08m5bmtjdlk5GT8aWpWr0DvGy9j3F9rCzVTajAozKbg2bhfPwAAIABJREFUAasEqxeGJg2rYbWYeOerv1m18SBOlyerQBf4Zv9/+fwh6teunPXc978tZco//+Jye/B6NfNX7ObHccv44aP7qF6lXKHG4XS5WbnhACdiU6hbsyJXtr+oSGnvQgghRCASsJaS+OMJDL33K/ZtPIjZasbpcNHuhstof+NlbJy/NWDQ6na6WTx+BduW7aJK7bxnY4Uo6yIqRtD/0wfxer3ERSWwdNJqtH8dn1KKa+68vNQCVofTzVODfycmLiVHIZwzg0yPRxObkMKnQ25n0qyNrN18OGB7LpebOjUrkmF3MunvjYVO6/V6NfYgFcrFhcFqMfHcw11JSExj2dp9AbfBcbk8/DljA4Oe7g7A6o0H+WvO5hwVqjPsLpxON29/MZMfP7qvwOPYsfcErwydjMercTjdWC1GLGYzw969i8YNpCCUEEKI4iMBaylwOly80OVN4o+fwuvx4rT7Ugo3zdtCo9YNGPL7C8z/bSkrp67LVWjF7fKQcCKRmCOxpTF0IYpNi8ubAr4iZINGP8d9r9/O2lmb0Fpzee/21G1au8T69no1Ho83aDrk4lV7SEhKy7dqK4DD6WHctLU8ee9VbNl5DPsZhdPMJgPtL2tA1cqR/LsjCoNB1heKwlEKPnjlVtq1qs/6LYexmI0BA1aPV7Nt9/Gsx3/O3BAwdd3j1ew7FEv0ySRq16j4/+zdd3gU5fbA8e+UbakkIXRC7016BxUEqSIKqGDFgl0vV72W37Vd9drbVewNG6AgCAoiHUTpJXRCL0lIL9um/f4IREJ2UyANeD/PwyO7MzvzBrO7c+Y97zklHofb4+cfz88o0IbH7TFxezQeeHo6P308SaQaC4IgCGVGBKyVYOXMv8hOy8E842JY8+vsXpvAs9e8hmWaQWdhtGLWzAlCVWd32Rj/1DUFnguPDqPvNd2JqRuNopTPxW56ppv3vlrKopW70A2TWrHh3HF9Hwb1a11gv2V/7cHjLfn77MDhVDq0rsekCf14f+oygPwUTVmW6dGxIYaR1+LEr4kZUiEwmy2vvU2wz36X045+Mu07ItxVZBp6VGRI/t+PJ2cG3U9VFZJTsksVsC5csQMjSHE1v2awfM0eBvRuWeLjCYIgCEJRRMBaCTYu2oonxxt0+5mBrCBcSBwhDh764A5a92wBwIFth3njjg/Ys2EfiiLjDHVwy3+uZ/idV5Tped0eP3c89jUn0nLy14AeT87i5Q9+Iz3TzbgRfxers5dydii6WigA1w7rhM2u8OYni/K3+fw6U6YuZ+3mgzw3eQS6aEMlBBEZ7iQn1x+0kJdlWeSe7P3dvFENqkW4At5YcTpsXDOkY/7jRvWrczw5M2AgfCpdvTT2HUoJOkav18/hY+mlOp4gCIIgFEXkplWCsKhQZNF2QLgIyarM9MSPad+vNfu3HuTI7qM82PtJdv61G92v4/P4yUzJ5oN/fMmsd38p03PPXbSV9Ex3oYJFPp/Ox9+tKnABPrh/a1zOklUndthVrjsZ7Pr8Ou99sbRQUOr16azfeoi3Pl0U6BCCAEBKWm6RVad9fp0lq3cx+7fNeH0az08eSYjLXiD91uW00atzY/p2awbkBbnXjeyC3Vb4/rRNlenULo7YUhZdql0jEoc98P1up9NGbExYqY4nCIIgCEURM6yVYNBN/fn5/QX4PP7idxaE84g9xI7u1zEDzCJKskSnAe34R79/c2jHkbx1pAHW3wH43D6+/Pc0ht91RZm1tVm4ckeBojOnU2SJLTuO0u2ShgD06NiYVk1rsW338aCvgbyZ2N5dmnDlpW0AWLflIJIkBdzX49WYs3Druf0QwkXNNC3+WLePjfGH+ejbFUy+YyD333IpO/YmknDwBJERLq66ogO9OjcmO9fHh98sZ/7S7fj8OpERLnTdQD2ZdmxTVRrWi+bph4aVehyD+7fm4+9WBt1+2cnsCUEQBEEoCyJgrQSN2jVg+N2DmPfhwqAtbAThfOR3B78JIysy8at24S0iHf50lmlxIP4wzTo1LqvhBSflFbQ5RZYlXn/qWqb9vI4fft1IVo6XOjUiadW0JsmpOWTneGgUF8uIAe3o0LpefpDq8WpYos+UUM48Xg2PV+PpN+fictoxDJNWTWvx1H1DiAh34fVp3PWvbzh+IjN/tj8zy4PDrtKjY2PaNKtFmxZ1aN+ybtAbLEWJigzh6YeG8exb87AsC79m4LCrSJLEi4+OIsRlL+sfWRAEQbiIiYC1ktz16k1ccmlbvntpJttX767s4QhCuTM0I+iMaiCmaWJzlM3sKsCgvq3YdygFX4C2MIZh0a5l3QLP2WwKE0Z3Z8Lo7iU+R9sWdcQaVaHCWFbe2myA+N3H+OcLP/LRfycwf+k2TqTlFPpd9Pl1Vq3by+P3DCYs1HFO5+7XvRnT3r+dXxbHc/hYGo3qV2fo5W2pFhFS/IsFQRAEoRTEQspKIkkSPYZ35tJxvbGXcK2cIFxMwqLCaNC6Xpkdb9jl7YiJDEVVC37sOR0qd43vg7MMguNasRH07d6s0Pq+s5jEEoRS0XWTfYdS2JmQyMIVO4KuhdV1k8demlmoZdrZqB4Vxk3X9ODJ+4dyw6huIlgVBEEQyoWYYS1nB7cfZv1vW5BVmV4ju1AjrmBDdU+OF70Us06CcDFwhNiZ/MndACz/YTWz3v6FtMR0mndpwrjHRtH0kkalPmaIy87Hr0xgytTl/LZ8O5pmULdWNe64oc9Zt+DweP1omkF4mJOMLA8z529k/6EUXE4bmm5gU2X8WvA2JYJQpizYvS85aMuZU7bvSWRD/GE6t4uroIEJgiAIwtkTAWs50TWdF8e/zV/zNmCZJpIk8dGjUxkxaRCTXr8ZSZJIT87EFebEZldFASZBOKnzFe255fnraNG1KS/f/D9Wzforf6338f3JrP55HY9+cR/9ru0Z9BiWZaFrOqpNLbBGLzLcxb/uGcxjdw/CNC2UElTrXrv5AF/+8CcHjqQSExXGNUM6UrtGBF/+8Cfxu46BlLemLzvHh8+v589cKYqMzy9uRgkVR1ZkIsOdaHrRvX413eCXJfEiYBUEQRDOCyJgLSdfPjOdNfM24D8jEP3l49+p1agGy2esZsdfe7A7bfi8IlgVBGeIg5ueHcuYySMB2Lh4a4FgFfIKMfncfp4f+waOEDtterXkluevo1X3vBYefp/G18/PYNY7v+LN8aLYFFr1aM4/P72buk1r5x9HkiQUpfg83elz1/PRtyvwnlz3mpHl4ZUPfiu034nUnELPndk+RxDKm9vj57WPFpKR6Sl23xxR8E8QBEE4T4g1rOXA0A3m/G9+wFlTb66P9x/8nPiVOzE0A0+2F1FUVBCgfsu69B/bK//xr58uLrKKts/tZ8PvW3jk8mdY99tmLMviqREvMe2V2fmViA3NIH7FDm5r9RB/zVtfqvFkZnv44Ovl+cGqIJwP0jM9xX6lOB02enQsfVq9IAiCIFQGEbCWg6y0HHRNXOQKQmkkbD7APZ0fJS0xHYDstOyg+1qSlH9R7vP4eeOOKWxeto1tq3YG7AFrGibPj30DT27JWuoArFqbUKKUYUE434S47Azu37qyhyEIgiAIJSKuxspBaGSIKAsqCKVkGia5WR5mvP4zAJ0GtscerJ+jBHrjmvkPs9NymP/5EvyewJVRIa9NzurZa0s8Hq9fwzRFWq9wYYmrE8VHL90geqUKgiAI5w0RsJYDu8NGn6u7gYhZzxvRloc2VgrNrHQUSwQplUX36yybvhqAK2+7HGSpUHqjJUuYNSLR29THOjkDKskSVjFrRg3dJD0ps8RjuaR1fVHdV7hgOOwqfbs15Zt3bqNWjcjKHo4gCIIglJgoulROThxJFWtTzwNNrHTGs4f2nGAvcYTgJ5pMZlodmUVtLKnkKaRCWcl744RHhRExugfJs9dAri8va8E0MepEoXVsDJaFGRWGkpKFoigMvvUyls9YHbRNlM1ho3GHBiUagc+v8+UPq9F0UeVXOL81b1yD+nWiGX55O7q0jytQNVsQBEEQzgciYC0HJ46ksmvN3soehlCM9lYyT7GWV5nILQxGR0fGTxOSmMw0nmELT1q9UKWUyh7qRUOSJboN7ZT/OKRGJL4B7ZGyPUg+DTPcBQ5b3kZNB9PEEeLgthevp+Pl7WjduwVblm4vfFwJYutFE9uqHq9+uJANWw8RGupg1KAOXNm/NaqqFNj/xffms3JtgphhFc57tWIjefbh4ZU9DEEQAtiXnkam10vT6BjCHY7KHo4gVFkiJbgcnDicgu3URbVQJdktgydZw108w9cMwsKNgh8J2EdN7uM+MgjjajLQrLDKHu5FwzItVsxYnZehAAwb0BanXcUKd2FWj/g7WAUkC6rbVe579zZGTBqMJEm89OtTXDnxsgKzSKpdpUGbOCZ+cje3/vMr5i7awuHj6ezcm8hbny7ioWdnoJ82k5qSlsOKv/bg84vCacL576+N+9l74ERlD0MQhNNsP5HMFVM/Z/h3U7ll9o90//QDnlu2GF3UTRCEgETAWg5qNaqB3xe8+Iski5SsynYph4mnASvpjErhnoUWMq8zhuuZTy6NK2GEF6+cTDefPP4NAMMua0vdWtWw2wrOgDrsKo9OuoLvj3zIlbdenv+83WFj8sf38Kv/O16a/ySPfXU/b614ng83vcrb36zA49UwjL+nTb0+nZ0Jicw/bVZ2Z0KiSJsULhiaprN6w77KHoYgCCcl5mRz3Y/TSEhPw6vrZPv9eHWd77dt5dlliyt7eIJQJYmAtRxE14qi04B2qPbCGdeSLGGZIs+wsnUjkZkMw0bw1in7qcMxqhOHqKZZkUzDZMUPfwLgcNiY8uINjB/VjerRYYS47HRuF8ebT49hxLDOyHLgjzBFUegy6BIGTuhHi65N2XcolYyswjcmIC9onTrrL3TdwOfT+PCbFWJ2VbhgmBYFMggEQahcX2zagC/Ae9Kr6/ywPZ50T+DvKkG4mIk1rOXksa/u59GBz3Fkz3F8bh+2k8Fr9brRHEtIquTRCXYMMohCxl/kfjk4UcV9nQqn+TV2rUsg80QWDdvWZ+J1vZl4Xe+zPl6ux4dcRGbDsaQMRt4+hT5dm3I0KeOszyMIVdHu/cmVPQRBEE5afuggmhn4JpJdUdianES/Bg0rdlCCUMWJgLWchEeF8f66l9mybDvxK3fiCnfSb0xP/vhpLR8/OhWv21foNapNybsTLiZgy91xQrmE7SyhLUqQoNWGTjOOcJQoXBU8voudIstMvuxpVFVB82l0HtSBx79+gJ1r9jLj9Z85tvc49VvUZdyjV9G2T6tij9ckLhZND742yLIgK9vLL4vjy/LHEIQqYdXavaSm5+Kwq1iWxbzF8fzwywYysz3UqRlJ785N6HpJQ9q3rIuiiBt0glCewuzBs7ZMC0LtogaKIJxJsqpQGcwuXbpY69atq+xhFJKdnsOc9+ez+NuVmKZF/2t7MOqBoVSLLX0vO0+Oh4ltHibteAbGaSkhjhAHlmni9wZf+yqUnYZWJi/wF935HilIWvBVrGA0q7ifMcRKGyp4hMLpbA4bsfWiST2ege/kzR5JArvLzm0vXM/oB4uugrr/cArPvDmX/YdSMSvgM0+RJQyR+i9UEZIEkeEucnJ96IaJJFGoArZNlQkNcfDcP0bQqV1c5QxUEC4CP+3czlNLfsetFb7ei3GF8OfEu1CCLHcRhAuFJEnrLcvqUtL9xTuiGOnJmdzZ4Z98+8JMDu04ypFdx5j+2s/c0fYfJB5IZu38jfz0v1/5c+76AgFoIJZl8efcDYRWC0GSJSRJQrWrhEaG0PfaHkUWahLK1gEpki3E8D4vIAdINGjPXp5gKm8ynmrsroQRCqfTfBrHEpLyg1XIu+D2uf18+vi3pB5PL/Qay7Kw9CP8+Msybn/0a/YfSqmQYLVahIu7xvelbYva5X4uQSgJy4KMLA+6YeY/PpOmm2RkeXj0pZkcTRRp8YJQXoY3b0mHmrVwqX9feyiShFNVeWPwEBGsCkIAIiW4GJ898Q3pSRkY2t/BqObTyEzVmdj6IRSbgqEZKDYFh9POS/OfomnHRgGP9c69n/D71GV4c/++6JZlmdEPDkXTRCpwRXuDS3iYeJZyN18zjK3EEUYuw/iTPmxmMpNJ4gAhUk5lD1UoxrLpfzD6wWH5j03PQsj+D0eTNN77aiR+rWQfdYqSt8719ErCpeFy2vj8tZvYmZBEwkHRv1c4/+i6yfdz1jH5zoGVPRRBuCCpsswXV13Djzu2MXXLJrJ8XjrXrsvdXbvTIqZ6ZQ9PEKokEbAWY8l3qwoEq6dYppWXvnsqhder4cn28siAZ/n28Ae4Qp0F9k/YfICFXy3F5y64XtLv9TPtldn0HFniWXGhjGiSwit0oIHlYxDb6cMGvDhZTgde4XJUEayWHSkvLdEqhxZzfp9GbqY7/7HlXQKZkwEvv6zugmmWvEWNoihcPbg9s+Zvzjt2gPd+UXx+nf99uZT9h1PxiPR+4TykGyZbdh6t7GEIwgXNpihc17Y917VtX9lDEYTzgghYi2BZVqnTdHXdYOn3qxgycUCB5xd/txLNF7hVhmlarJz511mPUzg3ByUHH1P/tGfScJFWaeO5IFkgyTLIFrIsYegmdqcNSZIwTQvtHNLhXaFOWvdsDsDRvcfZ8ftrhIXb6NjXR1J6GLqhBH2tokgYhoWiyFiWhYTFb8t3MuSytkRXC+GrH//CMEoeZZumxaJVu876ZxGEqiAqsuzLzCWlZJGUkk2dGpFUjw4r8+MLgiAIFy4RsBZBkiQatYtj3+aDJX6NN8fLvi2F9/dkezCDXPjqZdTz0e6y4feIWR2hajr1+x8eE06bPi1xuOyERYVyIP4wO9fsxe8pusXQqXXfp7+PVJtCjQaxtOrRjCeHv8SmxVtRFAeG2QjDlKg5VMFu0wOmBLscKpf2bEFSShabtx/BNC18fgOf3828xVuJDHPitKvkFjMuQbiQyBKMvrJjmR0vLSOXZ96cS/yuY9htKn5Np2Ob+jz90DAiwkX9dUEQBKF4YmV3MSa+OB6HK3gJ8jPZnTZqxBVcg+DJ8bDtj/KZdXGGOoipG81nO99i8C2XodiCzyYJQlWQcSKLLSu2s3rOOua8t4Btf+xC92soqkJoZEjQ95t1suquzWEjNDIEu9NG2z6teG3x07x66/tsWrwVv1fDk6vg90gYPjg6Jw0ttXDAKcsS0VFhPH7vlWTleAtV9NV1k8wcL7Ex4WX/DyAIVZhpQUSYs/gdS0A3TO558js27ziCXzPIcfvwawbr4w9x/9PTqUpdCgRBEISqSwSsxeg2pCMPfXQXYVGhhIS7CInI+6Pag0xOSxIDJ/Qr8NR/b3yXwzuPBT9JyZfY5XOG5s1OaT6dnPQcHuz1JIkHkoPO4gpCVZKdkoMnxwuAoRmYhoViU7jx32No1aN50Nfl/X5b3PzcOD7b8TavLnoaXTP465cNgVtCmSbqnuOAhV3VcdpNXE4b9WpH8fYzY8nIcnPwaOD0b103SUnPwS5uAgkXmUdenMnho2lM+3k9L7+/gG9nryX9tHXiJfXHugRSM3ILFTHTdZNjSRlsiD9cVkMWBEEQLmAiJbgEBo7vx2XjerNnwz5M06JZp0Zs+H0rz499A9Mw0XwaNruKpMg8+vm9RNWslv/alKOprJ2/qeg1emdxk9mb64fcvJkjQzfwuf2s/XVT6Q8kCFWE3+Nnw6KtZKcVXehK8+n8+MZcRt03BIAD8YexO2xoAQJWyQI5PReQqBHlZsK1g2hQrxFtW9RBkiQSkzORpeB3jCzLolO7ONZtOYiui5tBwsVB0wwmPPQ5qqrg8+s47CqfTlvFfyaPpGfnxiU+zqbtR4IWH/N6NbbsOEJn0fNVEARBKEa5z7BKknRAkqStkiRtkiRpXXmfr7woqkLLbs1o3aM5NruN7kM78cXud7j+8avpd20Pxj4yks93vEX/sb0KvO7g9iPYnbZKGrUgnF+O7jlOgzb1keWi0w5Sj6Xx66eLsCyLqJqRRfZAtk6+/9xaDYYP7EO7lnWRTgapNapHEBbiCPg6SZLo3C6OFx+9ihED253lTyQI5x/dMDFMC9/J+go+v47Pp/N/r88hM9tT4uOEhThQlMCXGTabQmiQ954gCIIgnK6iUoIvsyzrEsuyLqjeLdXrRHPjv8fwf9Mnc8vz11MjLrbQPtG1qhV5MX2K3WWjVqMa5TFMQThv1Gtem2v/MRxbMTd5dM3gvQc/556uj1GzQXWia0cF3M9SZPTGNQFoUDfv/akbJqs37OPn37ewbfcx7r6xH44AKf4Ou8LEcb2x21Qm33EFbz8zNr9PqyBcjCwLflu2vcT7D+zbMmjACnBZz+Dp/6ckp2bz7ey1fPD1clau3Vuqqt2CUFI5fj+v/7GS7p9Mod2Ud7lx1gw2HC9iKZcgCBVKpASXs4Zt44itX50ju45SVH0Jy4IWXZuScjStzKoGC8L5xOa00fmKDsTWj+Het2/jrUkfFbkm2+/xs2/zQf7R/2nuf3ci/7nuTdy5PizdyMuyV2SMOtGYtaNwOlRuuqY72/cc59EXZ+LXDEzTRJIkalaP4K7xffl61hpy3T5My6J+7SgenTSIJg3+vgl14HAKqqJgGOL9KVycfH6dY8mZJd4/rk40143ozPS5G/CetizG6VC5dWyvYouaTf95PR98sxzLAk03cDltREWE8P4L14vWOEKZcWsao6d9w6GsTPxG3gTDqsOHWH/8GO9cOYyBjZtW8ggFQZDKu0qfJEn7gXTyVmp+aFnWR2dsvxO4EyAuLq7zwYMlbyFzvji44wgP9/2/YtfmKTYFQyt+NlYQLkSqTcER4sDv0+h3TQ+G3jGQxwY9X6IbODaHjRF3DyKkWigzv1hCrmFiNq6JXKsapgV3XN+HYZe35dq7P8Z9RpsaWZaoW6saX791K0kpWdhtav7FsM+nYVoWsiQx/Lb3g67HE4SLgSJL/OOOgVw1qEOpXvfnxv18N3stRxMziKsbzfhR3Ypduxq/6xgPPjM9Py05fwyKTIvGNfnov+NLPX5BCOSzjet5bfVKvHrh75oYl4s/J05CkUWNUkEoS5IkrS9N5m1FzLD2tizrmCRJNYCFkiTttCxr+amNJwPYjwC6dOlyQda4b9CqHl/ueZdrYm/Lb80RiAhWhYvSySxbXTPQT1YiXfHjn3jdPl5f8gz/vfFdju9LKvIQmk/jl49/519TH2D2vvfZtS+JTdsO43TY6Ne9KVGRoXw3e23AdELTtEg6kcXmHUfo1DbvInrPgWTe/nQxW3ceBaBm9YgiMyQE4WJgmBZX9G1V6tf16NiIHh0bleo1381Zi18rHEAYhknCwRMcOpZGXJ3oUo9FEM40Y3t8wGAVwKvrbDuRTPuatSp4VIIgnK7cbxlZlnXs5H+TgVlAt/I+Z1UUHhWGahMZ2IJwOsWmYLOrhSpl+70aa3/dSFStany5510ata1f7LG8uT6+e2kWAC0a12TciC5cNagDUZGhAOzYm1hotib/fJrBmk0HANh/OIV7nvyOTduPYJgWhmlxLDmzQEqjIFyMJAlCStGX/FwcOpoW9CaRqiocTyp5arIgFEUzgk8WSJKEr4otA9FNk0X7Evhkwzrm7d6FL0iwLQgXknINWCVJCpUkKfzU34FBQHx5nrMq6zG8M1Ix1U8F4ULVpncLHCEOZEXGFe5k8G2X07xLEzRf4C9bxaaw86+9SJLEjc+MwxlafEXRYwmJQbeFFfP6+F15BTY+/GZFqYJTmypSxYQLg1rM73KLkwXMKkJc3WiCdZzSdYPaNSMrbCzChW1g4ybY5MD9tg3Tom1sxf3eFychLZU+n33EQwt+4dU/VvCvRQvo9skHrDt2tLKHJgjlqryvtGoCKyVJ2gysAeZZljW/nM9ZZU186QZCwl0l3l8WFUmFC8i2Vbvwe/15LWssWPrdyiLT4CVJIiQi7/3S5+puDJ80qNjqwbH1Y4Jua9W06JSujJPtOtZuPlji9F9FkZGDXOgIwvnEYVe58eruRFcLCbpPv+7FV/UtK9eP7Io9QFaSosg0aRAr0oGFMnNbx864bCpnXnG5VJX7unbHZasarQl102T8zBmccOeSq/nRTJNcTSPb7+PW2T+S6fVW9hAFodyUa8BqWdY+y7I6nPzTxrKsF8rzfFVd3aa1eX/dy9RrUadE+5uGWDQnXFgs00LXDDw5XnweP/u3HiyyT3HHAXn9TyVJ4q5Xb+KjTa8R16puwD6tzlAHY/95VdBjtWpaG7WIFhu1qkfknysYWZaoFRuR/9gwTHx+kSosnP+aNqjOxOt68+UbNwd9n3z142py3b4KGU/bFnW464a+2G0KNjXvplCIy0bN6uG8+Gjw97lQdZ1w5/LhujU8seg3vti0ocoEWDVCw5g59ga61KmLTVZwqSrVnE4e6dWXSV0qdxXb6sOHuGnWDHp++iGDp35Bps975goaAAzLYubObRU+PkGoKGJRZQWr06QWry1+holtHiI3wx18RwlUu4oeJF1SEC4Eul8ntn51slKz8ebmXQjLsoTNaeOxr+7H7igYzNZrXoe3Vv6Hh/v9m+SDJ/DkeJEVGZtd5fLxfbn8hj5Bz9WkQXXq1qrGwaNphbY5HTbGDOsEQO8uTViyehdmgAJpjerFkHgi61x+ZCGAphnHGL1vFf2OxROq+TjhiuTXBp2Z3agHac6I4g8gnBNFlhl6ed7NoQXLdqAoMnrAllISS/7YzfCB7SpkXGNHdObSXs35feVOsrI9tG1Rl56dGhXZ21Womn7ft5cH5s/Dsix8hoFLVXlt9Uo+GzmabnXrVfbwaBwVzbRrryPd48GtadQKC6v0ysCfb9rAa3+swFOCNapeXWdnyokKGJUgVA7xqV8JYmpH8f7alwkLknql2BTGPnIVoiypcKGzLHBne5j88d2ezNXyAAAgAElEQVS06d2SOk1rcel1vXnnjxfpNbJrwNeER4Xx4aZXeeLbh7jqviGMfWQk76x+kYc/uKvI2VFJknjx0auICHPidOTdq5NlCadDZcTAdnS7pCEAd97QB5fTXuhYDrtKjttH7hltcYRzM+zAGt5Y+Qn7jCYMrD2dJvX/5ObodwlLhc8WvU2zDLE2q7wZpkmnk4XNEk9kBi1O5vVpnEjLrsihUSMmnBuu6sqkCf3o07WJCFbPQyluNw/Mn4dX1/GdLHDk0XXcmsbtP8/Co1WdLJUol4u6ERFlFqwez85m4/FjnHDnlup1aR43r6xaXqJgFcCuKNSPqHY2QxSE84KYYS1nx/cnMfeDhRzedZSGbeoz5PYB1KhfnTpNavHe2pe5v8fjeHJ9aCf7OzpDHTRq14CbnxnL3g372fD7lkr+CQShfLmz3BxNSOStFc+X+DWKotBjeGd6DO9cqnM1qBfD9Pfv4Nel29gQf4ioyBBGDGxHq6a18/epW6san74ygQ++WcGqtQkYpsklretRp2Y15i7aWqrzCUVrn7KP27f9xrAaU9knNUa1e5BlP9vNpjwgv84I5vPyqheZMGgybpuzsod7wZKA31bsYOK43jRtUAOX0xaw53CI00ZcXbF2VCidmTu2YQW5AW9ZFgsS9jCqZesKHlX5OpGby4Pz57Ex8Rh2RcFnGPSNa8jrg64kwlH8Z9lvCXuRi7gBeyZZkhjTuu25DFkQqjQp2IdIZejSpYu1bt26yh5GmVn8/UpenzgFUzfQNQNJ+nvStHXP5kx64xbqNKnJ3A9/Y/Wc9ThC7AyZOID+Y3tis9tIPJDMjU3uLdTyQxAuNKGRIfx44jMUteoVMNq+5zj/fOFHcnN9GEX0URZK74VVXzFHH8V31Uai2grPXOuanQ8Sn2BXo2rMbn5RdkSrMP26NeXFx0bh9vgZfeeH5ARYq1otwsWsjyZhs1W996lQdT25eCHfxQe++S4j8XDPXtzbtUcFj6r8aIbBwKmfczw7C/20a2y7rNCienV+Gje+yGwggE83rueVVcvRzECp+aBIEoZlYVcUJCReuWIwI5q3LNH4tiYn8d6aP9mclEg1p5Ob2l/CmDbtUCs5BVq4uEiStN6yrC4l3V/MsJaT9KQMXp84Bf9p6YOn3xvYvno3jwx4hlcW/pvxT17L+CevLXQMh8uOza4GbfshCBcK3a+TnpxJ9Qqu/GlZJviXY7lnguUGxwAk10gkOa93a06uj4efnSHSgMtBiOaly4k93FJnWMBgFUC1+fk65BoeP/CmCFjLWb06UUBen9W3nhnD5Od/RNN1NN3Epig4nTbeenqMCFaFUmsRUx2XqgZMb3XZVBpVu7Bm7RfuSyDN4y4QrAL4TYOE9DTWHT9K1zpFr9vtVrceqiwHDFhDVJWBTZqiSjINIqsxpk1baoWFl2xsCXt5cME8fLqOBSTl5vCfFUuZn7CHz0aOrvR1u4IQjAhYy8nv36wodg2qz+3n/Ye/4H9/vhRwe2hkCBQqtC4IFx7TNE/+vlccy9Kx0ieBti4vWAXwr8XKfQ9ifkBSajF/2TaMIHe4hXMTrnnIlCLxuySKCoESQ6KJLKpAnVCs07N7gmnV7O+0+JZNajH7k0ms3rCf48mZ1KtdjW6XNCqyyrYgBDOqZSte/WNFwG0OVWVg4yYVPKLy9eeRw+QGWZfr03XWHi0+YG1XoybtatRkc1Ji/rpfyJtZjXKF8PKAwTjU0l3C+w2DRxbOx3vGjQOPrrP+2DEWJOxlaLOCravik5OYvm0rKR43PerW5+qWrQl3FN8TXRDKmvj2KScph1PwB1gDdKY96/fh8wRuE2B32ul7bQ9Uu7ivIJz/ZFVGCtCORrEpdB/aGVdoxa5RtNzfg7b272AVAA+YqViZjwGwZ38yXpHhUC6ybS4izGxCKbqITz3jKGlSVAWN6sJUkpU/P8zbUOCxqir07daUscM706tzExGsCmctwuHks6tGE2a3E2qzIUsSoTYb0S4XU68eg125sGbtwx12lCApvzZFIcxuL9FxPh05moGNm+JQFMLtdhyKQtc69fhx7PWlDlYB1hw9ghnkw8Cta0zbVjBt+6WVyxj7w/d8F7+F+Xv38PKq5fT/8hMS0lJLfW5BOFciEionTS5phCvMiSfn3PqM3ffObexZv4+UI6nnfCxBqEiqQ0WWJGo0iKV192Z0uqI9Hz/2NelJmZgnW2Y4Qx1EVo/ggSl3VPwA3V+B5QmwwQD/eiwzjdo1IrHZFDTNCLCfcC7cNid/uDozInMJM6KHBt1vXMY8ZkdcUYEjuzht3SmqMQvlp2udeqy5fRILEvZyNCuLRlFRDGjU5IILVgGuatGKzzdtwAiQAm1ZFkOaNg/wqsJC7XbeHTKcNI+bI1lZ1AgNLXHqbyBuzV9k0l6O/++lGSsPHeTrLZsLzMZ6dB2vrnPXvNksnHBrsetwBaEsiVumZcyyLLJSs+k+vDM2R/H3A1p0bYLDFTy94lQLj398PIneV3dDFne5hfNAbP0Y/vfXS/Qc2YWk/cn8/s0K/nvju6QeT8c0TCRJQpIlBt1yKZ9se5OoGpEVP0gzPfg2yQZmOkMvb1uqSo1C6Xzd8Aomp35MnD9wsDQoezndPZtZ0KhDBY/s4iNmUIXy5lRtXNWiFfd07c6Qps0vyGD1QEY6M3dso2ZoWKEiRi5V5R89ehMbGlqqY0a7Qmhfs9Y5BasAHWvVwW8EvvnqUBT6N2iU//iLTRvw6IWzBC3geHZOoZ6vhmmS5fOJJTRCuREzrGXo96+X8ekT35KRnIVlWbTo2pRje4/jzfHhDVBx0RHi4O43by32uDa7jUvH9aZFt6as/20z3tzAKcSCUBVIEgy5fQCTL30ad5YH6/TKuif/alkWWLDg8yXc9p/rgUpYE6M2z0sJDsQyQalDjRgXj04axEvvzUc3xBdxWdvVOIo3ku7kx4N381n0WH6IHEKqUo2m/oNMSJ/F0Oyl3FjzHaSaRdxcEM6ZLEn06do04Lb9h1OYMnU5azYdwAK6tIvj7hv707RhbMUOUhCqiEOZGSw/eABFlrmsYaP8QHL6tq08vXQxpmWimSaqLCOT19u1VWwNJnXuRq/6cZU27tjQUK5q0Yqfd+8sMHMqAU5VZXy7v28MHssJvlRDlSWScnNpFZu3LvbNP1fxzZbN+Awdh6Iyvn0HHu7R+4K8ISFUHtHWpozMmbKAjx75Cp/775QKSZYIjQzh/vdu59D2I6z5ZQN7Nu4HC8KiQrl0bC9uenYc3hwv3/13Fuvmb8LmUGnduyUZSZlkp+XQvn9rRt0/hBr1q+PJ8XBtjYklWhsrCJVJkqWCgWoQzlAHd79xC0PvGFgBoyrI8v2BlX43cGZasBNCrkeOeDz/mbse/4Ztu49X6PguFp6cakTHh3Jb5jSGeX4nzHSTqMbyfcgovoy8BrXVIVxhGZU9zAuWIkuEhDj47NUbqX1GpsO+QylMevwbPD6twDpYl9PGe89fR/PGNSt4tIJQeQzT5F+/L2Dunl15WUKAaVnc2L4jN7bvwKCvvyhQIOmUUJudtXdMwqnaKn7QZ9BNkxdWLOX7+K3YFBnNMGkaHc2bg4fSNDomf79HFv7KTzt3YASIEVRJpkOtWtQMDeVwVhZ7UlPxGn8HwE5FpVu9enw+crRIGxaCEm1tKoHm1/jsiW8LBKsAlmnhy/Wxd8N+7nzlRiRZ5tDOo2hejZz0XH77aim/T12OhYXu1zH0vBmcYwlJ+cdI2HyAuR/8xn8XPEXrni3oe21Pln6/CkMXa+qEqqskwSqA1+0jLbFyghHJ0Qsr4gnIegEkBbDAMrAcV7Bh/xgys3dit8lEhLk4liQCpvLiCssgq5OX51NvYvKJpzD8dhS7n2qxhwiP2Y3NLtbulxeX08bA3i25ZWxPalaPKLT9va+WFQpWATxejXe/WMq7z42roJEKQumledxM3xbP+uNHqREaxvVt29O2xtnfZJmy7i9+2bu7UFD6zdZN7EtPI/jXnsVvCXsZ2aLVWZ+7rKiyzNP9L2dyzz7sz0gnyumkXkThJTm3d+rKvD27A67DNSyT9cePBT2H18irhLwlKZEOtWoH3U8QSkMErGVg3+aDBJup1vw6q2atod+1PZjx2uwCQa3fU/xMqe7X0f06z415nW8PfcDIewaz6OvlZTZ2QahMrjAnDdrUr7TzyyHjsJzDwL8SLC+b99Th8dfX4PfPxefP+6JWFAnDqDqZKBcim91LdO19RNfeV9lDuWh0aR/HG/83BjlA5W7IS9tfu/lA0ArDm3ccQdcNVFWk/QlVz5akRCbMnIFumXh1HUWSmLVzO3d06sLDPXqX+niGafLJxvUBe8l6dJ0/jx5GMwNPJPgNg+Tc3FKfszyF2e20KyJ4bxFTndcGXsmDC+YVmmUtybehV9f4LWEvNkXBJis0jY4Ws63COREBaxmQFTlowAqgqDJzpiw4p1Red7aHHX/u4cc35571MQShKpEkCVeYi54jOlf4ub0+jW27jyPLEm2b18HmvJJDR9P453+/KtTGRgSrwoVGVWQG92sdNFgtCcu0StQuRxAqmmlZ3Dn3J3K0vycIDMvC0HU+2bCOSxs0omPtOqU6Zrbfh0cL3uLMo2k4FCVgSrBlWexNS2XCrBlE2B2Ma9OOfg0anlMAZ1kWGxKPsSslhRqhofRr0KjM14yuOHwgYEpwicYHfLppPV9t2YSFRTWnk5cGDKJvXMMyHaNw8RABaxlo3KEBdqcdT3bh1DW708aACX3ZuCi+xGmSgciyTFZqNjvX7Cl+Z4mS3QIThEqg2GTsDjsRMeH8d8FTqLaK/Rj6dvYaPpu2GkWR8t8nD9x6Gdv2HBfta4SLgm6Y/LnxAEMuaxt0H0mSaNeiLpt3HAm4XVSsF6qqtUePkOv3B9zm1XW+2rKp1AFrqM2OIktoQWrvhdps5GiBJyV0y2LWzu1oJyvoLj90gH5xDfnf0BGlrkJvWRYLEvby+KIF5Pj9yJKEXVFQZJmPho+iW916pTpeMG5NY/q2bed0DL9h5Fcldmsad82dzXfXjKNDzVplMUThIiO+ccqAoig8OOVOHK6CzaBVm0K1GpFcde8QWvVoVqI2N8FoPo06TWvhzg7UN/I0IuNCqKJqxFVn/FPXcv+7t/Pc7Mf4KuF/1GteuouGc/Xz71v4bNofeH0auW4/uZ68P298sog1mw5gnMNNJUE4n/y5cR8/LdiMUUT16+4dGwbdpioyy0tyA1UQKliyO5dgF0N5bVmySn1Mm6IwqkUr7HLgWcxAqcKn005r9+LWNJYdPMCcXTtKNQbTsrj/17nc+8scMn0+DMtCM01yNY0sn4/xM6ezMTH42tLS2Jx4HKuMZz68us6bf64q8JxhmqR7PGhB2u0IwilihrWM9B3dnfCox/nsyW/Zs34fdpedAeP7cvOz4wirFsrIe67kp3d/RTsj3VBWJCyr6CI1dqeNrkM68vrEKXhzim5po6gKhpglEiqZza6i+XUUVUGxKYz950hufrZyC7RYlsWn0/4olPIL4PPrZBZ3M0gQLiC5bj+vfbSQb2evYepbt+KwF74ccHsCz1JB3ntmz/5kBvRuWZ7DFIRSax5THcMKfCPGJsu0r3l2hYCe6HspW5ISOZCZgfuM2dTSps56dI3PN21gVMvWBZ5P93j4aMNaZmyPx61p1A0LZ1LXboxu2YYfd2xj0f6EoGGkYVnc8ON0ltw88Zx7tp7ZQ7bIfSWJcIeTDK+HKKeLLL8PPUg/1r+O5GVsaIbBO3+t5sstG/EbBrIkcU2rNjzepz8htsqvpixUPSJgLUOXXNaWd/54MeC22Hox/Ofnx3lm9KuYpolpmFgWNGxTj+7DO/PDaz8DeUWWNE3H7rCh2BQ0v07XKzsy8p7BPD3qlSKrA0uyJIJVodypdhXdH/xusqxYtOmWTkwtleq1DQbe0JAGndpV4AgDy3H7yMh0B93u9WllnE1/6kgi7UGouo4lZfLuF0v4551XFNoWExWGw6biC7B2z2FTiYkKq4ghCkKptIipTqvqsWxNTioUOKmyzE0dLjmr44bZ7fx03QQWJuzlwQXzggZlJZXqdhd6PPy7r0jOzc3/9kjISOeRhQv4cvMmPJoWcI3s6fyGwZR1a3j20gGFtummSZrHTbjdgauYoLBj7TrYZLnAzPDpJPLSpC0sXhs0hMFNmgF5lZl7fvph0OP6DJ2lB/YzfdtWlh7cX6Af7A/b49mWnMwPY68Pmirt03V+27eXvamp1AoPZ1izFkQ4St7H3afrzE/Yw/KDBwi327m6VRuRonyeEAFrBepwaRtmJH3CugWbyUjOpMklDWnWqTEA4x4dxaHtR7C77NRtVovtf+wmKzWbZp0aUSMulm9fnIm/iLvdtZvUJPNEFu4sMUsklA9ZkWnYNo7eV3dj/qeLSDmaVigzQFYsXvhmH6275OIMObVtP1bqKqj2NpLzsoof+EkOu1pk7Fg+BWREsCpUfb8sjg8YsA7s05IpU5cFfI0l5W0XhKro4xGjuHX2TPampQIgSzKyBP8bOiJgG5eSUmWZFtWrY1eUcwpYJaBdzZpYVl7irSxJvLNmNSfc7oA3TbclJ5WoqJIFLEjYUyBgNS2L99f+yccb1uM3DCwsLm/YmLs6d6Oa00lcZGShAlCqLPN/fS/j38sWBRz7P3r2pmVMLH3iGuBQ/w4lol0hNI+pzrYTyUHH+OjC+WT7fYWCb59hsDsthVWHDtK3QcNCr9uVmsL4mdPx6Tq5moZLVfnP8iX8b+gILmvYuNh/m+TcHK6Z/h0ZXg+5moYsSczYHs+I5i15acAgUcW4ihMBawXISs3m4PYjRMSEEdeqHj2GF66KanfYaNqxUf7jdn0L9utyuOwoNhXTV3hRvyRLdLi0Dcum/VH2gxeEk7qP6My/vryfkHAX1zw0jOvr31Wo0FifoZm06uw+LVgFMAEvVuaj4FiNJFXOx47dptKrc2NWrk3APCPQlqTyCFjFl59wfvAHycyJigzh8Xuv5L/vL8AwTHTDRFVkFEXmsbsHERUZUsEjFYSSiXaFMPu6CWxJSmTHiWSiXC76N2hUILgqimlZQWf5wu2Oc55ddagqFtBmyjt4dZ0GkdVIzs3FDPJFZEGxs6unnDnqfy/5nVk7txdYZzs/YQ8LEvbgUFWqOZ080/9yBp2cJT1lQodLiA5x8eyyJZxw5yKRN3v91pVDaR4TG/Dcc3btYFfKiSLHl+IJnunk1vLa4ZwZsOqmyU2zfiDN8/ekzKmf575ffmbpzbcTGxpa5HkfXvALiTnZ+enbpmXh0XV+3r2LPnENGN5c3ICrykTAWo78Po237/6IJd+twu60oWsGNeKq89T3D9O4fYNSHavvNd357MlvA26zTIu9Gw/gCnfiySlcqVgQzoUkS1w6rhdPfPNQ/nOhESG8OO8JHh30PNpp7ZqGjE/FFRrsi1wH/zpw9CjnEQf38MQBxO86To7bi9+f9+UvUV6zq4JwfggPDZ5Sd0XfVrRsUotZCzax/3AKjepX5+rBl1C/TlQFjlAQzk77mrVoX8KUT8uymLplE1PWrSEpN4cIu4Px7TvwQLeeBQLd2NBQWlWPZUtS4lktIbErCi5VZcn+ffkptwczM87iSIXZZJmhzVoAeQWNtiQlMmN7fMDUXou8QkiJOTk8tOAXpgwdSf+GjQrsM7RZi/zjFed4djaP/b4A/Ry+UCXyCrqdafnBA7i1wFmGpmUxbdsW7uvWM+hxk3Nz2HD8WMC1xh5d4+MN60TAWsWJKsHl6LXb3mfZtD/QfBq5mW58bh+Hdx7lH/3/TXpS6T6casTFMvaRq1BsgVNCDm0/Qrs+rXCE2ANuF4RzsXXFDm5qdh/PXPMqW5ZvByCuVT1i68YU2C80vKg7wBJYwe+sVoTYmHC+fvtWbhvTi2YNY4kMd5ZTBygRAQvnj1vG9ipye/06UTxw62W8+e8xPHDrZSJYFS5Izy5bzMurlpOUmwNAlt/HZxvXc+NPPxSa+Xz1iisJs9lLnUdjk2U0wyDd6w26PvRsyZJEmN3BnZ268NXmjXT9ZArXz5xeovN4dZ2XVgVO/y+pGdvjz/mbz6XaGNqseaHnD2ZmBP05fIbB7tTUIo+blJuLrYiU6lP/z4WqSwSs5eTEkVRWzvwLX4B1p5pPY86UBaU+5k3PjEVRA7/h/F4/a+ZvpFG7BsjKaR+hUt4f1V62DaWFi4dlWqQcSeN4QhKrZq3hiaEv8sXT3/PShLdJPlww9Wft4nB83iBf4ZYfbO1LdW7TNDmw7TD7tx7EKKOy9xFhTiaM7s7HL0/AE6Bi8LmzAAuJsr0YEYTyMLB3S8YO65T/2DBM0jPd+AIsPxGEC9XRrCymbdtaqD2NzzDYfiKZlYcOFni+aXQMA5o0KXUfVc00y+V2piJJDG3anJ+vn8Dc3bt4edVyMrze/D6oJbE3LQ1fMe15inIoM6NU5zuTU1XpXq8eXWrXLbStfkQEtiCVi+2KQpPo6CKPXT8iosixNY2KCbpNqBpESnA52bV2LzaHihbgS9/v1Vj/2xZufqZ0bT4M3UDzBi+85Mn2sm/zAUzjtI/Dk3/V/aJ6sFA2fG4fM16dg2lahX6vfv6yOqNuT8Fmtyj43eIE10gkpToAlpECZioo9ZDkwOtO/pi9lrfv+Qh3lhdJymvvdPdbtzLghr5l8nO4Pf4i20mdnVPHk7EwTz4Wa1mFqklVZRrWj0GSJCzL4vs565g66y+8Xg3Lgl6dGzP5zoFEVyt6bZggnO+WHtwfNPh0axpzd++ke9162BUFSZLI9vn4dc/uUrezKS8yeVWME9LSeHnVCvQgbX2KYloW8/fu5qozWu0AJOXk8NWWjaw+fIgol4sb2nXg8oaNCxQqahUbi3OvWqDyb0nZZJlRLVrx3GUDAxY/OrX+OFcrfE0tSxLj2hTdiaCa08WVTZoxP2FPocDVparc3bUbAFk+L2keDzVDw4qtpCxULBGwlpOQiKKLUUTElL4dgGpTiYyNJCM5M+g+fq+4Ky6UXs0GsWSmZuPN9ZYom1Xz66gBZvszUmw8PLIZT318lAbNdZBUsHQIGYcU/hiWkYiV+Qj4N4JkB0vHco1GingCSfo7nX3j4q28eMNbBTIUPDle3rzzA5whDnqP6hZwXKZpcSwpA0mSqFOzcOVDyFuntG3PcbbuOIosl3UwefrxZERqsFCV6brJjF82cMuYnkyZupyZ8zcW6FO8al0COxMSmfrWrYS4xHIT4fyR4nbzw/Z4dqem0KhaFGPbtKNmWPDrrlPVeoOZtXM7P+zYhozEsObNGd/uEmyKUuJCSOVNsyymbdvK9O3xQQs3lcQTixeSlJvLnZ275j+3JSmR8TNnoJlGfrD319EjXNawEW9fORxZkvAbBm6/dlbBKuTNPM/etYMW1atzc4dOhbbbFIUvr7qGCbNmoJkmbk3DeXJd8ZuDhpao7+yLAwaR7M5lc+JxDNNClWV0y2Ryzz60jInlzrk/sfzgAVRZxrIsxrZpx+N9+peoOrNQ/iSritwdAujSpYu1bt26yh5GmdA1nbG1bic7PbfQNmeok8e/eYBeI7sGeGXRfnjjZ7749zR8bl+B5xWbInqwCmdFkqW85aVG6T4LZEXGNALfxe04oB0v/3orWBmgNEKSw7DMXKyUwXkzq5z+u+oEx6XIUe/kP3Nvt3+xe11CwGPXbVabL3a9U+j5Jat38dani8l1+7CsvAqnj9x1Bd1Pq76dnetl8vM/su9QCoZhYmGh6yJ1V7i4zf38Hkbf+WHAasFOh417burH6Cs7VsLIBKH0Vhw6wKS5szEtC59hYFcUZEnijUFDGNCoCVk+HxEOR4E1jYcyMxj89RclDkBjnC5yNH+VCVjLklNVWXP73YTZ7ViWRb8vPuFodlah/WyywtuDh9KvYSPGzPiOvWmp57wu16Eo/HHbXUS5XAG3ezSNuXt2sSvlBHXCIxjVshXRrtJVK49PTmLN0SOE2mxc0aQpoTY7V37zJUezswpUf3aqKv3iGvLB8KvO6WcSApMkab1lWV1Kur+YYS0nqk3lX18/yHNjXkPz6fkX9s5QB50GtgvY2qYkRj80jIPbj7D42xX5zxm6ETRwKAs2p61AJVjhAiJxVmmxzlAHMXWiSdyfhHFGwOcMdTDmnyOR1PpA/fznLc8cMLMpGKwCeMG3BEs/hKTGYVkWezbsC3ruxP1JeHI8uML+/kJbtS6B/7zzKz7/33d3E09k8cQrs3nj/66lQ+t6ADz71jx2708qNkh12NUCxxKEC1VMVCibtx/BpioBA1avT2Pxql2lDlgty2LRql18M2sNiSeyqBUbwfiruzGgdwvR71AoNzl+P3fPm1NgLeqpWcH7f52LXVHyW9Zc27oNj/fpj1O1ERdZjVEtWjNn945C61gDSfV6qBcewbGc7HOa0ayKVFlm9eFDXNGkKdtOJJPu9QTcTzMNHlu0AL9hlFngrkgyC/ftZWyAFF+/YfD7/gQ2HD9GlNPJyOYtSx2sArStUZO2NWrmP569awcp7txCrYq8us6ygwdISEulSbRY41rZRMBajroN6ci7f77EtFdms+PP3URWD2fUfUO49LreyEEWjxdHlmUmf3I31z9+NX/N24Cu6Xz1zHS8ub7iX3wG1a6il+CiXFxaXMDO8nvW7rLz4i+P86/BL5BxIhNPthdFVVBUhdEPD6fr4EsKv8i/FAj8xQcGVspwLHygNODy0TKLfgiS4iNJqPaCH13vfbk0YIDp8+vc+3/f43La6NutCeu3HirRjKrLoWFX/cRE5mKZcCg5CssS7wTh/CJLElHVXKSmB67O7XSojB/VDaWYlDdbkGJ/RXnn8yX8/PuW/BTj7Fwv/31/Adt2H+PB2y4v9fEEAfJmx37YHk+G10uv+nGMaN6ywFrDX/fuDtqmzCNhQJ0AACAASURBVDjZd/OU6dvi2ZGSwrRrxiFJEi8MuILGUVFMWb+GDG/xLQK9un7BBasAWHAqQTrd68EoYtY02x+8rsrZ0K28dF+A3akpfLxhLVuSkgiz29mRcqJAyvEH69cytnVbXhow6Jxugi3evy/g2ljI69G++shhEbBWASJgLWeN2sbxr6/uL/Pj1mlSi6sfGEr8yh3IAXpWBeNw2YmtX53uwzvic2vM+3AhxaWFi3WxFwAJ+o/pxW0vXM+Tw1/i6O5jJeo9eup3yzRMJFnC5lCpGRfLc7Mfo06T2ny+821W/7yOzUu3ER4VxuXj+1KvWW0ALP0wGAdAqY2kNgWpqDuh+sk/gHGAh15Tia0Ty/fv1ACgUSsP/UZk4AixyHV3RD2tvZPb4yct4wTXXraDK7rsBcli0bomzP2jFW5v3ro7j1djw5aNKJIN7eTHnl3VaR6XgmlK7DoUi2H+/T66svsmRvXdwfqddXnnh94BglVRTEmo+iRZ4pG7BnE8KYOPvl2F52QRQFmWUBWFy3u14NqhnfD6tKAXpU6HjSsvbVPkeTxeP3+s30dWtpdWzWoR4rIzZ+GWQjeRvD6NOQu3cPWVlxBXp+iqnoJwOsuyeG75EqZt24rfMDAti9/3J/D66lX8OPZ66kVEApCYk41XL9k1y6kKwGuOHqF7vfrIksSYNm35dOP6Er0+2Mzj+c5v6PSol5cd1TImtkLTniVgzdEjvL56JbmaltcnvYj9p2+Pp1G1KO7qEriuRUk4VTXoeWRJKtCDV6g84v9CObEsiy3LtvPrp4vITsuh86AODL7lUkIjy7baYnFBR0ydKBq0rs/ejfsJjw7jqnsHM3zSIGx2G0f3HmfuB7+V6DyKKmOaVjlUVRUqhAV/zVvPxBdvIHF/comCVUmWuOu1m6nbrBa6piPLMrH1YmhySUP8Xj/Jh1OIrB5On6u70+fq7n+fykzHyngI/BtOFlbSsNSGEHIz+JaUqBer3a4z/qEk5n8by82PHuHy0enY7BaSDEgrsFJHQ/RUJDkcRcrk03/NoFqYG5cj74u1Ue10rr10G3e+cjUZOS5qx2Tx7MSF3P9m3lqU0f3juXPkWkwr7wvSsKT/Z++8w6uo8j/8TrstPaF3SELvhN6LgAVQQUBQ1MVewe7qz1VZy66Kurq2tSuKBUVQEATpndB7b4EQSE9unfL74yYhIfemkVDnfR4fycw5Z84k986cz/k23vmhJ3+ui8dm8TFu0FZCHW4+n9sJjy/QY9IUqyYXP5qm84+ps/HleRXYLDKabqDrOn26xvH0/UMRRQGH3cJ9t/Thg2+WFkm6ZFEkGtSNYkCPZkGvsXjVHv757hxEUUTTdAQBwkJtqEHCVFRNZ9HKPdw2qlvl3qzJZc1fhw7ww/ZtRSxsTp8/yc8Dc2bz69hbAIiNisahKEEtZmfj8vn46+AButarj2EYTPjlJ1KcxXOPBOJiyRBc2RjArb/8RK8GDRjbqh2SIJy3e1V1nT8P7Cu4Xlmu+p+1qxjdqg0GBlE2e7mtrTc0b8nve3bjDLDRoek6Axs3Kdd4JlWDKVirAMMweP2O/7Jsxmo8eQlgNi/ZwbR//sQ7K16mXtM6lXatZp1jg4pIi83CiAeGcvMzNxbMa/H3K3m4x7OcOppKVM0IFKsSsPTO2UiyRNP2jdi5em+lzd3k/CJKIhsXbUOSRdQyePFIikTd+Fp0Htq+wIXd4/Lwzv0fs+CrpZBXCmPAzb24/507sIfY/JkW0+4AdQ+ggpHnqq7ugew3QOkC3jUEdw0+g2Kz8Y+vrDRploHNXvgz7gJ1H0bWPxAip6K4p1I9MhdZKpQswaIRE+HkvhvW8OrX/bh7+Fri66XSqHYajWqlc8+ItditRa0/T4xbSpcWR0lODcWiqOQ4bWTm2EqcoyCUvmlkYnIh8RQqPeUuZPFcvm4fX/60ijtG9wBg5DUdqVk9nE++W8GhpFRC7FaGX9WWCSO7oiiBXYIPHDnFlP/MKWZJ9XhzgrtlarpZ49Wk3Hy6MRFXAEGhGwZ701I5lJFOo8goBjWJwyovLLNgFRCQ8zyJEk8c50B6eqXO+1LEp+tsTTnJ1pSTfLB+3Xndnq2IMHapKl0/+QBBEKkbHsbzfQbQr1Hj0jvm0bVuPXo2aMDyI4eLuIzbZZmHunSvUJysSeVjCtYqYNmM1SybsbpIXKnH6cHr8jJlzFQ+2vhGpV3LYrNw579u4eMnvsLjPKNCJFkiLCaU6+4dXHDsPw98woKvlxTMK/NU8axvwdA1nX0bDlbavE3OP5qqU6dxjTIn0DI0g5fHvkVUzQheX/gPqtevxrPXvsrO1XuKuIkvnLaMwzuP8c7yf4Iv0e8GzNnxpDoYuWAbDNZB4PoStLxswUbgMk2CINCy41EI6KroBfd8dC0b3DOLiNV8FFlnYKd9bD9QnQGdDiAIMOXOP5Eko5hYBb/IHdDpAEs3NcJhUwGhxLhVh83LqH7b+XVZWzJziy/oLbJK27hkZEln24Ga5LisQccyMTnfuD0q385cx603di0oUdWrcxy9OseVeYzps9ajqsXdBUtac9ptCp3aNiz3fE0ufw5nZPDN1k3sTU0lNjqaW9u2p1FkFAAnsrOD9pMEgZTcXBpFRmGRJKbdOJpbf/kRl8+HV9ORRTGg2AWQJZEhsfH8tGMbry1fGtDKdqVzKezJqoYBhsahjAzunzOL1wYOpkOtOtQJC0MqJWeMIAi8f81wpm/bwqebEkl1OmkcGc1DXboxsEnseboDk9IwBWsV8PM7cwImQTIMg6Q9Jzi253ilWlmH3zeEiJgwPnv2W5IPpiDKEn1GdeOeNyYQFuWvO7Z/8yH+/HJxkbqWZcXqsCLJIs6syzNe40ohunYk7fq3JrJmJGknSt9F1lQNV46Gx+Xl6aEv89gn97F73b5iMc0+j49DW4+wZckO2iSsO2NVLYYbsl9DiPkZIWQ0AIbzJ4ysKQS0uBo+MIIvUhBk8G2leNbhMyiyzgMjV5PvIVQrJjew/s1DlnSycq1oGjhsPtrGJrNxb20MQzyrncaQLnu4/Zr1jB+8kcf/ew1b99cqOH9Nt108fNNK9DzBq0g60xe25dPfEjDdiU0uFlweH5//sIq7xvUKeD4/v0EwF7vdB1LQgnj4iKKAIAhohVyDFUWiYd1oOrauH7CPyZXLb3t28eSCeWi6jk/XWXXsCN9t28JrAwczvFkLmlerxuHMjIB9c30+wi1nNgSbxVRj5d/uYcmhgxzMSKdeeAQLD+5j5q6dxSx4qq7z75XL2HjieJmyA5tc/LhVlUnz5mCXZeyywqPdezKuTbsS+0iiyPi27RnfNkDCSJOLgoqlqjUpkfTk4GJAtsiknwxsUaoI2ek5rJq9Hke4nY+3vMnsnG/43TmNZ755hOhaUQXt/vpuOb5gGYEFUKxywfwkWcTqsCKIAtXrV+O2F0dXSOiaXBwIooCkSHQflkB2Wg6TPry7XP11TefU0VT++PyvoJ8Dd66btbM+hpz3KElAYmRjpI5Bz34T/WQ3jKwX8tqfZaEU7OCYAHIJC1vDAzlvlzr//LhWAE0HXQ/+2PP6JOavi8en+efz2NjlhNh8yNKZMfL/PWt5C0Y8fStfzOnI42OXAgaCYPDgjSuZNGYFDptKqN1HqN2H1aIxesBWbuq/tdT5mpicT77+eTUnUoq+kw4fS+WpV3+h/5i36Dt6KpNe/JG9B1OK9a1ZPUgmb/yZhRPaNsCiSNhtChZFok/XeN55YbRZ1sakCGkuJ08umIdbVQvqePp0Hbeq8vSC+Zx2OhnRrEWJY9w680cyC2X2lUWRgU1iubNjAkPj4rm+WUvEAJuFumGw8ugRU6xehrhUlTS3i5eXLebLzRsu9HRMzhHTwloFxHVswomDKQFjS70eH/WanZt1dff6/Xz+3HdsWrQNzachKRIWq4IB3P/W7Vw9cWCxPu5cd9BaraIoMODmXsTUjSE0wkHfMT2oUb9awfnczFw+fWbaOc3Z5PwiKRIhEQ6y03IQRQHNpzH7g3nMfHcuLbrFU2rqvWIY5GQ4ESURTS8uSAURZGErUJo7lQFGGuR+SlG3YYmCx5EYAY57wTEW3C0h61kCx7xqoG4q+WoGFF4bHz8dztGTEXRteZSzK3n4VJEFibH8Z9JvON0KomBQt3omXzz7E9/92ZblWxqh6SLp2TY03d852ykxY3FrVm1rQKPaaYwesI2EZknYLMV/R3aryoShG5mxuDW6Ye4Vmpw7lRFDrRvwyx+buH9CX8AvVu96ehout7dg7PVbDnPfs9/y3ktjaR53xpPgpms6sWHrUdxnxaQKAtSsFs4bz47E6fJyOi2HmKhQQkNMt3iT4vy2Z3fQ95GBwew9uwi1WLCIIt4gLjJpLhevLV/Kq4MGBzz/y64dqEbV1as3uXhxqSpTV61kXOt2KKWU8DK5eDFXTVXA2Keux2JVih232BR6juhMVI2ICo+9adE2Huv3PInzN6PlFXnXfBquHDfuHDf/feQz1s0ruohXfSrofktbICw2C9fcfRV3TBnLTY8PLyJWAUIiQgiJMIPOLxVsoTYShrRHkiUM3UDLyxDq86joms72FcEXB8EQRJGuV3dAClKPUVF0+lx3uoyjGRSPcdUAGaotBPsEyH0fUtpC9j9ArHj9s7NDVyyKxrszupPltOHxnrkXt1fidKaD7xe0RZF1IkL9ydJOpofidCu0apLCO5NmkeOyFIjVfLyqzInUMOpWy+KqhH1EhwfPgmy1qESFlV7fz8SkLFRWwq8de08U/Pv9r5cWEav5uD0q73z2V5Fjnds15IYh7bBaZMS894vNKhMWYuOVp0YgCAIhDisN68WYYtUkKCm5ubi1wBZOj6aRnJONTZaRS4hFNICZu3cELdOX7fVcErGYJlVDfnIuk0sX08JaBcR3bMLjn93Pm3d+gCiK6LqOrhu069eKxz69v8LjGobBW3d/VCS50tl4nF6+/Mf3dB7i98NXfSpPXjWF3ev2BbT4KlaFuA6NadE1PuiYp46l4sw2F9kXM/ZQG617NccRZmfIHf3RdYOXby7dXbas2EKsXHVbX/YkHuDPrxYXidG2ORT6DEulcYtz/IwIMmQ+Br5tQN5YRrY/WVMlUTMqF1kymDBlNDf03caAjgfQdIG1O+txOsPBhKFn3IasFp0aUbls2VeLXm0PsWF3HSQx8A69x6fQuUUSFkUjK9eKRQkcxyuJBk5P8c0sE5MLSWS4veDfazYeDCqEt+89wdzF25FEgY5tGlAtKpQHbuvHwF7N+X3hNtIycunQuj5X92tFiMMUqCZlo0W16oQEKUUToii0ql6Dvg0bB42XzsejaeT6fIRaLMXO9WvYmGVHDhcpi2Ny5aAbOtZKsq6qus72lJP4dJ02NWqadVrPE+ZvuYroN6Yn3YYlsO6PTTiznLTs3pT6zeqe05jJh1JIPZ5WaruDW44U/HvhtGXsTdyPN0jsYe+RXZn04d0lxhTtWrMXi01BDRYDWwYkWSyw9JlULha7hVuev4nRjw8vODbr/XloAbJ3loZkkRDy1gSqT8MWYkWSJV7+7RkkSeKh9ybSrHMs01+byaljp6lWJ5pRj/Zg6PBXz/1GDA18mynuVlzRz40VlI5+t2HjjEvxU+OX8Oi71/Ll3I5892d7/nnXfG7os8PfQyn6O5NEgw5N861PpcXdGQgC7DsWTesmKVjPcgtWNb8wdgUUrAaiYNCsQQo7D9csw7VMTCqPZk1qFvw7mIUKQNcNpv5vAeCvpzpsUBsm/W0gzWNr0Ty2VtB+JiYlcVVsHC8u/Qunz1fECioANllhSGw8Vlnmub79eX7RghItpe+sWcmzvfsBfqvatK2b+ShxLSeys00L6xWMLIo0iYo+53Hm7N3Nc4sW4NM0hLzSfpO69WRih06VMEuTkjAFaxVic1jpfWPXShtP9aplWsc6Cu2W//bh/IAZi8FvNRv16DDsofaA5/Oxh5V8PhiSLGF1WNBUjevuHUyLLvH8c+xbFRrLJDgWm8KXz09n+mu/MPj2ftzy3ChqN6kR1H23MI4wjcGj0+g8IIucLIllv9fljn+/y8JpS1GEbTRpG0O7q8YQEuFPfiQIAkNu78+Q2/sXjKHrOpx+D/RztcJXZmIvGZRWCNH/A/cfGDnvgnYUhBBax6p8+MRMPv+9EwM67ad90+NYldJFcZeWR2lYO50dB4svzK2KjxCbf/5N6qZx/HQYtWJyCsrnuL0SuS4LU6cHzsYKArohsOdo9fKHF5uYnCOHC22Edmhdn/WFNj3PxlUoS/icv7ZRIyaMW26ovPecyeWLqut8v30rX27aQIbHTavqNXiwSzc61a7L9JFjuP3XGaS7XAW5ByJtdr4YcWOBBWt8m3YkZWXyYeK6oNf4btsWHu/eC6ss8+zCP5m1Z6eZUMkEt6riUlUcSsU9nFYdPcLjf/5RzEo/ddVyQhULY1q3OddpmpSAKVgvIerE1SpVhMgWmWvvuarg59zM4PF0kiyVqVRNu34tg8a/Wh1WZIuMx+kpsMBa7RaaJsTy2Kf34cp2Uze+VoEojqgezss3v0Xm6SzM/AeVQ06632XW6/bx63t/sHp2Iu+tfRV7qA1XdvC/b+2GHt6evRerXcceYqDr0HVQNvbIp7ntoYN5VkkBXF+gG9ciRExBEIq7WpHzL9AD1fSVQGoEcjyo+0BwgLobCFb2xqDi1tR8RP91wp9HsF2LIChgHw62oRjpD4F3GeClSR0PU+5aUK6RZcng3Um/8cDUYew6fMYiJYk6AzrtZ1Dn/QBEhXnwqRLz18XRuFY6kqSzYU8dvvuzHTkuW4nX0HQJSVQxEANkMzZoG5tM73aH8Hhlpv3ZDpcnwN/DxKSc5BYKM+nSvlGJgrUw+bVcbx7eGUkyU2KYBEfTdSbO+pn1x5MKBOSSw4dYk3SMVwcOZkSzFiy57U7WJh3jcGYGDSIi6Vq3XjHvryd79uGTjYmoQZIvqbrOkYwMNqck8/OuHfgCJAk0ufKwyjK7T5+iQ+2KJz19c9XygC7lLlVl6uoV3NSqNaKZAb3KMAXrJYQkSTTrEsfGBcFLY0RUC2PsU9cX/NxhYBuOHzhZkKCpMD6Pj9j2jYocMwyDQ9uOcPLIaU4eOoXqVWnWJY4nv3iQV25+G5/Hh54XR2ILsdJhYBue+OIB/vxyCStmrsXqsDL0jv70vL5LEXFtGAY7V+/Bme3iP6teQVN13LluDm8/ysdPfE3ayQzTrFQJqF6VU8dSmffZX7w27zkeH/ACrmwXPk/xh+wzHxwmLEoryJYrimAP0cEXYPfaPRdDkBAiXily2FAPgPNbgorQyHcRlTgA9NwfIXtKCbM/F7EqgqU3guMmsA5AEM482gw9B+PUUDDyy3JUfLddlnRenLiQe/59A7luBQyBJ8YtZUjXvUWyEVePdDKs5y5cbpnZK1rw5ZxOeNWyPG4NNF2iZlQ2mbl23F4ZEBAEnadvWULf9odw2HyomsCwXjt47N1r2ZcUg+lCbHIu5LsEuz0+/li8o1x9XR4f2bluIsPNxHwmwVl4cD+JAWqdulWVR+fNYfmRwzzUpRtd69Wna72S6/TWCgnlWHagTVLwahpDvv0SkXPf/jS5fNB0A0eA2ObS2HkqhffXr2HjiRMczwleFz7T4ybd5SLGYT4HqwpTsF5ixLZrxMaFWwOKO0mWuHHSddgKJbsY9dgw5n+1pJhgtTqsXD1xAKGRIQXH9m44wD/HvkXKkVOoXn97QRSw2i3Ua1qHV+b8ndkfzmfX2n1EVA/n+gevpv/NPZEkiRsfuZYbH7k24Jx3r9/PiyNfJyc9F0EUUL0q7Qe05tnvJhPXvjEDxvVmQtyDJAeo82dSfrwuL/O/WsLIycP47uhHLPtpNT9Nnc3+TWeSqdSs76FRM3ex0i7BcYNrNkbY4wjimTgQwzWX4AJQRPD8hSHVwsh+E1w/Urluv4URwLscw7sCBAeGWBfsgxHsYzAyJxcSq+d4FQFqx+Qw89WvyXJasVt8WC3Fl0WCAF6fwMtf9mfV9oZoer6jb2Bh2bpJMvddv4ZWjVPQdIHN+2rRPv4E2w/U4KG3R/DC3xbQs80RLHmuy7JkEBXm4b+PzWLKF/1ZvqVxpdyfyeVJfqxVMPYcOImq6Tz0/PccOlb+TJoOu2npNymZH3dswxkgqRL4n4y/7NzOH/v2MH3kGFrVqFnk/M5TKaw8dhRFFImy2WkSFU1ybk5QKyuYYtWkKNF2O02jg1cccPl8zN6zi3XHk6juCGFUy1Ycysjgobmz8Wgaeikp2Q3DwH4O7sYmpWMK1kuMfqN7MPuD+XicxS1akizSZ1S3IsdqN67J6wue55Vx75B+MgNRElG9GtfeNZC735hQ0O50UiqP938B51kupIZu4M71cGjbEb568Ufe+OuFcs03/WQGTw58sdi4Gxdu5aVRb/DavP9j55q9ZJ4KvFt6uRIaFVLgylsVqHkbFBarwsDxvRk4vjdj691N6vF0ACKrqfi8AlZ7eczaIvj2gNX/GTPUA+D6Fn9JmkD4MPRsSLsZ1AOUXqP1XCg0ByMLtCzI2Y2R81GVXFcUITI0mGuzH6ti8NQtS/hlaStSsxz8vrI5qlZ8h6BjsyReu/ePgtqtomjQselxRNFfLijU7qFnm6MFYrUwdqvKHdckmoLVpAiCIOCwKzhdXkRBoF6dKA4fC56wb9WGg6xYv59Dx1ILPGjKgiyL9OsWj0UxlxImJZPjLXmzUgdyfT6eXDCP38f51yYeVeXe339lTdIxVE1DM4yCvXrTp8SkPKQ6c3ll2RKSsrPYmpJMNUcIt7XryPBmzTmUkc7on6bjVlWcPh+yKPLZxkREQQhabqkwoiDQrV6Dc4qPNSkd8y1zidGscxzdhyewatb6IqLVFmLl2ruvolajGsX6NO8Sz5d73+XwjmPkZjpp1Lo+IWe5b818by4+b/CFverT2Ll6D8f3J1OnHNkgZ384318H9ix8HpWty3ZybM9xTh66jC2rZ2XQsdgU+o7uwfJf1lTZJRWrEjDZV2GBnHTAimItrw+2G0PdCZYE0DMxUkf7y84EQ3CAIIJ6lKoVq8HQKSiPc4EID/Fy+zUbAWhSJ433f+6GqolouoQo6OgGPDZmeYFYzUcU/TU2M7JtxIQ78WkiFiXwxkCN6Krb+DC59LBaZK4b2IYHbuuLJIrs3JfMIy/8UGIfTdf5a8XuIgmVAiFLIqrm3zix2xRiIkOYPHFgpc3d5NJDNwxWHDnMuuPHCLVYuTa+GXXDw4u1G9CoCZuTk0sVAAfS0ziRnU3tsDCmLF3EmmNHcWvFn31mBJFJeXBrGp9uSiz4OSk7m0fnz+Ht1SvQDcOf7CvvXEmW+7OxSBIOReGf/QdV8oxNzsYUrJcgz3zzMLM/nM+Mqb+RlpxOzYbVufmZGxk4vnfQPoIg0KhV8LiQjQu2BoxzLIxiVTi250S5BOuWJTvwBlkEibLIB5O/YPOS7SXWlr1UEQSB0U8OZ/+mQxzecYzq9WIY9dhwdF1n2c+rq+SaoihgD7Ux4sGri52rE1eLg1v9yVRyMmUW/xJJ3+szsJXZympAzn8wcj8B62Aw3ARfNigg1gTnj0DwxF+XM/mZLvMZ0mUvsXXS+H1VM9bsqI/NojKk8x5qRgcW/YIArZukcCozBFkK/gJdt+PcymWZXF4M7NmMyXeeEZE//p6IN8CmYWHat6xXosuwKAjceHV7OrSqz7wlO1A1nf49mjGgRzOsFnMZcaWS4XZx84wfOJaVSa7PhyKKvLV6BfcldOHhrj0AOOXM5ecd29mdehpREBAQMEqQm5Io4vR5cfp8/LxrR0CxamJSWRzJyqxQP5skUzsslIGNY7mjQydqh4ZV8sxMzsZ801yCiKLIiPuHMuL+oZU2Zlh0aKltVJ9Gtbrlq2MVVTMCQSBgIXqv08uGhVsK4mWDYbEpQUXvxYogCdRuXBNPrpc7/nkzTTvF4vP6+PSZacz67zx851DTNhiSItG+Xyse+fBuompEFDt/y/+N4t+3/5cadTIZP/kkHfr4hZLqA91QsFgVMEoRl0auv43re0pMXiRGgXac4BmBrxz2J0Uyf21TGtdJQxAMWjVJYeJ16xAFgaTTYZScVNDA6baweENj+nU8UKz8jssjM/X73pQUH2tyZbFqw8EiPx9JSg/4/M1HlkXuu7UPJ09ls2rDgYBWVlkWueWGrlSLDqVvt6aVPWWTS5TJ8+ZyID0NX55FKv//HyWup32tOvh0jYfm/oZhGHg0DUte0oSSSnfJokiDiEiOZWeZGVdNLkqskkSXevXYcSqFLzZv5OstmxjerAXP9u5LuLXkSgAmFccUrCYAXHv3VWxfuTtozVZBFKjVqDpN2jYs17jX3TOY1b8lBhxX1w30UsQq+DPf9h7VjYYt6zFtyowSLQHBiK4VyT1Tb+PVce+Uuy8AAlhsFhxhNjJSSo63FSURxSpzfF8yM9+by8z35iJbZMKjQ8lOz6kSsSpKIrIiMfrJ66nduGbANn1GdceZuoregz7GYtWR8r79qk9EsYaB0gm8Cyjd2cogeNwqgA30NM4lG29lc7a183xxLCUMj0/htqs34LCpxeYSHR58g0DTYeU2//dt6ve9qVMtm9i6qVgVFU0XAYEdh6rhUyVMsWqSj++sBHsN60Wz7/CpgM9NQYBn7h9Ci7jaxDeuSf3aURw8llpkDFkSadeiHqEh1mL9Ta48Vh87ylebN3I0K5Odp1ICJjdyqT7eW7uK7adSipQB8eZZS62ShG4YBQI3H7ss81CX7iiSRIzdUS7XTBOT80H+ZsvaQq7qKjBz1w4STyTx+80TCuoGm1QuZuE0EwB63tCF9gPaYAuwKLHYFCJiwnhx5pPlHrdt35YMvKVPkXFlRUIsR80+XTdYMXMtM9+dW6IrUUlE1Yqk703dkS1la5E7rQAAIABJREFUTotbBFEUufX/RvFj8qc8/vn9KNbgDyRd04u5OKtelbTkjFLdroPVuy2N/Gu+Mu5ttBJcqAbf+Cf2kDNiFUBWdAQjC7yLKHtkUEnt3FxMYhUujFgFOHwykti6aQVi9ey5iOKZeNXCGAaIAsxd5bdmuTwKD0wdzuPvXcPncxL47PdOZOZYsVlUpBLchU2uPFo3K1pncOywBCxK8eeeKAiEh9pYv+UIK9bvRwDemzKW4YPaFmmvajpbdycxfOIHbNhWtvqsJpcnLy9dzMRZvzBv/162BxGr+exOPR30LSEJIne070jdsHDsskKIYiFEsfBw1+78rX1HAMKtVgY0alLp92BiUlEEoFFkJLphFHNV9+k6yTk5/L5394WZ3BWAKVhNAL8ge+Hnx5n80T207N6UGg2r0ahVfboNS+Ch9+7k64PvUzeudrnHFQSBR96/i+d/epxu13UivlMTrrn7KvqO7l6ucXRVJzfTWaRkT9nnALUa1WDe54tLdI0r8fqazuwP5mMYBm37tCxWzPxcESWRhq3qExZVumt2SfjcPravCPzANLQk0I4F6alysYnMSx1BgI5NT2ANkizp7LZn/6zpMKL3dqxK/t9FYNvBWnz1R0fSsh2E2D3E1k3PK5lTdhw2D7JkxoVdjtisMhPH9ihyrHlcLR68rS8WRSoQoqLgjyPMzHYzd/F2XnjrN+579jtEUeCh2/thtxUtU+P2qDhdXp569Reyzsr4HoysHDfHT2YUs/iaXJpsOHGcaVs34VJ9ZdrWdChKEetqYZyqD7uisPT2O/l17Hi+Gzma9Xfdxz2duhR5t7avVfZ8GSYmVY0kCNQJC8capB6gM680jknVYNqtTQqQJIkB43ozYFzw5E0VQRAEOg9pT+ch7QuOJf65mdWzE3HllD2Lq6EbeJxeJEUqVle2JKwOKz1v6MK7D3xSrn5nk3Eqk9TjadRuXJP2A1qzcWHpiarKhAC9buzK/33/KC+NfpPlP6/BKEdpicIYhhG8XI7hxtyjOr/YLBX/fMgS9OtwiO6tv2LG4tZ8MjsB3fD//Xq1OYzD5v8sj+q7jR8WtcGnBnuc58e3+j9Tk25azitfD6jwvEwuLiRRQJElQkNtBe69Z3PD0A706dqUhSt2sXDFLnbvP4mqnXnGuNw+9h1K4X/fLqdty3r41MDPSV03mLt4B2OGdQo6n+RTWbz2/jw27ziGJIlIosDNIxKYMLI7YgU9SEwuPJ9vTCxzAiS7rDAkNp6fdmzHqRaPibbLCoYBKbm5xAWpjZnqdPKvFcvOac4mJpWJahisO56EIgb31JMEc41VVZi/WZMLQoeBbYjv1ARLOQvOC6JA+36tytzeYlO4akJf3n3w03KJ40AYuoGclxHzuemTaT+gDRabgiPcjj3MRnhMGFIFXI5tDisjHvAn0LrluVFYbCXX8hJEAYTAgtad6yauQ6Pic/ftxch4GiibdSQ4FXOpvlI5V0O8KPprrY7st40HR60ivt5pnh6/mHZxJwq8Be4cvp4RvXfgF6TBNzrqVc+gecOTfDyrW4ntTC4dRAGG9G3Jh6+O55eP76Frh+D1eGOiQrjp2o4cOppaUJqmMF6fxqwFWzialIYnSIkzj1flwJFTQa+Rnevmrqe+YcO2I/hUDbfHR67Lyze/rOW9LxaV/wZNLhrWHk8qtY0iilglmYkdOvF4j95BNyhcqo/PNq6n35efMHHWz2S6i76bDcNg3M8/mE8pk4sOr6bh0wNv3DgUhRHNW5znGV05mILV5IIgiiKv/vEcIydfS2hkCAAR1cJKzUJcr2ltJn98b4kxpPnIFpl737yNlCOnceeUTagJghBUZDRsVY/I6v7su/ZQO6/8/nf+t3Uqj31yHy/NfIofkv9Hr+u7oFjLXjza6rDScVBb2vT2P+SatG3IS78+TUzdaJQA5SJESSQ8SsLuKL7gtNh0ug/JZP+mfUWOG9pxjLQxoG4pYSZ2oCzzFgAzC15JVNTtvCTsVpXre+/gv4/NZEjXvUSEnkliJokGD49azRd//5Fa0dlYZBWrUlhw+K2rMeFObBadXLeC+ei/PMh3xIhrVL1MYQqabuB0BS8h5vGoVIsJxaoEfr5aFIl6taOC9p+9YCtOlxf9LA8Rt0dl5vzNZJbRndjk4kLTddJcJWeQv6lFKx7p2oN5t9zGo917Emqx8OWIkYRbrYQoCpIgFEkNl+Pz4dE0lh85zISZPxVJCrblZDKHMtKr6G5MTCpOs5hqPNC5K/azEitZJIlGkZEMiY2/QDO7/DFdgk0uGBarwt/+OY6//XMcuq4jiv5F9OIfVvLv297D5ym6y2+1W7jrX7dSs2F1xjx1Az+9OStoVmPw1yR1ZrtYP29TmUVE4EyaAlaHhUc+uLvYuTqxtYrUpX3sk/tIP/kaO9fsxRegPIRikYmsEUF2Ri5RNcK5cdJ1DLtvcJHFZseBbfjuyIfs33SIpT+vZv7ni8g87S9BkzC4Hd0HH6Ru/TW88UgDMk7LSBJ4vQJ9rsvg7heOs3nNv9BPngBBAtt1oOeUXDNVikWIeA1Dz4GMO0r5Dakg2MAoqTDBlUtp2YhLOl9aX0k0kKUzv/Oz2zapm8H3L01nz9Fq/JXYhO8WtC90VmDz/rrUiMrG7SnpsW+Wx7mUUBSJ5uWoiy1LItFRIaQGCRuICLfTr1tT3v7kr4DnBUHgmv6tg46/dM1ePEGyoMuyxJadSfTuElfm+ZpcHPh0rdSn/auDhhQrQ9Ohdh3WTLyXPw/s488D+5m/fy+eAMlq9qensf5EEp3r1ANg1bGj6FWx82dicg7YZJnHuvdkUJM46oWF887a1RzJzMAqybSuUYM7OyQgi+ZmcFVhClaTiwKx0Je83+geWGwK7z7wCTkZ/oWVPdTOA+/cQbfr/LFTt70wmmYJsXww+XOO7z8ZcEyv28fXL/2IplY8i6ooiXQb1onbXxpL49YNSm1vD7Xz5qIX2ZO4n0XfLmfpjNWkn8xAlCQcYTYmvjqeIbf3L3UcQRCI69CYuA6NueOlseRk5GK1W7DYLOxedBvxLZx8sWoXB3bY8LpFmrRyYrX5RU9ss31geP3awzkNfwmakn4H4RieZf7aqdQGTpQ8OcOJKVYrhiCArvtdfQvj9Ylk5VqJDncVO1e4b1nGb1Azg31JgePCUtJDEQW9lA0cHdMCe2ng82n8/McmBvdpWeayM7fe0JUPpy3FfVb8vc0qM/6GLtisCq8/eyOPTfnJnw3To2KxSAgIPD/pGmKiQoKObQ3gFZKPIPgFtsmlh1WSqR0aRlJ24JJuDSMii4nVLI+bvw4eINfnI6FOXVYdO1pMrObjVVUSjx8vEKxWWUYWxRIz3puYnC9E/BbU53r3Y1AT/4bbDS1aIYoiTy+YhyjAxhPHefz0XGJWOJg+cgy1QsMu7KQvQ4SK1LSsKhISEoz169df6GmYXCQYhsGxPcfRdYP6zeoUEbX5eN1eHujyNEl7k4tZZCsDW4iVF2c+RceBbSo8RvrJDDwuLzUaVAt4D+Ul5/g7SN73sdoCf3cvVM1Rk7Lh9Ym4PAqypGPgT8zkfwwLiKJOOSo+AWf+3v6wRIE/Vsfzr2l9CW4pLdmKKosaqm4Ki4sJm1VBEgVyA7jzKrLE0L4teer+IWUayzAM3vpkIb8t3Frg2WEYBlf3b81jdw0qiDvMyfUwf+kO9h85Td2aEVzdvxVREcHFKsD8pTt4/cM/cQV4FtttCr99/kCJotbk4mXGzu08v2gBrrMy/9pkmTeuuppr4psWHJu+bQsvLlmELApoeWvMag4HJ7KzC34ujF2WeaZXX25p6/cKScrOYtBXnwUVuCYm5xMB/+d8cGw8bw6+GlEQ2JuayojvvymWCVsSBJrFVOO3cRMuzGQvIQRBSDQMI6Gs7c03h8lFiyAI1G9Wt8Q2FpuFd1a8zDdTfmL2B/NKdBGuCO5cD0t+WHlOgjWqZmSlzcfQjuOw70IrIYuwKVYvbjJybIx5fhxNG5xmWI+dDErYi82qU1GrtU8VycyxER3ut3zXjsmmZLfekj8gLRqdZPeRGiCA11dYuJofrAuF2+MLWEsVwKdqzFu2k8fuuQq5DLsdgiDw6F2DGHd9F1ZvPAiGQdcOjaldI6JIu9AQKzde3aFc8xzQoxk/zdnA/sOni7gGWy0yk+8caIrVS5iRLVqR5nTy9pqVSHkbr5pu8GTPXkXE6tqkY0xZugiPpuIppDdP5uQEHVs3DIbGnRmjblg4Ezsk8PmmxGIC2cTkfGMALlVl/v69zNjZgJtatubzTYn4AmyoaIbBwYx0dp5KoUX1GkHHPJSRzuJDBwHo36gJDSMrb514uWK+PUwueRxhdu7+961oPpWf35kTtF1MnSgyUvwuTQ1a1OXUsVScWS70ABkzC6MHKfFwvjG0kxinbwAjE0n2u5YKgilQLyUMwy8wNV1k1+HqvP3wb3litez9C/+9VU0gJT2UqdN7MvXhuQC0bnKS6pHZnMoIpSIic8+R6nz53A+s3t6QrFwrpzMdzFndDE0zra4XCosi4S2hJJeu67jdvjK7BQPUqh7O9YPbVcb0CpBliXdfHMP3vyUyc95mcpwe4hpW529jepDQtmGlXsvk/HNXp86Mb9uexBNJCAgk1KmDTS6arO+D9WsCikyfriPit6bmn8+3XD3RozfVHI4i7R/u2p10t4vp27aYASgmFwUuVeWTDeu5qWVrdqeeDugtACCJIgcz0gMKVt0weGrBPH7bs5v8Tep/rVjKiGYteGXg4GKu9SZnMAWryWVDtXoxWGwK3gDJjqwOC+OfG8XA8b0xDIOQcAenk1J57+HPWD07ES2IKLWH2uh5Q2cM7QQIIQhieFXfRlCMnI/AyCY/HtWM7b/0EASoEZVLqN1Dr3aHKlSnNT3bwozFbZi5rCXZTiuG4X/B/bkulkEJ+1F1EYuiUVGLqEeV+eS3BJ64eTkOm8qa7fVYmBiH0xSsF5Ra1cNJPhU4htBht+AoZ4mwqsJqVZgwshsTRna70FMxqQIcikLvBo2Cnt99+nTQcyEWC4907cHCg/s5lpVF7bAwWlarjkWSSMnNoUZIKNO2bub1lcvI8lSut5SJSWVwMtfvKdA4MorNJ5MDJgfL9Xr5dEMiYRYrvRs2KnLukw3rmbN3Nx6t6Lt/9p5dxMfEMLFDmT1krzjMJa/JZcPA8b1LSMEK/cb0wBFmJyTcv5NbrW4ML8x4gtk5X9NvTA8s9qI7xYpVoX5TK506PohxaghGSnf0tAkY6pGqvpXAeOYCpnvUpY4kGrSPT2LitevKbR33qSKT/zOMr//oSFauHcMQ8QtTgVe/7ssb3/XC0AVS0oMlfCiLrUJg4fp4nv90EFv21aRaZA66bu76XggEQcBqkXnwtn5MHNMDW4ByXjarzPjruwSteWlicj6pERoa9JxX0xgaF88XI0bSukZNtiQn8922rby8bDF9v/iEG76fxv8tWmCKVZOLloYRftfddrVqBc1kbQAbT57g3t9n8e8VS88cNww+TlwX0APBpap8lLiuSuZ8uWBaWE0uG6JrRTH543t4++6P0FQN1achW2QkSeSprx8mLCrwi1SxKDz9zcN89+pMZrw1G3eOG0mRGTyuGnc+vRhJLFR/zrsWI3UUVJuLIAXOxGpiUhKiCM/f8ReyVP7s1Ys3NuH46XCMANZTVZOZtyae2LqpOKw+MtVAFtHCL9iS41zX7mjA2h3+zNiKrCIIep411xRGVU1YiJWa1cNpVC+GMcM60SKuNoZhcPREOtNnrUfKi1XVNJ2h/Vpx8/DOFb6Wy+1P5GS3XRwWWpNLm4kdOvH0gvm41KKeTpIg0KZGLeqEhfPgnNn8sW9Pse2zzSeTz99ETUzKiV1WeKBzVzafTObV5UtLbe9SfXyxeSMjmrekWUw1vJpGhscdtP1ppxNV183SOEEwBavJZcWg8X1o0TWe2R/O5+iuJBq1asCw+wZTq1Hw4HcASZK45bmRjPv7Dbhy3FjtOmJqD+Dsh4sOhhPD+TVC2KQqu498DD0dDA3EGLAOBNcM/GVqTC5lbJaSS8sEy/Q8Z1Uz3F6l+Ik8vKrEsZQIfn3tK5Zsasy/pvXB6fbHNYqCjsPmw+mSMBAwKLuLryLpRIS6OJ1hpuo/H9hsCl+8eVuRY4IgcPe43tx0bUdWbzyErut0bteIGjEV+5ts2ZXEO5/+xb5DKQDEN6nJpIkDaN20zjnP3+TyI8vjYe7e3ZzMzSU+JoZBjWNRpOLPkOvim7Hk0EHm7tuLW/Vh4HcjDrNYeHvoNczYuZ05+/ac/xswMakgoiAgiyL3dEpgcGw8d/w6o1h24GD4NI2fd27nmV59sUgSdlkm1xe4okWYxWqK1RIwBavJZUfduNrc+8ZtpTcMgCiKhIQ7MDxrMAQ5iAelF9x/QhUKVsO7GSPrH6DuBQSQakDIfeCeC0YOZh3USx9dFxAEo1gssqoCgoAsFf8bewNaTQsjkuu2IIrQr8NBerc9xB2vjsQwBP731C/Iko5XlXj7+x4sTIxF1cr2CjAQaNHgNCuzQtB084ValQgCxDWsHvR8VEQIV/drdU7X2Lb7OJNf+hFPoVqsu/YlM+mFH/jPi2NoGV/7nMY3ubz46+ABHpo7G/C7LoYoCnZFYfrIMTSJii7SVhAEXr9qKGNbt2XGzu1ketz0a9iYYU2bI4kiLyxeeCFuwcSkwiiiyKIJE6kVFoZhGKxLSipzX80wSHe5AP93Y1ybdny1eWOxkk02WeaWtpWbBO9ywxSsJiaBEBRKFIVC1bnPGb6dGGkTANeZg9oxyJoCYc+AZyF4S3dHMbnIEYxiVtR8y2pKuoOa0bnFxGzvdofYc6QaXrX4o7tBzQxuHbKBPu0P+YcXQJIMPntmBqomYbf6xYki6zx283KqR+byzfz8siXB3XwFdKLCnEwYmsjanfVMwVqJBMr+a7XIVZ6w6N0vFhURq/m4PSrvf7WE96aMrdLrm1w6JOdk8+Dc2UUsSrk+H06fj+unT6N+RAQORWF0qzaMaNYCiyQhCAIJdeqSUKcuyTnZLDx4gB92bCXUUvYs1iYmFwuaYaBIEgsP7GfKssU41cAW0kA4FIXu9RoU/Pxot55sSj7B9lMpOPMsrQ5FoU2NmjzcpTtZHjffbd3C73t3I4kiI5q1YHSrNjiU4J5VVwqCUZJf2nkmISHBWL9+/YWehokJhqFipHQHIzPAWRuEPYEYcmuVXFtPuxu8SwgomKUGEP4apN9CfrZgk8uLfNGqaf5418KiNttp4daXRpOWbaewyGwbe4LXH5iLImvFLLPB3IsNAw4nR/DQW8PJdVtQtbOFqIDd6sWqqIzst5V1u+qz61CNgGLZpPyE2C20a1mP9VuPIAp+7w5N03n0rkFcO6B1lV1XVTX6j32bYO9+QRBY8sOjZhInEwDeWr2CjxLX4Q1Qc7IwAmCVZe5o35E72ncixm7njZXL+WxTIgL+sjbByoCYmFzs1AwJId3tLvV7UBhJEIhxOFhy251Y5TPvTd0wWHH0MH/s24sAXB3flB71GnDKmcuI6d+Q6fbgzssibJNlaoaE8uvY8YRbbZV9WxcUQRASDcMoc1pkU7CamARBd82FzCeBwhkLBRDrIFSfgyDYq+a6yW0pHjubjwUsvcD7VwkjSHn/eSt9biYXnpNpIbz2TV827KmTlwTJ4Mcp31EzOrfcYxkGeL0iT388mKSUSPp1OECdalmcygjB5VVo3uA0/TocQJF1vD6J17/tzfx1TSv/pi4RREEImhmyvPTv3pQpjw/nSFIaW3YmYbcrdO/YpMrL06iazoCxb6Hrge9DFP2CVTDrAZoA9/3+K/P27ytze0kQCbVYeKBzV95avSJgRlQTk8uZUMWCaujERcfw/jXDqBceUaZ+D8yZxfz9+4pt7CiixNjWbXix38CqmO4Fo7yC1dwqNzEJgmDthSGGg36aM9ZOA/RU8G4Aa89Kv6ZhlOZqYuTFtZaA0haEEPAur7R5mVw81IzO5a2H55DttJCcGoog6tSIKr9YBb/l1WrVef3+ecxbE48gGAzpuhcpgNev1aLxxLhlbN5fm5NpV2bypfKKVUHwWyzPFoc2q8Ido3sA0KBuNA3qRgfqXiXIkkiHlvVJ3Fa8PJcgQOd2jUyxalJAk6hoFFHEp5fNo0czdLK9HqauXlHmxDQmJpcLDkVhUrce9GvUuFh8d0l4NY0FB/YH9ELw6f7ETZebYC0vZjCSiUkQDOfXoGdR3DXXjZH596AudRVFd831uyGXVGtVbu7PGFwSvi2gngDMMhWXM2EOL/H104irm1Hueq5nI0sGV3fbw6CE/ew8FDzhjyAYXNtjF/GNqhMTFUJYqI1enWMZOyyBmKgQZEnEqshIpjspdptCnZqRTJo4gOhIB9a8ElthIVZ6dGqM13vhFvMP3dEfu00p8rkRBL+QfmBC3ws2L5OLj5tbt0UqZ+ZS3TBMsWpyRSIA1RyOcolVoNTvS2FPBZfPx6GM9CuuXrFpYTUxCYbrZ4q6AxfCyAB1HyjxFRra0JIxnD+AugfkWJBbQOZTBHcFzsN2LQhWyN5UQiMN9P0VmpfJlYsogkXUadX4FBk5VqyKxp/r4lixtSE2i8rQrnvo2vIo466WuX1kEiAh2IawdKODl96eg9enYhh+l9MrxUCnyCI+tbjlSRCgQZ0o3ptyM3abQsv42kx+6UcMj0F2rodFq/awIvEAQ3q34Il7B593i2Zco+p89Oo4PvhmGes2H0IAurRvzL239KZx/WrndS4mFzf1wiP416AhPPXnPHRDx1tGS6s/WMHE5MpCNwxiyylWAcIsFqLtDk7m5gQ8Hx8dg0dV+eeyxczYuR1REFB1nQGNmvDKwKuItFVNiNrFhClYTa4YDMNg/bxN/PLuXFKT0mjWJY6Rk6+jYYt6QTqU5J4rAmXPFFdkWPdfGBmT8CdN8oLHkjdWGV7vOW9DtbmQPRXIrtD1Ta4cdB3W7KiHIhu0jz8esFTO2QgC+HwSE18dSbbTWlD3dfW2BrSJTea1+xaA059rwJf9Ha++dzMeb9FyOxdDaoTaNcJ5/O6rePndOWTnegqEpSD45ydLItGRIaRm5KBpFZuwqgVevBsGHE5KJ3HrYZas2csfi7cX+Z0YBng8KvOX7aJT24YM7Nm8Qtc/F5o0qM7rf7/xvF/X5NKjWUw1GkZGsj8tDVkUMQwDwzBKTPtnkSRUTUczZavJFYIkCDSMiKRVjZrl7isIApO79eDFJX8Vi/u2yzKPd+/Ffb/PYtWxo3i0M+cXHtzPTT+mMnf8bZd9DVdTsJpcERiGwbsPfsKfXy3Bneu3mh7afpS/vl3Oc9Mn0+26TsU7WfuD6wcCu+iKIMeVfx56FkbGZIpaUsuTHMmA3P+BEgu+kqysJlcyhgGpmXZe+Gwg+5KqgQGyrPPi3xbQqfnxUvv/a1ofUrMc6IVK2Li8Cpv312Lm0qaM6r8dgC37I9B1DX+Sr6JEhTmJDneRnBpGrvv8uqcLAnRq0wCLRcbj0xAEoSCeNDrCwdTnR1E9JoxQh5Wf/9jIB18vxR2gzEtplCTM3R4f//lsEafTc4K2c3t8TJ+9/oIIVhOTspCck81NP35Hjtfrl55l0J92Wea6+GbM2LXDNLOaXDHohsGhjHReW76UJ3r0Krcr/ehWbUh3u/jPmlUFfQ3D4Nne/agTHs7qpKJiFfzZt5NzsllwYD9D4yrm8XepYApWkyuC7St3M//LJXicZ1x8dU3H4/Twyvh3+CnlUyzWonWuhNC7MNyzwDjbRcMGoY8gVKQWq3tOBWZfGA+4pmOWtLkyCFaSpjTe/qE7v69sjlct9Jn2wDMfDeHzv/9E7ZjsPAFXvG9WrpWNe+sUEasFQ3gVZixpXSBY3R4F4awVaUy4k2dvW0Sb2GR8qogi6SxYH8tbP/TC6zs/rxzDgBMpWTzx8s+4Pb5Cxw3SM508/+ZvfP327QiCwI1DO6CqOp99vxLdMHC5K+Y5EYjjKYHKYhVl976TDJ/4Pj06xTJhZFfq1IxE1w2OJacjiSJ1akaYSZBMLhifbkzErZZuJ1VEEZss49N17mjfkT2ppysto7aJyaWAAbg1ja+3bCTX52VK/0HlHuOeTl24tW0HNpw4jiQIdKxdB6ss89nGxKCZ3XN9PhYdOmAKVhOTy4E5/1uA1xXYkikA6+dtosfwzkWPS3Uh+nuMrGfBtwMQQXBA2GREx5gKzcPQkgFXhfqewRSrJsE5mR7C7ytbBKyXqqoiP/zVhkmjV3IoOYJGtTKLidbMHBuypOMLYnDMzDlTC65142T6d9qPx6uwalsDvD6JD5/4hZgIJ7JkYFX8NesGJuwnOtzFUx9cDYAiafi04lZZSdTQdJHCNWYrgtXiv3c9QLydphucPJ3F1t3Hadu8LoIgMGZYAjcMbc+BI6c5cOQ0U/+3EF3X8fo0RFFAFARiokNwOr1k51ZuogvdMEjLcDJ30TYWrdzNhFHd+H72epwuH4ZhEBnh4Il7rqJbh8aVel0Tk7Kw5PBBfHrZak/+e9BQetRvQJjVynXffV3FMzMxuThxqSo/7djGpK49iHE4yt3foSj0atCwyDGrLPtrYwdY/gn4vRoudy7/OzQxATJSMoNm9dV1ney0wIHughKPEPMDhp4GhhvEmghC8YV2WRHkeAwhBIxAZUgE/F/JyrPwmFya5FtWK2JYO5AUjSJrgQWrLrHtQC0EAepVz0TXQTrr41wjKievvmtgGtdOK/h3eKiXSTetRNUkJFFn2eaGhDk8xWJlbRaNDk2P06h2Gm6PjIHA6QwHmn7m4lZFJdTuITWr/C/4s/GpGhu2HQnqimsYBoeOptK2ed2CYxZFpnlsLZrH1qJL+0b8Om8zO/adoFb1CK4f0o74RjW49+/T2Lb7xDnPLxCabpDr8vLhN0uLzPvkqSzV4S/lAAAgAElEQVSe/fevTH1+FO2CxdubmFQRdlkpvRGg6TpZXg9hVisArarXYPfp02iGucFqcuWhSBIbk48zqEn5Q8cCMahxLFOWLgp4ziYrjGjWolKuczFzeUfompjk0a5fKyz2wC68hm7QrHPJDxVBjEaQ6pyTWDX0DAzPyiBiFSAMwp8FpT1IjQEb5lf0yuRcPEAjQtzoJQjOqHAnAIp8RqwWFkhWi8aI3juwWopvnFgtPm67ZkORYxZFx2HzYbVo9O94EIcteCxou7hkerc7zLuTZtG5RRKKrOKwerEqKtf22IUgGFTUuuqwK9is+ZZVo8T4UsOAatEhBT/7fBo//r6B8Q9/xvCJH/Dmxwvo1SWON58bxRP3XEV8oxoAtG1esmCURMG/C34OBJq3x6vy0TfLzmlcE5OKMLpVmzJZb3TgqQXzeHrBPFKdTiZ26IQSqKCzicklhEUUqRNasbrjtjJu9pSFmqGh3NepS7Hvol2WGdC4Ce1r1a60a12smE8TkyuCqycOLBajCqBYFVp0b0qjVvWr9PqGno1x+gZw/xikRZ5yyHoNkBDCX4Lw5zDdf03KS4tGpwhzBHZbtVl8jOy7vdjxswXyPSPW0qfdISyyit3ixWH1YlFU7r9hNV1aJAW9tiQFF4q6LuDxyoiiQXS4i3/f/we/vDKNj5/6hdn//pJJo1cGnXdpWBSZCSO7U7dWVJnae7wqh5P8lmJV1Zj04g98OG0ph5PSSMvIZfm6fdz/7HesWF+0PNSYYQlBNxMkSWDCyG60ii954SCKQoUk+fa9VWPZNTEpiVEtWhEbFY1Uxl20n3ZuZ/j0b6gZEsqbVw3FLsvYZdlcbJpckgxsHEet0NBy9xOALnX9G5yGYbDq6BH+u24NX23eyKncYEaLknmkWw+mDrmG1tVrEGqx0Dgyiud69+OdoddeEXkOhGBukheChIQEY/369Rd6GiaXKQe3HuYfN7xO+skMJFnC6/bRYWBr/v7tJELCz90NsST0nPch5z0CZxwOhDXv/1dWYWiTymHn4epMfudaVE3Mcw02sFlU+nfcz9O3LC2zBTc5NZRN+2pjkTW6tjxKiL3i7uoen8SNfx9PtQgnHz35CzZL8bi4Hxa25v1fuqEb5VveCgI0aVCNA0dSg7r+n43NKjPr0/tZumYvb3y8oEhypnzCQ23M+ux+5EKWov9+uZgff99QpKSNJArUqx3FZ29M4IsfVzJ9diI+X/H7EwWBe27pzZqNB9mw7Wi57tFmlVnw7aQytXW6vMiyiEUxo35Mzp2XlvzFN1s2oZbxuyUANzZvyeuDrybX6+XLzRt4d+1qDMCrlS0e1sTkYkWkdFPCG4OGcGPL1qS7XIz/+QeOZGXi9qlYZAnDMHiqZx9ub9/xfEz3okUQhETDMBLK2t58m5lcMTRu05Av977Lvo0HST+ZScOW9ajZsPr5ubjrV8ouVsEUqibnQouGp/jmH98zc2krNu6tTVSoixG9d5LQPKlc7sa1YnIYGvP/7N13nBXV+fjxz5mZW7d3OkjvTXoTxYKgoqCgYm9RE5OvxugvGv1q2jcxMWo0RpNojJXYRUUUESwovffeWRa2794+c35/3GXZcu9ld9nC7p736+ULd2buzLnA5c4z5znPs6PStrU7sujfNSdGT1dX+I5Vniwu5vUbvPjRCIo9Too9TpZu6siofgcqBa3+oE7fLjkx05mjkRJ27Tteq9domsbqjQf4+Mv1EYNVCPdZ3bTtMIP6nkwFvvuGc2iTmcSr7y6loNiLrmtcOL4PP77xHBx2gysmD+GdT9cQpPKNua4LunfOZPblIxjUtwM/+9+38Qcq/5sghIgYcOuaYOLonqd8T18v3c7zr31D9rEiAIYP6sy9t06ifZvkU75WUSIp8vt4a+P6GgerEK6W+t7WzXRNTeXOs0fw1sYN+FWgqrQQuqZhCBH14YtN05jQpSsAP5v/Kbvy8wiWFQD0lfVY/dP339IvM5Ph7VRdgppSAavSqggh6DG0axNcufY9HhXldKQnebnt0vrNWHn506HM+XIQv7rxK0b3P4DNiFSyMA3i7yBY/BolRQfZm53Mq/OHsGrbyS/mx18+n1suWcn0CZuw2yw0zSC7+Bze/mE8ULuZx9NhWRaeGG1shACPL1Blm2DGxUOYPnkwPn8Qu81ArzADm5mWwB9/eQUPPfEhUoaL0QgEHdok86eHpwPQv2c7/vD/LueJFxaQX1iKEAKH3eDqS4fx2gfL8PmCmGUtDAxDIyHOyY+uHR/zvcz/ejN/euGLSkHwsjV7ue3B13n1LzeSkVa3dVhK67buaDZ2Xa9TwPmn778jzrCT7zvdyviKcuYQQtAuPoG9hQUR96e740hzuThcXMSKwwfLg9WKfKEQ/1i1QgWstaACVkVpDI5J4HkNUE+ZlebpSG48by4YTCBo8Oe3JvDCLz4kOd5bXmSpvGesPAqBJTjafMrtf3yF3furz3yalsY/547g5U+GkZ5icc/NlzJxdD+u1g+zdPU7UWc8oxnVbz83T11JlzYFFJU6eO/rfry7aAChCK1zTvB4A7zwxrfkFZSiCYjU4i4YNOkbZU2qEAKXM3Iht7MHdOLjl+9m6eo95Bd66NYlg3492lZaZzR8UBfefv42DmUXEDItOrZNQdc1zh/Xm1ffX8Z3K3aiCcGksb2ZfcUIUpPjIl4LwDQtnv33omoztlJKvN4Ab360gp/dcl7U1ytKNE7DiFnA7FR+8+0itFawvk5pPQKmycj2HThQVIgZ4cORXFYp+0BhYdSHPRLYmZdXbbsSnQpYFaWByOBmZMkLEFwb7t+KDRWwKs3VvB96YprhG8+CEhc3/OYqzh26m/GD9mBaGk57iNH9DwBB8C9Cmsd59GcX8eNfvUEgaBIMhYNHQzcJmeFeq6alcTRXY+GSXUwc3Y9+Pdty/rjefPnd1phBq8NuIKVFIGgx45yN3DFtOS5HOFhzOULcMnUVo/oe5L5np8RcD7v/UPQbBqfDYOp5A0hKcNX+N4twIagJI2M3chcivPa1ojaZSTxw54U8cOeFNb7W3oO5BKI0zg2ZFl8v3aECVqVOhrRpF17DXcfl66aUEW/qFaW50oB5O7dH/Xu9r7CQVUcO0y4hMeaa7Y5JSQ00wpZJBayK0gCkfzEy/6dAgJPL8x1AHFC3CnGK0pTyilz0PesYk0duI94VYPnmjny5sjufL++JrlnceumKsoAVEA4w99M16UVee2Qt7y3uzvLNHXE7A1w8ahuJcX4eenFy+blPTMAIIXjwrgsZNeQs5sxdwbbdOZimhVXlxkBKSSBoEecMcOfly3BUKeDktJsM6pnHbdODvLswlbyCmn/mXE4bUkqunDKU268ZV7ffrEZ2qgqRp9tqR2m9DE3jT+dP5p75n+APhVChp9LaWUBxIBB1vzcU5PNdO3h4/ET6ZWaxLvtIteDWZRjcPqTG9YYUVMCqKHUipYx6kyhlCFnwAOCrssdPnR9TK0oTm33helISvNhtIXQNRvY9wC2XrOSuP11OsddOvy45Jw+WAaSZB4GlpCd5+dG0Ffxo2ory3V6/Qe9OOWzdn4nLaWPS2N7l+4QIFxiaOLongWCIuQvW88+3llDqOVmILFBWgXdE3wOETA1HhMwFXfi47uIjSPt5/POtJTV6jw67wSM/ncKIwV1wRmiDdabq0iENt9OON8J6XJuhc/643hFepSg1M6lrN96aMYvnli9l0d7d1R4gKYpS2Yniec9dfAlXvTOHAp+X0mAQQ9PQhcatQ4YxvnOXph1kM6NaYylKLUjfV1jHpyKP9sbKHohV+DDSqpJWGFxN9CJLqq+q0jy1TSvG5QgHqwBuZ4iUBC+P3rKQtmnFDO5xok+oBrY+EFwFsupDmzCHLcSYAfuw23Q6tU9l7PDuEY+z2wwuGN+HYJR0V0O3Ylc9ln6+/G5rDd8hGLpGRlp8swpWITyDev+PLsBhr/wMWtcECfEOZl2qnuQrp2dQVhv+eenlDMrMauqhKBEkejyM3bad8Vu20TY/cjEgpXE4DIMLuoa/09rEJ/DVDbfwx/Mnc/3Awdx19gg+vfZ67hs9tolH2fyoGVZFqSHL8x4UPc7JmVMfeD9A+pdA+scIrawKp6VSfpWWJ1JgaOiSPp1zeOZ/Pi7brwEJYB6B4EtRzyUlGIbBFZMHc9vVYyv1Oa1q74FcbDa9fFa1orU72mJEqlQMgJvs4nM5mH005vuqyLQsunRIq/HxZ5LxI7rzxEPTeeH1b9i26yg2m865o3ty5/UTSElq2D7TSsu3IzeXNzesY2tu7dpHQbjNR6RKqcrpSy8q5v4PP+eiDetZ7+pNUNoZ7Hmb5e168uTlF7CrW2pTD7HVyYqLZ0T7k9V/bbrOlB49mdLj1K3JlOhUwKooNSBlAIp/T/U03xBYuUjPHET87eFNtoEgVeqv0jrYDEmSEV7PE06CKgSrkFgTn5ru4vprHkLY+p3y/EmJLkwz8s3usYJ4VmwdyNj+W6j82TRAS2Z3zhDstgUEIwS7VTkdBldNPTtq5d/m4OwBnfjnH69r6mEoLcyLq5bzzLIfCIRCdcoRUinEDSO9qIi3//J35romM7j//5GX7EbYQrh8QWbvnc+bz/+dG2/4MVsHqT7MjUUDZvbtf8q6AkrtqZRgRamJ4HqIWm7CD7655T8JPQ1c0wBnY4xMUc4YAgskMYNVcCIcE04ZrErpQ5o5dG6fSJuMpIgzvE6HDZ/jEYi7GUQcCBdgB8cERNq7pKUkY0XqV1NG0wRxbjsOu9GsiiwpSmPZlHOUZ5b9gK+OwSqgqgQ3kMfmzOXtuGk80uOnFGTa0ewhhACfy8ZLfS7l5z0f5K9vvopV3HwfwjU3Nl2nY1ISwbLqwP5QiA+2bOb6D97h6vf+y3/WraE0RsEmJTo1w6ooNXKqL9zK+0XiY0jhAs8bRF/PqigtT6wHy4UlDp5672qWrHVjyac4e0An7pw9ge5dMsqPkVYhsuhx8H0BCBAGj915PT/5vYNAMFSeGuxy2hgxuAuTxvZD0/oj438M1jEQSQgtHoDe3STJiW68vsJqY3E6bPz4hnPo1iWD7p0zcLvUTZ2iVPX6hnXlN9/KmSOroJDRO3dy64C/o7kiB0AfdxzHz/e+wsglx1kxObGRR9g6BS2LXy5cwK8Wfcl9o8by9uYN7C0owBMMZ91tOJrNP1et4IOrZ5Phjt5bW6lOzbAqSk3YBhI9aHWAc0qlLUIYaIkPg2tmDS9Q8S5f3TgrLZPDFqJb1hr8gRDBoMmyNXu486E32bozGwin3svcWeD7nHBLKD/IUrqmvMKbvz/A9dNHMbB3e8YO68Zj917Cb35+WXnLFiHsCL19ebAa3ib4/YPTiHM7ygsSCREOVqec24/LLxrEwN7tVbCqKFEcLCpUM6RnoNE7drIocTSexBi38ULwYZtzGbdhd+MNrBWy63r5/1tSUhoMUOT385tvFrE9N7c8WAXwhkLklJbw2OKFTTHUZk3NsCpKDQjhQMb/HIqfoPJaOR20RIT72igvrOkTNAEiEfRO4JwGJb89zRErypnH6TCZed5G3lwwhFKfHSnB5w/y1EsLefH/ZoNvAZjZVG//5CPZ9hk3Tb+Hm2eOrtU1e3TJ5L9/u5WPF6xn9aYDpCbHcdkFAxnYu71aZ6QopzAgM4sVhw8RULOsZxR7KESpjEPYYmdwee12bKXqz66huHUdK8oiGFPKcIXBKkJS8uXu3fhCQZxG86pI35TUDKui1JAWdx0k/g70DoAO2MBxISLtfYQWpahBsKYtNSyQheCYAO5rQcSf+iWK0gwFQxoDumVX2rZ111E83gDS9wXgifJKAYGa9VOtKjnRzfUzRvHUo1fxyE+nMKhPBxWsKkoNzB44GF19Vs44uzMzGeZZjwzoMY8bmr+VnQkdYh6j1J3QNHxmXZZ9SUoDqjhnbaiAVVEoS0X0vIuVezXW8cuxip+t3l8V0NyXItIXIrJWI7LWoaU8g9Aj96WTVjEEl9VuIKUvQu5UEAmcqnSNojRLIhy0VtmEaVkQ2nGKF6un0YrSmNonJPLCJdOIt9uJs9lx6AY2Td06NrWVXbug2QNMOLI+6jFtfce44Pj3zB0zsBFH1rpUTPetjaBl8eKqFYQqtHsqDQRUJkMMKiVYafWk9CFzZ4O5E6Q3vDG0C+l5FdLeRRidKx0fnplxnfrEVi4IA2RtKsKFwNxTi+MVpXkRwPpdbSttczk1Dm66kJ4ds2MUbQqAY2IDj05RlKrGd+rCitvu4qu9uznu8dAnPQMBPLdiKRtyjuIJBvGFVHHBRiUET1w6hRf/+zAz4v7K1oxOlXan+/N5a+WD/LX9Dfh7FxJ7HlapC8Gpy3HG8saGteR7PVzUvQe//+5rDhQWAoJxnTrz2Dnn0TlZtSOqSMgzaDH9sGHD5MqVK5t6GEorY5W8ACXPU73Hqga2IWhpb9XpvNIqReaMAvynO0RFaRF8AZ2/zBnH/GW9AEhw+7h07FZmTNxIWqKH2BM3DrQ2GxplnIqi1Nw5r/yLA0XVK3ErDW/qV9v49SdzWJQ0mnltxhKyCcYdX8f07AX8o+0s/nbDCGxZ6s+mvp1usHqCoWnoQuCvMLOqCUGC3c68a2+kbUJCPVzlzCSEWCWlHFbT49UMq6J45lA9WAWwILgBaeUhtNRan1ZocUjXpeD9GBW0Kgrk5MezLzsFXTPpkFnI8z+fi8MWwm6rQYdHLaXhB6goSq11TUlRAWsT+fS8Xiwa9CiXfbmNi3d8h2bC5oSunDfrSfL7+7HFqT+XhlBfU30hy6rW+DBcaTjIC6uW8/jESfV0peZPBayKIouj7xMGWEVQh4AVQCQ+ggwdhOBaIgfFitJSCXBeGv7V9wlg0imrkBcf+JBSrw2nPYSmyZh9W0+yg+vKhh2uoii1tjs/j5151es9AOhCIKWkBo+jlNPgSRPMmdWbOfSusFWlATdnIcvii107VMBagVo5ryi2ATF2aqC3q/OphXChpb2KSHsdEX8/aHU/l6I0LxJ8H0NgNVS5dYpzBWsRrGqgd0DE3dIQg1QUpY5KAgFmvjuHw8VF1fZpQtArLZ1rBwzGaRiqhKDSKrltNlxG3QqVqUr2lakZVqXVE/H3IPNWU30G1AVxtyGEvR4ukoSUXrD1B/8xqveZVJSWSIJ1gHB1XxcIHaQJmCBCcMq5FwGu6xAJ9yK0mvY0VhSlMXy0dTO+YChieqQhBL+fdCEDs9qQ6LDz/MrljT4+RWkKDk3jou49aZeQSLuEBN5Yv5Yd+ZGzEKKxaRqX9uh96gOr2HL8GK+vX8v+wgL6ZWRy/cAhtE9MrPV5zkQqYFVaPWE/G5n0BBQ9TPgGWoAMgns2Iu7O0z6/VfI8lPwdMIEQKrFBaQmkBK/fhsMeQtdOtaInCEY/RPztYBUg9a6I/JuItrbbkgBOtPhb0RJ+Vr8DVxSlXiw5uB9PKPLDV03TWHc0m0SHg3+sVsU0ldYhzmbjkQnnMrNfOHPvvi/msaewAKuWBW5dNhu3nV3jekQA/Hvtav70/bcETRNTSpYfOsRr69fytymXMbHLWbU615lIBayKAmiuyUjnJAiuAekH2yCEVrunUlJKCG0DqxBsPRBaKjKwHEpepPKNuVrRozRfUsKxAjf/+GgEC1d1xdDhopHbuevyZcS5YmQOhNaCYxJCaAjAcs0A7/tUzWyQEgQC4RiCiD/9B0aKojSMNJcbTYiIN+O6ECQ5HMzZuL5Sr0lFaYkEkBUfT5rLzdPLvuerPbu5ecjZfLZje617qxqaxmMTziPDXfOsoj0F+Tyx5Fv85skSTkHLJGjBTz77mOW33YXb1rz7mKuAVVHKCGED+4g6vVYGNyIL/gesY4AB0h+uEGzlAd56HaeiNCUhIDPFw8+v+Y4+XXJ45p1xfLa0F5v2ZPHPB9/H0KM/SZbSixDhL2GR+DBSloJvHpJg+Rq38LIdCYHVyMKHIelPENoe7pFs64UQNeiBrChKg5vZbwDvbdkUsQerJSWTzurG/J07mmBkitJ4tLKHMwVeH9klJQBkl+zk63176lRNWBcak7p2q9Vr3tm0AVNGfjAkgAW7dzKtV586jObMoXITlSYlzcNYxU9h5d+NVfxnZOhAUw+p1qSZjcy7Hsz94ZtqWQwEwPspBFQqVJOyJCwuRdx6GDFuL2LiPsTDObBNtRk6XS5HiKljttGz43GCIZ3DxxP4bl2XGK/QwPtu+U9C2NCS/wTOCxERS7L4w8HssQnIvFnI/FuQOaOwSp7jTOofriit1YDMLK7pPxCXcXLmRhMCl2Hwh0kXEWe30zcjo04FZxSlOTA0jct798ETDOIzKz+48ZtmzNlVXQj0KoWVXIbBnWcPJ8HhYHd+Hv+7eCGz3p3Dg19+zuZjOVHPdaSkOGomQ8C0yPV4avGuzkxqhlVpMtL3JbLgPsIpsgHw25ClryKT/g/NNbWph1dj0vN6eM1rNT6QAeqvxbRSKx4L8aMjmPslG8YM4zPXZHzFNsas+Y6J7y7AvCUV14Puph5ls2bTTS4auZ3tB9Lx+u0sWtOViUP3RDnaCqcAx91YeXNwK9E/HyGwjlbeVPJPJDoi/q7THL2iKFXllJYwd9tWjnlKGZTVhvO7dseu60gp+WbfXt7ZvJEiv59zzzqLGX3686vxEzmn81m8vHYVR4qL6ZuRwR1Dh9MnIxOAWf0H8vSyH5r4XSlKw9CA/YWF+KMEplpZUBqsEky6DBsPjB3HkgP7+XrvHoQQJNgd3DNiFNcPHMy8Hdu4f8F8QqZJSEpWHznMx9u38sCY8dw0eGi16wzKassXu3bijZDtYNM1eqdn1Mv7bUoNHrAKISYDzxDua/AvKeUfGvqayplPWoVlwWrF9WvB8H+F/w9pH4XQ05podLXk/x4IRNnpIFxoSVUFbmzi/qMUk8CDw5/A73ORnpWPs7Ofr3xT+aLTFB54+XGK3Day7lF/NnWl65DgOjlbrYlTPJiJ9GBH1PahgRdK/4GMu7V+KngrigLAnI3refzrr4Dw7FCczUa8fRFvTp/JH5d8y3f795UXWVp15BB/W7GM92dey4TOXZjQuUvEc9o1XbW0UVqs3umZbD9+POp+KSXd0tLYW1BAyLLKZ1RnDxjEDQOHcOOgofhCQUoDQVJcLjQhKPL7uH/B/Eqp9qaUmKEQf1zyDZPO6kbHpKRK15nepx9PLf2e8P3mSboQZMXFM7pDx/p7002kQQNWIYQO/A24ADgIrBBCzJVSbm7I6yrNgO8ziPE1Jr1zEfE3N954ToeWFGOnANdV4H2Pk1WClQa3O4D8xscvL3oWh9skMzG7fJfL5YfuMEfezDXPvkTetR1ITaveR1A5NY/PYOW29gC4HAEmDdsV42g7OC+stEWaRyC0s24XD+0BW6+6vVZRlEq2HD/Gr79ZVGmmqDQYxBsKcc17b1Pk9+GrsM8bCuE3TX42/1M+mDU76nkDpomhaZi1LDyjKM3B+V27lgWKkUngrxddgt8M8d2Bfdh1gwu7dqd9YiL5Xi/Pr1zGh1s34zdNRrTrwL2jxrDxWA5alPtjS8J7WzbxP6PGVNqe6HDw1oyZ3DL3fUoDQSQSJHRMSuLf06a3iJ6uDT3DOgLYKaXcDSCEmANMA1TA2spJ8yjRixH5wTrSmMM5LcJ9NTKwBoi0RsAD3v8CBhj9w5VSlQYn3i5i54i+eK040isEqxXld8/C/42T/W+2Zcw9KmCtrZApKPHaWby6K3ZbiK7t8hjdf3+UozUQboT7+kpbZck/iJ6dEIMMgVB9WRWlvvxn7WqCEYJKS0qOeUojJu1bUrL1+DEOFRfRPiFyVf10t5tkp4ujpSX1PGJFaVoacMuQYTy7fBlBK/IDGUE4aHQYBv0ys8q3F/p8XDrnNY6VlpanCy/au5sfDu7nsl69K1X7rShomRzzRP4s9c3I5PtbfsTSgwfILimmW2oaAzOzWkSwCg1fdKk9ULGKzsGybeWEEHcIIVYKIVYeO3asgYejnCmE0SP6DadwI2y1b5jcZBwXgGM0RK1eagJ+Faw2piMhVpScTXpqfvRjhCAvPYPdP7RrvHG1AFKCaQp2HUrl7j9Pw2EPcdW5G3j6p59G6MeqAzawj0SkvVs9zd+/gPDnIxIDiJLya3RGGB1O630oinLSrvw8zCjFzGIl+tt1nTxv9YfPeV4Ph4qLkEDHxNgt4uKM5t1uQ2l9NODladNx22xc0LUbWpSgcHj7DjiM6nOD/1q9kuOlnkprWyXhzIWv9uzBoesRz+e22RiU1Tb6uIRgTMdOTO/Tj0FZbVpMsAoNP8Ma6Xeq0r99Usp/AP8AGDZsmKpM01o4z4fi34D0UP3r0A7OKU0xqnJSSggsRfrmASbCcQE4JhDOcq9MCA2SnwPfZ8jS18A8CDKP6DfiSoNL0LBtDeB0xq4GnOgvJC+U0kiDahmEAF2X9OqUyzu/fYuY34f2MWAfDrYBSN98ZGAJiCSEeybYxxL7makBelswj3ByrbsBwoFI+mO9vR9FUaB7ahprs49EDFqj9VqFcMpv56Tk8p+3HD/GQwu/YMuxY2iaIM5mJ98bu0Kp226jNKRqCShnPpdhcGnP3tw3eiyZcfEAPDh2At8d2EdJIFD+OTlRLfuxc86LeJ4Ptm0mEGVWtjjgJ8XpxF9aWunzKACHHr5+a9TQAetBoOJK3w7A4Qa+ptIMCGGH1NeReTeH28BIE4Qenl1NeRkhnE02NimDyPwfQWA1J9J8pW8e6GdB6usILdLMcAAZ3ADmtrIgXGlKcko8495bxArveFzuyCmnyYW5ZOUc5tD5XRp3cC3IKR/eBpZD4AfCa7dPVsuWgW/APhEcF4P3FSLO4QgbpH0A3nfA8zbgA/s4RNxtCKP5F5BQlDPJTYOH8tG2LZhVqoxqQtAuIYFjpaXVKqE6DYNpvfqQ6HAAsL+wgP1TWYAAACAASURBVJnvzKE0WPZvrkXEHq1VRZvZVZQzSfuERL656bZqs5Ydk5L49JobeHrZEr7YtRNLSiZ2OYt7R42la0pqxHMFzcgtaCD8mXvi/Mn87tvF7CssRBMCiSTN5ealy67AZWudGQkNHbCuAHoIIc4CDgFXA9c28DWVZkIY3SBjcfiG1twHegewj404i9mYZOm/yvqnVqhgLD0Q2oEs/gMk/hp8HyNLXgDrEIh0QIKVQ53W4yn1b7QLV2I+A5ctZ8e5g6vt1iyTixZ/wNdnTWL8xNVNMMDWouIMd4WbUukB/yJwXUnUhEPpRRBAxN1YvRWOoih1JqVk6cEDrDuaTbLTyeTuPeiVls7j55zHo4sXAuEqwW6bDadu8MCYCfhDQR5dvBBd07CkxLQsJnY+i+m9+/Ln77/DkpIdubl4g7X/Dhzaph1LDuzHq2ZZlTNYrtfD4eJi2peluOeUlvDWhvV8f3A/IdNiYFYb5syYVd7SKZZzOnfhg62bIz6ssWs6Izt0ZN7sG9mYc5Q9Bfm0S0hgaJt2LSrFt7ZEQzdgF0JMAZ4mvJjpZSnl76IdO2zYMLly5coGHY+inIqVMwasaGXKneCaBd63iV40SjkTFKxzIGccYVPnwawZNYa8lAyQks4Hd3LO0s8JWHaem3A/j//qOVJTCpt6uK2TSAEZbZ2xG5H0OMI1rVGHpCgt2XGPh+vef5uDxUUEQiFsuo4lJb8593yu7NufY55SPt62lfk7t7PuaHbZWjqBrgkeO+c8XDYbnmCQQVlt+f13i/n+wAF8ZYFmXe4mdSHolZrO1tzjWKpfuXIGcxoGn8++iY5JSSw7eIBb536AzwxVSpfXhWDSWd149uJLsEVZhwqwr6CAS956ldJg5Yc0TsPgkQnnck3/gQ32Ps4UQohVUsphNT2+wfuwSinnAfMa+jqKUm+sGIV6MMH7Jqqv6pkveZCf3W/1xftLFze/8QzS0DDMICWuRBacdTFL+pzD/9z9mgpWm5L0xdoZuW+roih1dve8uewuyCdUVuwlVJay++jihfRJz6BfZhZBy2TTsRyCllWpKMwvv1rAfy6fwQVdu3P/F5+xaO+eqGtba0JDgITNuargpnLmS3G66JCYSMA0ueOTD8t7EldkSsnifXt4bsVS7h01Nuq5Oicn8/aVV/PQVwvK13vH2+38YvQ4ruo3oCHfRrPV4AGrojQ7ejswD0TZqQHR1x7EdnINX805wegLIZW2Whddh2eT/I6Xr3+4ipVf9qHAk4jI0Dlv/FJ+M/xZFaw2KQ1sPcJ9WCOu+7bAPrLRR6UoLdWegnw25hwtD1YrCpgm/1qzij9dMJm/r1yON8LaU18oxF++X0JmfDwfb99a53HYdZ1r+w/kP+vW1PnbVFEa25V9+iGE4D9rV1MciJ76HjBN/rN2DT8bOSZq9WCAPhmZfDBrNrkeDz4zRNv4hJjHt3YqYFWUquLuhKLfUj3l1wm2IRBcXouT2cKFZaxs0LuC/xOQtehHJ+IhtL4W11OqSk0pZOqUb5g65ZumHopSiQHx90PRQ2AGCBdmOsEJzgtUcSVFqUd7C/KxaRqR8hosKdmWe5wjxcUxC8KsPZqNOHp64xjSpi3HPR6VAKw0Gw5dp0daGgU+L3/+4btTHu8JBfEEg8Tbo7RmqyDN7a6PIbZ4Dd2HVVGaHeG6EtxXAw7AWfarA5wXQMJ9QE0rtLnAdSVayp/R0l5HS/41IuXfMfq1RiCPU/lGXlGaO+3kr/l3gH0U2M8B7OHPhnCD+zrVukZR6lm7hMSIs6sQzv/plJREvN0e9RgAS1r4zNjfSXZdxyai315O7t6T1dmqYYTSfEgpGZjZhrc3bYz5+TjBoeu4W2k134aiZlgVpQohBCLxl8i4m8KVTGUo3IPV6AKAZT+7rIpwxQqoNhAJ4W3SA1o6xN2BcN9Q+dz2Qci0eXD8fFSfVqX1qXgTWzbP4/0EnJMQmUtBFoKWEW57pShKveqVlk6npGR25OVWW3vqNAxuGXw2KS4XQ9u2ZcXhQ9WOceg6LpuNAl/0ted2TefuYSMwLYvnViyLOIs6Z+N6bFrTdgNQlNoIWBZXvfsWJYHAKTMDbELj2gGDVHpvPVMzrIoCSOlHBtYhg1uQMvz0TOhtEe5rEXE3lAerACLlBXDNAJxls6UOcE5BZHyJyFyNyNqIlrkELe7GiCXIw5vUR09pbQTh7ISqT6d94FsAVgFCb6+CVUVpQC9eMo00l5u4stkfQ9Nw6AZ3DRvBiPYdAPjDpItIdDjKKgSHuQyDrimpjO/UBT3GjfiYjp24e/goPti2JeqN/d6CAkZ3UOn+SvNyzOOJuLa7qsFt23JfjIJLSt2oGValVZNSIj0vQ8lzhG+orfC60aTfIRznRHyNEA5E0mPIxF+G29+IFIRWcQ1C9DQQaRUgj89AVRlWWhWRBHrn6OuxhQHBlWB0aNxxKUor0ykpmW9uuo3Pdm5n2aGDpLnczOjbj7OSU8qP6ZyczILrbubV9WtYsGsnDsNgZt/+TO/Tj72FBSzYvROzyo27LjTGduzIy9OmA+ANRr+xNzTB1J69eH/zJgJSlV1SmoYuBJaUGJqGaVn1UgCsa3IKc2bMqjRZcdzjocjvo31CIg5DhV11pX7nlFZNet6Akr+C9FbciMy/B1JfQ9gHRX2tEA7Q29f8WjKIzL0KZN7pDFlRmolUSPx/YB+CZnTGyv9pjAJionZruxVFqTOHYXB5775c3rtv1GPS3G7uHTW2WmuOXmnpPHvxJdz3+WdIJFJKQpbFhM5deGby1PLjRrbvwPxdOyK2vRFCMLxdBwa1acuKI4fq740pSg2NaNee1664CkPTEEKw+shhbvjwXYKmWamVU23YdZ0nL7y4PFg9UFjILxbMZ+3RI9g0DQncNGgo944ag66pLLvaUgGr0mpJaULJs5WD1XI+ZMkziNSX6++C/oVg5tTf+RTljFYI/q/Q3JcDINwzkIFvIrewkRbYxzfy+BRFqYtJZ3Vjxe138f2B/RQH/AzKakOnpORKx/xkxCgW7d1dLYXSZRj8ZPgo7LrOwWLVVkxpGpuP5ZQHqwBD27bjP5fP4JcLv2BXXl6tK1iPaNeBRyZMpF9mFgCFPh9XvP0GBT4flpQEzHDNkn+vXUWR38evzz2/Pt9Oq6BCfKX1Mo+AjF48guDaer2c9C2keqscRWmpTPAvRMqyfnX28WV9VavOpDoh8ZEqafWKopzJ7LrOxC5ncWnP3tWCVYDe6Rm8fNl0OiQk4jIM4u124mw2fjJ8FLcPHQaEUyUVpSmUBoMsO3Sw/OeDRYXcNvdD9uTn1zpYndqjJ3OunFUerEK4sJgnEKyWYeANhXh704ZKf/cPFRXx12U/8NDCL/jvpg14gmrJWCRqhlVpvTQ3MSv1Cmf9Xk+4CK+TVd3nlFZEekDYEUKD5OeRnnfA8yrIfDB6IeLvRthHNPUoFUWpZyM7dOTrm25jV34e3lCInqlpldbwxdntMSsOK0pDkcCb69cxqqz41xNLvqXY78eqw/1Z1bR5gAV7dkVt/xS0LL7dv4crevfjjQ3r+O03i7CkJGhZzN2+lT9+9w1vXTmLXmnptR5LS6ZmWJVWS2ipYOtLOIisyl5WCThM+r7Cyr0GK2ccVu41SN9XNb6ODO4Ir9/zfkLsYNVNzXu8KkozoCWCSCz/UQgdLe5qtIx5aJk/oKW+Ui/BqpRBZOgg0io67XMpilJ/hBB0T01jQGZWtYIzM/r0w66r9jZK0/h05zZ+OLAfgAW7d9UpWHUZBrvyqtclccUoriSB97dsZnvucX737WL8FdbNeoJBCvw+bvno/Yjrv1szFbAqrZpI+j2IOConGzhAb4eIuwMAq+hJZOG9EFwFVg4EVyEL78UqfuqU55eBNcjcK8H/BVAS5Sg7xN2FyFoDtuGn+5YU5QzhhLgfh2dWG4iUFlbJ88ickcjcqcicMVh5NyPNIw12TUVR6sc9I0bRLiEBp6qcqjQBCdw89302H8vBrGO1am8oxM8+/5SFu3dV2n5l3/4xX7fi8CH+s24NQTNyll+R38eKCinLigpYlVZOGN0R6Z+C+1rQO4LeDeJ/ikh7H6ElIEP7wfNK9cJM0gulL4f3xyALf0l43WqkfwxtYPRBJD+JlnAv0vNfCK6pp3emKE3MdSXCPbv8Rykl0vsR1vFLsY4Oxzo+Hen7/LQuIYv/ACUvgiwp+4wGILAUmTtDzbYqyhku0eHk46uv575RY8v7wipKYwqYJs8s+57h7Wre8aEqXyjEb79dXF4x++ml3/Pooi9jvkZKya78PMwYs6iHitV3WEUqYFVaPaG3RUv8FVrGQrSMz9Dib0do8QBI3zyir3M1wfdZ1PNK8xCYMUr2a2lo6R8hnBchpYTS51BFmZSWwY4wOlfqRSeLHkEWPgqhbSALIbQRWfAAVvFf6nQFaeWB5y2qf2ZMsErCa2UVRWkSyw4eYMbbb9Lj2b/Q7/lneGDBfI6VllY7Ls5u57ahw3AaKmBVmsbyQwd5YOyEmGm8p3K4uJh8n5dfLJjPP1avoDgQiHl8vN1Ov/QMbFHa20io1BtZUQGrosQmS4FoDdBDSBktzReQfhAx1ufICv+gSQ9YuXUZoaKcgUykuS/8IAaQwa3gnUv14NILpf9Gmodrf4nAahDRbnJ94F9Q+3MqinLaFu7Zxc1z32dN9hFMKfGGQny4dTOXvPUauVEqAzvUWlaliehCY2BmFv+eNqPOhY4kkkPFxczfuR1fKNo9Y5jLMLhz2AhuHDw0Yj9WTQjaxicwuE3bOo2lpVIBq6LEIOwjQERptyHiEPZR0V+sdyJ6ESUBjjEVfnQA6gtbaSlM8LyLzL0CaRUgfZ8A0Z44S6hLanDUYPXEfkftz6koymmxpORXXy2odtMekpJCn4+X166qtH3zsRyeWfY9beLjG3OYilKu0O/j4a8WMLxdez6bfSPX9h+IISIV44yuf0Yma47EfvCqC4Fd15nZbwC3DRlGp6RknrpwCk7DwG3Y0IQgzmajbXwCr0ybUSlDSVFtbRQlNvtY0DtAaA9QsTeWLbzdPjrqS4UwkPH3QfEfqD6zJMG/FOv4LJDHQSSDrR8E1xOz1Y6iNBs+CO1AFtwLRjcir+MGCMXuhxyNfWSMc7oRFap8K4rSOHbn50VNhwxYJh9t28IvxozHtCzu/XweX+7ZRcA0y7MxqhLARd168O3+vZSq/pRKAzCl5J3NG+mXkcnsgYP56cjRfLpjO8UBf6VKvU7dQIjwutcTa081IXAaBo+fez7rj2bHDDKHt+vAM5OnkhEXV77tou49WNrxTubv3E6u10Pv9AwmdOoScea1tVO/I4oSgxAaIvV1cIwD7CDiw786xiNSXz9lBVQt7hpIfAhIplr7HHkcQmvAPAChDRDcgurRqrQsQQisBKN39EwFHDEf/EQjhBMSHgGq9kt2gq07OC+u9TkVRTk9ppQRG8WdcCIAeHntahbu2YUvFMKSsvybT1T4NdHu4P8mXchTF00hIy6u1rNeilJTppT879dfsTb7CJlx8Xw4azbjOnXG0DRsmka7+AQu7NadBHs4c0cTAoeuc2HX7nwwczYDMrM4t8tZUR+8uG02fjxiZKVg9YREh4OZ/QZw17CRnNulqwpWo1C/K4pyCkJLRkt5EZHxNSL1VUTG12gpLyC0pBq9XnPPgqTHgFOlKPqIPmOkKM2UsIPWBvT2VE+Rt4d7IdsG1enUmnsGIuXvYBsabk+lZUH8nWUPk1QRF0VpbN1TUrHrkZP3DE3jom49APjX6pV4I6z1kxV+DZgmT/6whONeD+9edQ0j2ndooFErSvhhyr2fz0NKSefkZF6ZNoP1d/6EpbfeSbfUVL7YvZMcTymmlFhSoglBr/R0eqSlAdAuIZHZAwZVK97k1A2GtGnLmA6dmuJttRgqYFWUGhJ6GsLWH6Gn1f7Fvk8JB6SK0srIIMJoh0h9ExznEc5UiAMc4JyCSHnptNbqCMdYtLQ5aFlr0DK/RYu/Ozz7qihKo9M1jV9NmFitt6omBG6bjTuGhnuNH/dUrxhclc8Mkef1cN/n81h3NJv+mW0aZMyKckJOaQm78/PKf3YaNjYeO8qqI4errcv2hkK8sHI5xysUEnt4/ET+95zz6JSYhC4E6S43dw0fwUuXTVdrUk+TWsOqKI1CzZwqrZEGRjeE0RUAkfJsuD+qlQNam/L2UYqitBxX9O6LU9f545JvOVxSDMDoDp14fOJ5tE1IACDdHcexGgStppSsOHyIjfM+wZLqe1RpWLqmUVJlDfbcbVvwRFk/rSF4eukSxnXqwtiOnUgoS++d2W9AYwy3VVEBq6I0AuGcggz8EG5fE5ONcOJDCFV8SWnWhDtcSTv52cqbtUTQEptoUIqiVBWyLJYdOkCB10e/zEy61EP/x4t79GJy954UBwI4dB1HlRnX24YO4+mlSyKmBUfiDamCS0rDMy2LHlVa2wTM6A9KvGaI97ZsYu62rQQti5+PHsttQ4c19DBbJRWwKkpjcF4EpS9AaB/R23sQbsWR/DKU/BGCq1FFmJTmRwOtJziGQGgrMv92pH0YIu5WhNGlqQenKEoFyw4e4O55HxO0TJAQtExGdujIcxdfSrzdflrnFkKQ6Ihcu+GWwUPZcDSbhXt24TfNStVYFaUpuAyDmwcPxW2rXP/ggq7dWLhnV9RZVr9p4jfDEwxPLV1Ch8QkJnfv0eDjbW1EtIpWTWHYsGFy5cqVTT0MRak30ioF/9cgPUijN3jngPcjwjOoBmCCcIEMgd4G4n4Kxb8BmXeKMyvKmU7jZCq8DtgRKf9AOEY24ZgURTnhYFEhF73+n2qzl3ZdZ0yHTrw8bXql7UsPHuDJH75jY85R3DYbM/r05ycjRkUNSmti87Ecvty9k32Fhczbsa38xl9RGsuJIkk3DhrK/WPGoVVZaxowTaa++Sr7CwsIWqdOS++Zmsb8625qiKG2KEKIVVLKGk9HqxlWRWkglud9KHochAbSAixwjIXMZQhkOGVSFkFoB2jJ4aA29xoq93tVlOaq4he7CXiRhfdCxnenbAelKErDe2XtakJW9QAxYJr8cHA/BwoL6ZgUrob/2Y5t/HzB/PLCM37T5NX1a1i4Zxdzr76OuDrOxvbNyKRvRiYAI9t34PGvv0ITIlwlOBSeebVUppFSzzQhsOs6U3v04ur+A+idlhH177Bd13n3qmt4dPFCPt+1AwExH6zsLshvoFG3bipgVZQGIAOroegxwFc5q9e/BIp+jUj+Q/hnkQT28AMm6/jlqGBVadGkF4JrwT60qUeiKK3eqiOHo84Y2XSdLcdz6JiURMiyeHjRl9WqpAZMkyMlxfx30wZuGXL2aY9nZr8BTO3Ri6/37eVgUSFOw+D3331NQM26KvVEFwKnYXBF735c038AfcoelpxKktPJM5On4g0GySkt4cLXX4n62TnRq1WpX+oxt6I0AFnyd8AfYY8ffJ8grcLKx1t54ZlWRWnRRDiroJakNJHBTcjgBqRUD3UUpT5kxMVF3SelJMXlAmDD0WxCUQrP+EIh3t2yqd7GVBoM8PLaVTy97Hue/GEJZ9KyNaX5O9FDtVdaWo2D1YpcNhudk1O4sFt3DK16COXQda7tP7A+hqpUoQJWRWkIoa1ELZgk7GDuL/9RBjcjS9+IfryitBQyAEb/Wr3E8n6CzBmDzLsOmXcDMmc0lufdBhqgorQe1w8YjMuwRdwXZ7dzdtv2AOGZpBgtJIP1NAMasixmvjuHddlH8IVCFAf8BC0r1qUVpda8oRDvb918Wud47JxJtImPL1//CuA2bPRMS+fu4apOQ0NQKcGK0hC0dLCORt4ng6ClIaUXmX8nBNYQXuNXs/L+itI8OcF5MUJPP/WhZaT/Gyh8CPBV3lH0aywRj+aaXL9DVJRWZFynzlzeuw8fbd2Cp6zwkl3XsWkaf59yWXnxmYFZWVGr+Np1nYvrqSLqoj27yfV4MKtcSz3KVepbXWbuDxcX8eKqFXy5exeGpnFJj15kuOP4au8ebJrGFX36MrlbD2y63gAjVlTAqigNQMTdhCx6NLxmrxINbH0Qejusgl9AYDWRU4cVpYUQ8eGZVddUROLjtXqpLH6SasEqhLeV/BlUwKoodSaE4Lfnns/UHr14Y8M6jnlKGdmuA9cNHExWfHz5cU7Dxk9HjOaZZd9X6puqCUGczcYNg+pnTfrqI4cpjdI6JBZdiGpBblWG0JjSoyfJTidzNm4gEKHYlNI6OA2Dy3r1qdVrduXlMv3tt/CFguVrV/+9djVpLjdzr7mOVJe7IYaqVKACVkVpCM5LwbcIAovLglYJuEBzI5KeRFpF4PuMmD1ZFaXZEuC8HFyXIdDA1guhpdbqDFLKstT6KMxDSOlFCNdpjlVRWi8hBGM6dmJMx04xj7t96DDi7XaeXvo9xQE/lpSM7NCR3517Aenu+rlZT3I6sGlaxGI2hqYhCP+7ECoLTjUhuGPoMOZu38rh4uKY57aQPDT+HFyGjTc2rKuX8SpnHqPs4UW0xxeGppHqdHFl38pLU4r8fv61egXvbdmM3wwxukMn7hkxip5p4YygXy36kpKAv3INTdMkx1PKM8t+4PGJkxrmDSnlVMCqKA1ACA2Sn4LAUqT3/XChGfsEhGsaQovHCqw/xRnigZLGGKqi1CM7aEmI1LcQRuwb4FMRQiCxEz0DQaC+whSlcQghuHbAIK7uP5BcjweXzUZ8HVvZRHNprz48s+yHiPt0IXh/5my+2b+XrcePcVZyCjP79adNfALXDRzM+H//M2bqsCYEr65bw61Datz2UWkmBBBvtxMwTUZ16MSsfv15dPFCvMEgpiXxmeGsAJumMaVHLx4eP7HS390iv59pc17nSElxeUXqz3Zu56s9u3j1iivpmZrOqiOHI/79ClkWb2xYx/qj2Vw7YBDTevXBrlKCG4T6tleUWpBmNtLzejiVV2+DcF+LsEf+AhRCgGM0wjG68jlkAAofIfbsqo/wP8Nq9Y7SjGgZiIzPEaKebmRdU8D7MdXXd2vgOBchIheMURSlYWhCxKwufDraJyTyP6PG8NdlP+ALhZCEvwWdhsG9o8bSJyODPhkZ1V4Xb7ejaxqhKG1GIBxYLN63l5+PHkdWXDyHS2LPyCrNgy4EEzp34ZYhZ3NWcgrtEhIBuLBbDzYczcYbCtE3IwND07HresTKvi+tWVkpWAWwpMQbCvHgl5/z1oxZ5eu5I7GkZN3RbLbn5vL2pg28MX2mClobgApYFaWGZGAFMv82kCYQgKBA+hciXdeiJT5Y8/N45oK5LcYRgnARJhWsKs1NsP6CVUDE34/0LwGrgJMPeOwg4hCJD9fbdRRFOTP86OwRDGnTjn+tXsmegny6pqRw29BhDG/XIeprEuwO2sTFc7A4dsusOJsNIQSZKmBtMey6zgNjJ9ArrXIxP00IBrVpW6NzvL9lc9Rev4eLi/EGgqQ4nRwtLY15Hm8oyKZjOfx343quHzSkZm9AqTEVsCpKDUgZQub/pEoRJRn+2fMm0nkBwl7DwhOel4DoT4LLz60oTerEE+Va/F00etfvCPQMSP8YWfo6+D4Oj8U5GeG+CaGn1eu1FEU5M4xo34ER7aMHqFUJIfh/4yZw/4L5+EKRq+27DRtX9xvI/sICthzPqa+hKk0ozeXiLxdNqRas1pbfjN6hQRcCv2nyizHjeWTRl5WKjkXiC4V4Y8M6FbA2ABWwKkpNBJYSPYXXh/S8VfOA1Tp2igNsMa6lKI1Ey4DkZ8PFwTz/JZymHit4FYj4u+p9GEJLQSTcAwn31Pu5FUU5s+V5PXyzby8hy2JMx07lKZ9VTenRC08wyONfL6I0WPn702UY9MvM5JKevfh0x3Z0TYN66h2rNI3+GZl8MGt2+M/yNI3t2IlPtm+LWGla1zS6JCfTIy0NXyjEE99/SyBklq+LjaTIrzo/NAQVsCpKTVh5MXZKMKP0XI1EywAzVuqS+iJVmphIgfSv0TQd7EMg8SGswEbIuw7wRH6NYwrCfnajDlNRlJbrueU/8NyKZeF1hxJC0uKK3n353XkXRFxTeGXf/lzeuy/zd2znv5s2sD33OElOJ9cPHMys/gOx6TrxdlvM9YixOHSDgBlS+U9NzKZpPD5xUr0EqwA/GT6KBbt34anSUsllGNw7akx5X9VrBwziqr792ZCTzez338Ef4aGHJgTDa5EdoNRc/fxpK0pLZ/QuW7saiR1qc6PuvjHGTh3sI8LnVJSmIgsg/wZk4GT7B83eH5H8BOCscrAOWmdE0m8bdYiKorRcn27fxt9XLidgmniCQTyhIAHTZO62LTy/YmnU1xmaxiW9evPa9KtYdvtdfHH9zVw/aEh5EZxxnTpjnaJna1Vuwygr7FS3QFepGQEMbdOO16+4km9vup0eqWnYqjxciLfbeenSKxjStl29XbdbahpvXHEVPVLTcOgGbpuNJIeDB8dO4IaBlVN7bbrO0LbtuXHQUFxG9Tk/h65z9/CR9TY25SQha/nBbUjDhg2TK1eubOphKEpEVu61EFwHVGlsLtyI9C8QemaNziNlEHn8YjAPUDnF0gDnZYjEh5H5t0BwO+CNchZFaQwuROpLlSphy8AKZPHTENwIwgXuGYi4OxFaQhOOU1GUluTC1//NzrzImU2Jdger7ri7zjNsn27fxi++nI8/VLPZ0h8PH8nRkhI+2Lo5YtqoUnuaEDh0A28oiE3T0ITgwbETuGnwUKSU/G3FUl5dv5Z8rxeHbtA3I5MfDx/BhM5nhTswNJBDxUX4gkE6J6dErCh8gmlZ/N93X/PGhnXYdR1LStw2O3+56GLGduzcYONrSYQQq6SUNe4zpQJWRakhaRUi8++G4AYQZSXLhROR/DzCXrsF9tIqRhb/FryfEC7ApIPjHEj6A5qWgJQSguuR3g/B+zbVgmREOFgQZEbI1AAAIABJREFUCSCywDxVX1dFqWObJKM3IvUtCHwL0gO2YafdY1VRFCWWXs89RTBKmxqHrvPtzXeQ7nbX+fzrj2bzwsrlrD+afcqKwW9ccRU/nf8pud4oyyHq2TX9BvDfTRtOWZqxOclwuwFBVnw8twweyuTuPXh38yZWHDpIx6Rkru4/gA6JSRT5fVz0+ivVKvLaNY2JXbry96mXNWjAWltFfh8bc3KIs9kYkNWmzunmrVFtA1a1hlVRakhoSYi0N5ChnRDcCloa2EcgRO37bQktAVzXIH0LylKNveBfAsfOQab8K1zAyT4IYR+EdE5CFj0O5pHwi/X2iMRHEY6xAEgrD5l7C5ib6/HdKi2PRp3WR4d2IHNGlz2kkSBNpGMiIvnJem1hoyiKckKS08lxT+QAUQIJ9tP7t2dgVhuen3oZAOe/+hK7CwoiHqcBQ9u2ozHjkCK/n3iHo8UU72kbH8/iG28rXwta6PNx7+fzWLR3D7ay/rlHS4r59bnnc8fHH0VsHxOwLL7Zv5fV2Yc5u237xn4LUSU6nIzpqB7gNgYVsCpKLQmjOxjdT+sc0vKE035lSYWtnnA8kH8bZHyL0MLN2YVjHKR/UV5d+ETqsQysQRY9CqHdtby6jeoztkrLJgivi65LirkZ/q/i5Kz/a2TRbxFJv66X0SmKolR03YDBvLByebVqrDZNY3K3HjgirB+sqwRH1XX5J2lCwxMMcnH3nry5YV2jpATP27m9Wi5MHfNjmoQuBFJK4mx2rh80hJ+MGFkerJqWxcx357C3IJ+gZZX3P/1kxzZ25eexMSd6AUtfKMTH27bWe8AabnV0jHS3myFt2qlZ0jOUClgVpSn45oOMkvAjLfB9Cu6Z5ZuEEFBhjawMbkbm3UTdAhAVrLY+klP3/q0NH3g/QCY8gNDi6/G8kUkzB+l9F0I7weiKcF2F0LMa/LqKojSNH509nCUH9rHpWE559Va3zUZWXDyPTTyvXq91uDh61X6nzeBwcRE/Hj6SdzZtwGyEdjiRAtMzNVi1axoBy0ITAruuc/3Awfxy3DlRj1+8bw+Hi4uqpXv7TZPNx4+F73ViPBQIWvX3+1/s93PPZ5+w7NBBbLqGlJIEh4MXpk5jYFaberuOUj9UwKooTUCae4jaHgQPMrSHWM/4ZPHTqIJMSu3Uc3qZsIULh2l96ve8VUj/YmT+zwgH3H7Ajiz5BzLpL2iu8xv02oqiNA2HYfDm9Jks3ruHD7dtJmhaTO7ek4u719/sqpSS1dmHsevRzxcwTdrEJ2DX9ahB46lmP08Ec75Q9N6dzZFN03DbbIiQSYfERH4xZjwXdoudffbNvr2UBiM/NDctK+b61P/P3n3Hx1FdDR//3ZnZqmY1d8vGvduAwZgSuoEAgVBCDxBSniSkkva8KYSWQoCEkEJCHkJLgNBD7x3jBsa4V9mWLVlWL6stM3PfP1aWLWt3tbJWzT5fPv5gz87MPSvLks7ce88xleKkQ8Z1K+a9/c/zz7C0fAdRxyHSmgc3x2Jc/tRjvP7FL1EczMrYWKL7JGEVog8oswRNkIRJqwqgrE6qzEU/6GQELxDt5BwhukFHwSjo2SHchtZkde+HM62f1/XfR/veRhn5PRqDEKJvmIbByWPHcfLYzCUpu9W2tPDFpx9nc11t27LUfXkMg2NHjaYwGOT9bVvwmmbC3pudzX7men2cO3kK/16xPOlYuxlKdbntTl+JuS51rftsS+tque6VF3jk/IuYNjj56peg5cFA4Sb4qFmGQZbXS01L4ofxI3JzOWH0IRmJfUNNNR9XlCf8+4g5Dg9/upxvz50HgKs1b2/ZzKMrP6UhEuH4kjFcNH0Gg/yBjMQi0iN9WIXoC/4zSF7FQYH/zKSXuqFn6DwZ7cqTXGlHcvDJIv7lXwFBUNnEPw+S7+VqzwDPjJ5flht+gZRLDVqe7dnxhRAHpG+88F/WVVcRisWwE1QjzvJ4GD0on9+dejoAfsvCTZFHluTmYSX4nj44K4unL76M7x51DCNzc1N+OTOV4oqZswhYnq6+nbSlatWy2+7vDNleLz4zvaKSjtY0x2L8+PWXU5539qTJ+Kzk97zr9LPI8/kw9/lYji8o4NmLr9jvVkb7WlFZ2WGM3SKOw8Lt24D4rO/Xn3+Gb734HK9s3MCHZdu4c9ECTnrgXjbVJm67JHqGzLAK0QeUkQ35f0PXfpV4paUWIABKoQb9NWlPS+02QMPP0hggG3QzaVWF9UyG2OKuhC/6jf1tVTMBcm9AmbmtbZpy0J7ZUHlYGhcHwMhCDbqt6+N2kXbKW/9tJBJGO+UpfwAUQoh9ldbVsqyiImHbHAWMyy/gJ8cez/Gjx7QlSLOHDMNjGglLQAQsi6/POZLZw4bzfx8tYXnlTgb5/Fw+YxZnTZrcdt5/L76CS554lE+TFBbymiYXTZuJ7WieXLOKFjtz9SYUcPnM2by5eRNlSfbsKiDP5+dzkybz9TlzGZKdzXWvvMh/165Ou9jUxpoadjY1MSQ7cW2DqcWD+dzEKfx33Wpa9loiHbAsrjl0DvNGlfD2VV/hydUreX/bVrK9Xi6dPpMjRozs6ltOKd/vRyX57qGgbTnwM2tX897Wre3+LsK2TcS2+c5Lz/PsJVdkNC6RnCSsQvQR5T0Sit+F8HNoexPKGgP+s1FGbvKLwq8BnT3x9EHwSgg/BW5ta+KajCeevMSWktmiPKLneaHgQWi8E2KdLRHfh70M6r6JLnoRwx+fQVCAaxSAW518PO9R4DsRFTinV4otKWs8WmUl/hxWQZQ1ocdjEEIcWDbV1uI1jbZ9i3vTxBPHkw4Z2+64aRjcesppfPul54nYdttjQr9lMa6gkHMnT8VnWdzaOiObSNDj4ZaTTuULjz/SYT+rqRRj8wuYXFTMjSeezOkTJnD7B++xbGdFN99t3KjcPG444WRWVO5MmrBmeb3cftoZnDhmz3v/xpwjeWnDunbJZSqmYRDqJNH+1cmnMnfkSP6+dDHlTU2U5OXxzSPmMn9c/Ot5rs/HVbMP46rZ6TxA3T9HjyrBNBInrH7Lw6UzZgFw3ycfJ3xwoIGNtTVsra+jJG9Qj8Up9pCEVYg+pIwcCF6S/iyRrqfT5b7Kj8q6CrK/DpE30A2/BXdbkpM9EFtF/61BKJKLQvhV8M6F2Id0+YGDWwY1F6MLH9/TSzj4ZWi6Ewjvc7Ifcr6HkXV1BuLuAv98aLgZSPTQxQOBM3o3HiHEgDc0Oxs7xfreEbl5CY+fMnY8/z7vC/xx4QI+rignx+flkukzuXr2YWkXgpo+eAg/O+4EbnrnTSC+/DTL4yHfH+BvZ50DxLsCHDNqNLG5Lt968dmkRYr2pVp/7fudwG9Z/PDo4wA4a+IkVlftSlgAynZd5uzTMmZcQSH3n3sB173yIlWh5nhCGosl3WfrNUxG7fPxK2uo558ff8SiHWUUBAJcNmMW50yawrmTp6b1vnqCxzS564yz+dpzT2O7bttse8DycOHUaRwxPP5xqE7SCxjie5xrWlokYe0lkrAKMZB4ZoEyk+eXqghV+MCeJcX++YCBrruOxFWFHbAlYR2wQg+Akc9+z47bq9Ghh1FZlwOgsq5GOxs77g0NnI0KXtm9WPeDUj4oeABdexXoMOhYvDoxPlTBvSglRS+EONit3lXJmqoqirKCHD2ypNN9jlOKihmek8Om2poO3/kClsXVKWb2Zg8dxr3nnNeteC+dMYtTx47nufVrqW0JMXPIUE4cM7ZD3PNGjkrrfl7TZJDPz4Ofv4A1VVVc//brxBwXBRiG4n+POZ4zJ04C4IIp0/nHR0vZ1dyEvVfSGbAsvnb4keT4fB3uP2f4CN668ho21NSwK9TM7Qve5eOKjjO/AcviO0fNa7dPdsmO7Vz1zBPEHKctKVxavoMTx4zlj6efmbIqcE87tmQ0L152Jf9c9hEfV5QzJCubK2bO5phRJW1xzRwyhIqmxoQ/IUUdh7H5UvSvtyjdj6qRzZkzRy9ZsqSvwxCi39Jao6vPB3st7TfTKFB5UPwWhhHc5xoHXfsliH5M+5kzH5gjwdmEJKwDmYdu9dY1SzCKX2t3SNtbIfJW/A++E1BWyf7fPwO0tiHyDjhb45+zvuNRqucKkwgh+r/qUIgvP/sU66qrUEqhUPgsk7+fdS6HDRue8tpNtTXxpbkxm5Adw1QKj2lyzaGHc928Y3vpHXTumbWr+d/XXiacoJrtyJxchuXkcG7rbGXAE/+a6Lguq6p24WrN1KJiPPsUTtoVaubGt9/g1Y0b0WgG+QN8Z+48Lpk+s9ME8vInH2NJ+faE1XW/e+Q8vjV3Xts9XK05+t6/UdnccYVM0OPhD6d9llPGpm6D09eW76zg4ice7TAj7bcszpk0hV+fPL+PIhv4lFJLtdZz0j5fElYhBhbt1qHrvgfRJaC88VknazRq0J+TJhZax9DND0LLg+DWgzE0/sM/MTKbrJpgjgNzNERfR/bFpsNPvOpzH32sVDbGkI/6ZmwhhNhP5z7yEKurdnUonpTl8fDGldd02kczFIvxzJpVLNxeRlEwiwunTWdSYVFPhrxfFm0v448LF7B8ZwWWYXD0qBJ+/pkTkxY2SlfUcQjbMXK8voSJakVTI3ctWsAL69fjapdDhw3jw7KyhMmqzzT55hFzufbIeW3HlpZv56qnn0i6pPm4kjHcf+753XoPveGF9Wv58Wsvo5RCa43tupwydhy3nXpGxnoCH4y6mrDKR1qIAUYZg1AF/0TbZeCUgjkUZaV+SqmUB5X9Jcj+UnyWdtdJ9EyfVhdQYASI94Lddy+k2MMEvBC8BMLvgbuuj8LITF87IYToLSsqd7K+pjphpV/bddv10Uwm6PFwyYxZXNJaYKe/OnLESB4678KM39drmniTtK3Z0djA2Q8/SGMk0rZ0+N0tW5I+3o63ginj2r2O1YXDGClmbGtaku8P7U8+O2ESJx8yjve3baU5FuWwocMZkZuiOKboEZKwCjFAKWskWPtR6t3ZCG53+4cla6eiwVkLzrokr4t2/KdA8HKIreyjCdYAKvsbfTGwEELst1W7KpO2JYk4Dh9X7OjliA4sty94n4ZIpF07m1Tf0RVQFGy/HWla8WAiCWZjIV6waG6GW9X0JJ9ldagcLXpXZjrwCiEGDt0SL9yUVDqNwnd/60p2riSrnXMg/AJUzYfYJ704rgkEAS9kX4vyn9yLYwshRPcVBbOSzt4ZSjE0O3Evc5GelzeuT7v3KsRbwVwyvf1M9dDsHE4dOw5fgllcj2ly9ezDux2nOHhIwirEwcaaSPLpPBN8p4DKpvMFGH7wHgvpN+XpY/2xoqxDvE1Rby2d9kHwKlTeL1CD38XI/kovjSuEEJlzXMnopH00vabZ1kdT7B87wVLrvVl7PSwIWh7OnzK1rRXM3n536umc0pq05ni9ZHk8DM7K4r5zzpdltaJLZEmwEAcZpXzorK9B0910aHWjfKicH4L6Cbrp7xB+FnRTkjuFwcgiPmOXXlPxNKKj52Znu1FJt9cZdGuNsCoEQvHZ9DZ+8B2NyvlRn7YSEEKI7vKYJnefeQ7XPPsUtusSdRwMpfCaJl877AhmDB7S1yEOaEcMH8H727YmfG1ETg4njhnLsopyBmdlc+WsQzm2ZHTC7yt+y8NdZ5xNRVMjK3dVku8PMHvosJR7W4VIRKoEC3EQ0lqjm++B5rsBDdoGswQ16Dcoz4w954WeQDfcSOIergYEr4Tw8+BWJhnJGz+vsxnE3TO6upH4rONAZxFPvF32KwFXg0FX0/WPhQnKD/mPxJcZh/4GTjkYBRC8EpV1NUrJc0ohxIGhoqmRh5YvY1lFBcNzcrhs5mxmDRna12ENeKnaudx1xlmcfMi4PopMHCikrY0QIm1aR8EuBRWMF3Ha93W3Hl15LBBJcLUfVfgYOrYcGm6mY1Lrh7w/gP0JNP8fSasSq1wofAKqzkoyzgBkzQR7MxBivxJwawoYhRB9n04TXpULKgeIge9EVNZXUVZ6DeeFEEKIRBZtL+MXb75GaX0dCkVRMMjPP3MC88dN6OvQxAFA2toIIbrARYdfhNDDaN0A5mhUzrdR/jMAUEYeOveX0HAD8WSyNXlSAQhchvJMQnkm4bp10PSneDEnrYEoGNnQdCsYxaRcjquB0CPEvxwdIAmrs4l4sppoWa+Kz4LqMImT0QAq+EXwzkJXX9i6rDfFnmNscKvAMwsVOFeSVSGEEN125IiRvHT5VexqbsZ2XYZmZ8t2EtFnZIZViIOU1ja65lKIraZ9ohiA7G9iZH91z7mx5eimf4C9Dszh8aWlvuPa388NoSNvQsMvW5Os3TOqFvFZxlRfa3yt5/efr0c9R0HOL8AzHWq/2vqxal0yrYLgnYca9CeUMtF2KbrpToi8Bbjx2VR3V/z3bc8b916y5UcNuhPlP7EX348QQgghRPpkSbAQIi06/DK6/segEzXv9qEGv4cy8rp0T7f22xB5lcTLYDsrqGS0vp7m1yTfORB5B6jtUow9r7P3YaEGL0IZ2Wi3AR16DCJvg8oCaxTYGwCN8p8NgTNRytfuaq1ddNNd0HwPCZdZG4NRxe+glBSBF0KI3uJqzdtbNvPftWuwXYfTx01k/rjxeBK0demP6sNh3izdTMSxOWrEKEYPGtTXIYkDmCwJFkKkRbc8kyRZBZQFkffQ/pMg/Hq8qJI1EbxHJ02EtHYg8hrJ92xq4jOpyZb9uoAHlCd5XG3xFaMG3YJuvA1C96U+t40FeCF4CYTupfPE2GT/CkB5wRoP9orELxtD4skpoIxcVPY16OAX0NUXQWxB23vXsY+g+e9Q+BjK2NNTUCkDHf4vSfcE66Z40uuZuB+xCyGE6KqIbXP1M0+yvLKCUCy+BebN0s38cdECHrvwYnJ9/j6OMLV7P17K7z54F8swcDW42uWUseO4Y/5nB0zCLQ5s8gheiIOVTtGKRmu0vRpdeTS64efoxtvRdd9CV52CtsuSXOSQuhVL61JYUnzjNoejcm8Co2M/tz0MVPErKOVFZV9LPAlOfm678Y0AGHmpY4B4ESPP3NbzPK1j7P3/1vthtv7yA974kt2cH4BZQtIvr24VuuqzaKe87ZBuvB2cre0Tdd0CzjZ0460d76FTJdKKzLUZEkII0Zm/LV3Msp3lbckqQCgWo7SujhvffrMPI+vcW6WbuX3Be0Qch+ZYjBY7RsRxeH3zJn7z/jt9HZ4QgCSsQhy0VOD0+J7JhGLQ/ADo5vgvYvH/OzvQtVeTaCuBUl4wUxT8scajAmeQcmbTKUc3PwA6xTJfVYAy9sxQUvAAe5LItpNa/793Ah0DtxZC/yF1Yu1B5d+LUXgfqvARVM73UTk/RhW/jhq8EJV7PfjPhaxrIPv7tGtho5uh8RaIvNQ6RqICFRFwStE1V8bbC2kNLU+ReMY0Bi1Pd/x4+04m+QIZEyyp4iiEEL3loeXLOrSAAYi5Ds+vX0skwWs9TWvNyxvXc8F/Hmbe//2NS5/4D+9sKe1w3p8WLaAlQXxh2+aRFcsJ2wOph7k4UEnCKsTByn9mawXffZO9QLytSsKkzo0X/YktTnhLlfMDEs9e+lHZP0AZ2RC8Ij5GQtF4G5xUS4J1Xbs/Gt5DUUXPgu+M+FJbld3a5iURF9ydJE+avZB3K8o7K/5+PFNRWdegsi5HmUNRRhAV/ALGoFtR/rOg6S7iS5wjxGc1dxeX2v2xSzaOE19mHV3Yek2qPrUxdldZ1k45bv2NEH6ZxMuV/ZB9HUrt+3cqhBCip9RFkn8N10BzLMkWjh50y7tv8f2XX+Sjih3sbG7iw+3b+Przz/CXxQvbnbe+pjrpPQylKG9q6ulQheiUJKxCHKSU8qEKHwP/6YA3/kvlQfY3WxPZZHskXbA3Jb6n/zTI/UVrb9Ds1gQyD3Jvaqtcq3J+AMEvEk9akyWuqQLP7XjIGouRfyfGkI8xhnwEKsUyYeUDzxw6JtYesEaj/Keh3Rrcpj/jVl+OW3stOvJuh1lO3fwPkn6M0qFtsNehlAXGsOTnGUNQyou2N6GrzoKWR0BXEv8xaPcMrheMwZB7A0bWJfsfkxBCiC4blZu8QKHftMjr5T2s66ur+feK5bTsMzvaYtvctWgBFU2NbccKAslWWkHMdcn39+/9t+LgIAmrEAcxZQzCGHQ7ashHqMHvoAZ/GG9nY5WQdMmpMuOFg5IwghegBi9A5f8TVXA/avACjOA5ey5XBkbudajBC1qXtnalr5sPgpd1fpo1OflrOgaD7oDg5a1Lov3x+/pPRxU8DPZG9K5ToeluiC2CyCvo2mvR9T9on7TGVpN6aXEnlAfM4vjvs78V723bQQCyr42H3XB9vKBSu/2pGrDAfxaq+F2M4Of3Px4hhBD75VtHziNgdfyeGbAsrjn0cEyjd3/c/u/a1cSc5LUOXtywvu33V88+LGHsllIcPbKEQf79eLAsRIZJwiqEiBcwMgpQKl4NUAUvJl5MKBEv7NODteP9PCjvLJRnZnwGMdE5RhCMIF3qvWrkofbqD5t0/Oyvk2xpMoFzMMwCjNwfxdvLFL+CGrIIY9DtoHLQdd8C3Uj7asYtEH4NIq/sOWQOTz/uZHwnxeMNnAdZXwF8e2am8UHWNajABWi3CaJLSfyxsiHyqjR0F0KIPnLOpMlcc+gcvKZJ0PIQsCx8psmZEybxjSPm9no8TbEoTpK2lTHHJbTXEuVLZ8xi3sgSgp49W0mCHg9DsnP47amn9XisQqRD2toIITpQ1iHo3J9Bw83EZxFjQACUiSq4J2kS2uVxfMehw8913sZmN7ce7I3gmZr6vt456NzroeGG+IwwOl5Z1/cZVO4v9pynvGAO3XOhva51j2siLejmB+PLngGVdRU6tiRezTel1vFRxPedeuMfx0F/buuxqpRCZV+LDl4B0Q/j53vntfXB1W6ElM8XdbJWQUIIIXqaUorvzzuGK2bN5u3SzTiuyzEloxmZYqlwTzp6ZAmPr1pBc6xjwSS/x+KI4SPb/mwZBn8/+1ze37aFJ1evoiUW45Sx4zhr4iT8ltRDEP2DJKxCiISM4EVo79HolsfB2QGe6ajA5+OVeTPFd3J8ebGzjfRascTQLU+iOklYAYzg+Wj/6RB9B9wQeOegrNGpL3KrSfll0a3a83vvsRC4GEIPE9/L2tpHlhjx2V03nixbUyDn/0HLM/H36ZmOCl6EMjsuq1ZGHvgTPNE2CuK/3IrEcXlmpX5fQgghelxxMIsLpk7v6zA48ZCxDM7KZltDPba7Z+uKxzAZX1DIEcPbt44zlOK4kjEcVzKmlyMVIj2SsAohklLWKFTO93ru/sqCwkfQ9f8LkXfoPGl1461p0r2/kQX+M9IPyJoIOlkhJRO8exJDpRQq93/RgXPQoSdAV4NnXnwWN/p+fL+p9zCUZ2b8Au/M9OPY930ohc6+Dhp+TseKwn5Uzvf3+95CCCEOLJZh8J8LLub7r7zIwu3b8JomUcfhxDGH8NtTTpctJGLAkYRVCNGnlJEPg/6MrpzXoWVNx5ODKO/RPReLWYT2nxLfr8q+y2w9qKyvdLzGMxWVt8+Mr3VBynG0XQqxT+L7VX3Hti0NTsUInoNLGBpvI57Yu6ByUXk3obyHd3q9EEKIg0dhMMj9557PruZmypsaGZGTS2EweUVgIfozSViFEH3PXsPuXqPJGfGqvoEzezQUlfdrtI5A5F3Aai1ibKDybkNZ47t1b61b0HXfgciC1r218afcOve3GIH5nV5vBC9CB84He31rleFx8qRcCCFEUsVZWRRnZfV1GEJ0iySsQoh+wKHT9jbWFNSgu1CqZ3vCKeVH5f8FbW+F2HIwcsF7VLxAUzfFlz4vACLtC/7W/wBtPYryTEkjPgvSOE8IIYQQ4kAgCasQou9Zk0mesFoQOA8j7+ZuD6Ojn6Cb/x6f0TWGoLKuBt8pCWcplVXS2o82M7RT3brUONEe2Si6+R+oQbdnbDwhhBBCiAOB9GEVQvQ5pbyQ/X069k41QGWjsr/T7THc0BPomisg8lq8Wm9sCbr+B+iG67t977TYGyDpXlU3PpsrhBBCCCHakYRVCNEvGFmXQe5NYAwj3h7GAu8xqMLHUWZxt+6t3UZo+CXxCrt7rcXVLdDyNDr6SbfunxazEHSKfbpG996jEEIIIcSBSJYECyH6DSN4DjrwuXi1YOVHqUBmbhx5AzCTvBhFtzyB8vZsL1NljUebo8DZQPsNrAABVNaVPTq+EEIIIcRAJDOsQoh+RSmFMvIzl6wC6GbihZ0SccHtpJ1Ohqj8O0Hl0m7pswqAfz74Oq8SLIQQQghxsJEZViHEgc8zm+RFnYIo37Gd3kLrMDhVYOSjjP1rEaCs8VD8Kjr0OETfB2MQKnAheOdJexohhBBCiAQkYRVCHPCUZyraMwtiH9O+Sq8BRhD8ZyW9VusouuHX0PIEKAXaRfvno3J/iTJyuh6LMQiV/WXgy12+VgghhBDiYCNLgoUQBwWV/1fwnQh4QeUAfrCmoQoeRRnBpNfp2q9Dy+NAOF6kiQiEX0LXXIbWyZYZCyGEEEKITJAZViFEn9DaAXsdoMCagFLJiiJlhjKyUfl3oZ0qcErBGBzvtZoqxtgqiC4GIvu8Eou3xom8A/4TeypkIYQQQoiDniSsQohe54aegcZfEV+eq0EF0DnXYwRO7/GxlVkEZlF6J0cXAHbi13QzOvIWShJWIYQQQogeIwmrEKJX6fBr0PBz4j1Rdx8MQf2P0EZ2WgWQeo+HeDucREmrAuXr5XiEEEIIIQ4usodVCNGrdOOttEtW24TRjb/r7XBS85+S/DXlRwWSF2sSQgghhBDdJwmrEKLXxFvDbE1+gr0Grd3eC6gTyhwOwS/Ge6W2EwDfSSjPzD6JSwghhBDiYCFLgoUQvcgi/pwsWVLqIXm/1L6hcn4Anino5rvjhZaMwRD8Eip4UV+HJoQQQgipX+xqAAAgAElEQVRxwJOEVQjRa5Sy0L4TIPIGHZNWE/xnoFQ/S1iVgsBZsvxXCCGEEKIPyJJgIUSvUrk/BZVHfDZ1Ny8Y+fHZTCFEWhztoLXu6zCEEEKIHiUzrEKIXqXMEVD0HLr5Xgi/BCgInInKuhplFPR1eEL0e4tqFvP4tifZGanEozzMKzqKi0ZeQLYnu69DE0IIITJO9aens3PmzNFLlizp6zCEEEIMAI52UCgMdfAsFnq14nX+U/Y4UTfadsxUJgXefG6efgN+09+H0QkhhBCdU0ot1VrPSfd8mWEVQggxoHxav4JHtz5GWct2DKU4dNChXFLyBYp8RX0dWo+KOBEeK3uiXbIK8cS9PtrAO7veZf7QU/soOiGEEKJnHDyPpYUQQgx4S2o+4o/r/8y2ljI0Gke7LK39iOtX3khttLavw+tRG5o2Jp1NjuooC6oX9nJEQgghRM+ThFUIIcSA4GqXB7c81GGGUaNpscM8X/5i0mu11rj9qMfv/tCk3sLTfzb4CCGEEJkjS4KFEEIMCBXhnbQ4LQlfc3BYXLOUy0df2u54XbSOR7c9xqKaJdjapiQ4ii+MvIAZg6b3RsgZNSF7PI52Er7mVV6OKjiylyMSQgghep7MsAohhBgw3C4UCmyMNfKLlTfwYfUibG0DsDW0jT9u+DMLqxf3VIg9xmf6uGDkeXgNb7vjJia5nhyOH3xcH0UmhBBC9BxJWIUQQgwIaxvXEdOxhK+ZmBxRcHi7Yy9VvEKzHcKl/VLgqBvlwS3/GpBLhE8beipfGnMlxb4iFApLWRxVOJfrp/2cgBno6/CEEEKIjJMlwUIIIfq92mgt/9rycNLX/aaPM4ed0e7YoprFbTOr+4q5UcpatlMSHJXRONNhuzY7wuV4lIeh/iEopbp0/byio5hXdBQxN4apzIOqrY8QQoiDjySsQggh+r0Pqj5MWnTIQHHa0Pnke/PbHe+0SFEf9CF/reJ1ntj+FK7WuLjkWjlcM/ZqpuZO6fK9PIanByIUQggh+hd5LCuEEKLfq43VJp0tddEdlv0CzMk/HEslfi5rKYuRwREZjbEzb1a+zaNljxNyWgi7YaJulKpoNb9f90dKm7f0aixCCCHEQCEJqxBCiH5vTHAMiuRLZ4f5h3Y4dsaw0/Cb/g7XeQ0vl5RcjKnMjMeZjKtdnih7skNLHojvqX2y7Olei0UIIYQYSHosYVVK/VIptV0ptaz112d7aiwhhBAHtmwrK+US32Y71OFYniePG6b9nMPyZ8f3emIw1D+E/xn3FY4tPronw+2gOlpDJEGyutu6pvW9GI0QQggxcPT0Htbfa61v6+ExhBBCHOB2hMsxMBIu/QXY3Fya8HiRr4hvT7gWRzs42unQEqa3+AxvyqrEHiX7UYUQQohEZEmwEEKIfi/XyklZZKi8pTxlESVTmX2WrALkenIZHhie9PW5hUf0YjRCCCHEwNHTCeu1SqnlSql7lVL5iU5QSn1VKbVEKbVk165dPRyOEEKIvqC1Zk3DWp4se5rndrzAzvDOLl1/eMFhKZcEbwlt5bnyF3C1y6f1K3h46394ouwpykJlacW2sWkTn9QtpyZa26W4umKIb3DS10J2S4+NK4QQQgxkqjtl/ZVSrwEdK13AT4EPgSpAAzcBw7TWX0p1vzlz5uglS5bsdzxCCCH6nxanhVvX3MH2lu1E3Agm8d6hJw05gUtGXZR2H9IlNR/xpw1/SZq4Bkw/xd5idkYqibgRDAwsw+SYwqO5cswVCcfZ2LSJu9b/hRYnhFIGthtj1qBZfG3clzM6I6u15qtLv5Gw6BLElwTfM+evXe7JKoQQQgw0SqmlWus56Z7frT2sWutT0jlPKXUP8Fx3xhJCCDEw3V/6IFtDW9va0jjE95O+Vfk247LGpb0cdnLORCxlEdOxhK9HnCjbW3bg4ADg4hJ1XT6o/pAJOeM5pqh9oaWaaA2/XXMbETfS7vgndcu5e+M9fHvCNxOOs7ZxHU+VPcPm5lICVoATiz/D6UNPw2f6ksbuaCdpsgpgaxtHO0nb8AghhBAHq56sEjxsrz9+HljRU2MJIYTYP7uXwy6sXsTm5tKU+0D3R4vTwpKapQl7qEbcKM+Xv9BpfK52eaD0Ib6z7LqkySrEE9TdyWr7cSK8UP5Sh+OvVryOozueH9Mxltd9SnWkusNrC6sXc9va37O6cQ1hN0xttJZnd7zALat/Q8xNHptlWBR6C5K+XuDNxzIkWRVCCCH21ZPfHW9VSs0mviS4FPhaD44lhBCiiyrCO7lj7R+oi9WjUGhcCr1FXDfpOxT5ijIyRm20DlNZxBIkrABVkaoOx+qidTxW9gQLqxdja5scK4eQE0qY9O62u9dqsuXC1dGaDsfWNK5Lek/LMNncvIVCX2HbMdu1ua/0/g4zpTEdozxcwQdVC5ieNw0XTZG3sG15b3Wkhke3PZZ0f6ylLDzKwy2rfsMRBXM4rvgYAmYg6XsVQgghDiY9lrBqra/oqXsLIYTonpgb41erf0NDrLFdklcRruBXq3/L72b9BlOZ3R5nkCcvZaJZ4C1s9+fGWCM/X3EDjfaeuBrshpRjeJQHj+Eh6kSxSTxWcYIEPMfKTnpPreO9X/e2vmlD0hnoqBvlgS3/QqFQCnKsHE4oPp61jetY1bAKN0EibSkLW9u42qUispOKyE5KQ1t4qeJlrp/2M/I8eanethBCCHFQkLY2QghxEFpSs5SIE+kwI+ni0mw380ndpxkZJ2gFOTz/0IR7M72GlzOHn9H252Y7xK1r7qDBbkhZEXhvJgafHXY6v515C0cWzmmbad2bgcEZQ0/rcPykISfgM5LtO9UM9Q9pd8R2bUhRFMnWNjEdI+rGqI7W8MT2p1jRsDJhsrr7fKBdb9moG6UuWs9DWx5OOs5A1BBrZEnNRyyr+4SIE+n8AiGEEKKVbJgRQoiD0KbmzYTdxIlD2I2wJbSFw/JnZ2Ssq8Z8kYrwTirCO9uq95rK5DNFx3FUwZEARN0YN626hfJwRZfuPSwwjPNGnguQNPl0cfnbpn/wXtUHXFpyESODIwGYlTeTw/IP5aPajzsUXnK0ww8++Qnnjjibs4afCcC47LHxpLWHOTh8VPsxMTeWsvfsQOBql4e3PsqblW+3zdhrNJeWXMQJg4/v4+iEEEIMBJKwCiHEQSjPk9e2JHVfXuUl18rJ2FhBK8gN037BqobVrKhfic/0cWTBEQwP7KnNt7B6YZd7oHoNLycNPhGIz36+X70g5czsyoZV3LjqV/xy2s8YHhiOUoqvjf0yH9d9wgOlD1Ibq2s7d/ee22d2PMe2UBnV0RqyzCxmDZrBJ3Wfpiz+lClRNzrgE9bndrzAW7veIaZj7T5m/9r6CIXeQmYMmt6H0QkhhBgIJGEVQoiD0NFFR/H09v8mfE2jObIgvVYz6VJKMS1vKtPypiZ8fUH1wg6znKn4DC+HZB3C8cXHAdBkN+O6bidXxZPAx7Y9yXcmXtsW19TcyTTZzUnPX1izuC0R9hk+in3F1ERr0LjYroOLk+YC5vQFzSBBM0hdtI6XKl7ho9plmIbJMYXzOHnIiQOiKJPt2rxQ8VLCdj5RN8qT25+RhFUIIUSnJGEVQoiDUIG3gEtLLubhbY9iuzYuLgYGlmFy1ZgvkuPJ3AxrOpLvDI0zMZmeO43aWC1+y88Jxcczt+CItlYw2VZW/CadZI4azSd1y9sdq4nWYCoz6azp3rO2ETdCdbSaK0dfzoScCQTNAPeXPsji2qVp77vtjEJx+tD5VEZ2ccPKm4m4kbaZ8Ke3/5e3d73DL6f9nKy9ikJF3Rgbmzai0YzPHofX8GYklu6ojzUkbBu0W1lLWS9GI4QQYqCShFUIIQ5SJw05gfE543it4nV2hCsYGRjOqUNPYURgeK/HMq/wKFY1rGlXgGhvnxt+FueO/FzS6y3DothXTHm4vNOxHBxe3/kmJw+JLyfO9eSmrGS8r4gb4Z7N9zIldzKXjrqYC0ddwNLajxP2gN1f71a9x4qGFYScULtEOKZj1ERreHr7s1w2+mIAXt/5Bo9uexxDKdDgorlg5HnMH3pKxuLZH0ErgKuTz3oHzWAvRiOEEGKgkoRVCCEOYiXBUXxp7FV9HQZF3qKkySrAixUv8X71BxyWfyinD51Pvje/wzmTciamlbACPLLtPxT5Cvm4dhmrG9fiSbKfNxmNZlXDan628nqKvcVJk1WFwsBAowmYAZqdxEuP9713TaSGneHKhLO2tnZ4v+oDLht9MYtrlvDItsc6LLt9rOwJcqwc5hXNbXd8Z7iSmmgNg32DKfQVpP1+90fADDA1dwor6ld2+Lv1KA8nDT6hR8cXQghxYJCEVQghRJ/SWvOXjXenPCfsRghHdvHaztd5d9d7XD/tZwzZp+3MjLxpvLXr7bTGjLpR7lh3537HvLdd0V1JX9NoHBwsZRFyQmnfM6KjGCkWSu9OUB8veyrpHtEntj/VlrDWRGv40/q/sjW0DcuwsF2bSTkT+fr4r5Kdoh9td119yJXcsPJmQk6oLU6f4WNEYARnDDu9x8YVQghx4JA+rEIIIfrUu1XvU283pHWurR2anRDXr7yJlfWr2o43xBq5b/ODPRVit9na7vIe11RnH5I1Ble7VKRoA1QVqSLqxog5MX6x4kY2NW8mpmO0OC3EdIzVjWu4dc3taJ3pklF7FHjz+c3Mm7lw5HlMzJ7ItNypXH3Ilfx0yo/xDvAKyEIIIXqHzLAKIYToM1prHt/2RJeva3Fa+P26P3LZ6Is5cfAJvFzxKi1uSw9E2HeSJbhew8v5Iz+PQuFRnqTFokxl0mKHuHHVLTTajR1ed7RDRXgn65rWMylnYkZj31vADDB/6KnMH3pqj40hhBDiwCUJqxBCiD6zI1xOyNm/RDOmY/xryyPMKzyKRdWLurQHdSDyGl68hperx3yRybmTAJhXOJf3qxZ02ENrYnJkwRz+tOGvVEdrkt4z5sbY0LghYcK6oXEDz5e/RGmolGwrh9OGnMKxxcdk9k0JIYQQnZCEVQghRJdpHS869GHNIhzX4fCCQ5k9aBamMrt0n5AdwjIsYk7iWcLOxHSMVyteT5mU7U21/peqwFN/ZbsxHNfhufIXKPYVMTprNF8YdQGrGlbTEGsg2jrT6jU8ZFvZnDT4RG5dc1vKpciWYRGwgmitWd24hk/rVuA1vDiuzYs7X2l7CFATreWezffy1Pb/ctP06wlaUuFXCCFE75CEVQghRJfYrs3v1/2R9U0biLgRAJbULmWwr5ifTv0JATOQ1n3qovW4Wu93srrbMzueTauljNVaCXh/+6WamBltXdNVLhpw2Nxcys2rfsOZw05necPKdntQDQwmZk3ka+O+wrqm9ZiGCSk+vlprpudO48ZVt7C9ZQcRN4KBah2ro6poFXes+wM/m/r/Mv32hBBCiIQkYRVCCNElL5a/zLrG9UT1nuq0ETdCebiCf295hGvGXp3y+rpoHX/b9A/WN67HUCY23VvKm2wP5766u2S4L5PVfUV1lKd3PNsh+XZxWdG4kh8u/wlfGvNF3E4KKh1ZMIentz/D1tC2to9PsmR1t41Nm6loqWBoYGj33oQQQgiRBklYhRCiH1nfuIHHy55kQ9NGPIbF3IK5nD/yXHI9uRkdJ2SHeKH8Jd6teo+oG2Nizng+P+JcxmSN7vTaVytfb5es7mZrmwXVC7lyzBVYRuJvL1E3xo2rfkVttDa+LDdFEmlgDMilu70l1Uxx2A3zj833kevJIRKNJD1vUc0StNZdemigUGxrKZOEVQghRK+QtjZCCNFPfFq3glvX3s6axrXY2qbFCfPurvf4xYobaIo1ZWycFqeFX666mRcrXqYuVk/ICbGsbjm3rP5Nu1YxyTTZyWNxtUPITt5vdHHNEprsprQS0UJvAVlmFn7D3+m5oqOYjjEjbwYBI/kSbUc7XX4oYCiDHCunu+EJIYQQaZGEVQgh+gGtNfeW3kfUbT9z6eDQaDfxYsXLGRvr9Z1vUBOp6bBENupG+cfmf3bal7PIW5j0NQeXH3/6//jVqt/yUsUrNMYa2RraxpbmLdiuzaf1K9r2vXYm7Eb4/oTv4lO+tM4X7Wk06xvXc2zR0RhJvt27rf91RcD0MzFnQiZCFEIIITolS4KFEKIfqAjvpMluTvhafKnth1w46vyErzfEGvig6kNqYrWMDo7iiII5eA1vwnO11ry2882k+z6b7Sa2t+wg3zuIp7f/l/erPiDiRjkkawznj/w8U3Inc/bwM7m/9KGk9wg5LaxtWsfapnU8vPVRDAxMZaLRFFj5aXw04prsJm5ac0va54uOysLb2RmpTJqUKhRF3kLqYvVp7QX2Ki/fmXAthpLn3UIIIXqHJKxCCNEPONpGoVK8njjhWFi9iHs23QvEl4D6DR//3voI/zv5R4wMjmx3rtaaezffT22sNuk4CoMmu4k7199FTbQGW8cLDa1v2sAd6+7kK4dcw7FFx7Cs9hOW1H2U1ntzcXFb46+M7UrrGki9R1OkL1UiqlAM8w/HY3ipDFdiY7c9YDi++DNMyB7Hm7veJupGmZY3jZMHn0C+N/2HDkIIIUR3ScIqhBD9wPDAcDyGJ+FyWQODQwfNbnesKdZEZWQX92y6t11CEnYj4Eb43drf8/vZv2s3E7ayYRULaxZ1Gsu6pvXURuvaktXdom6U+7c8yJyCwzi84HA+bVhBxO1YfEkMHC4uyxuWtzumVPzByYTscRxVNJejiub2RWhCCCEEIAmrEEL0C4YyuHjUhTyw5V8d9rH6TB9nD/8sABuaNnJ/6UPsaNmBo52ks5BhJ8zKhlXMyJveduyNyrdS7h/1KA/njjibd6veTzorZ7s2pc1bGJs1ptOWKWJgcrSDg8M/Nv+TkNPCIVljGJM1ui2RbbKbqIvWU+DNJ2gF+zhaIYQQBzpJWIUQop84rvhYLOXhsbLHqY81oNFMzJ7AF8dcTqGvkC3NW/jtmts6JLSJuNphV6Sq3bH6WH3Ka44qnMtxRcfxUvkrSc9RKp7QDA0MZXLuJNY0rE27D6oYWGI6xoNb/oXH8JBlBbl6zJW8Wfk2y+s/xVIWjraZU3A4V435In5TKjkLIYToGZKwCiFED3B1fN9msn6kycwrmstRhUfSaDfhMSwC5p6WJI+VPZlWsgpgKJMhvsHtjk3KmUhp85YO1YEBLCyyzCy+u+w6dIqqsVprRrf2av3W+G/wt03/4JO65Slne8XA5eIScSNEohHuWHcnBgYOTtvn0OKapVRFqvnplJ+0zcB2x65IFa/vfIMtoa0U+4o5ZchJlARHdfu+QgghBi5JWIUQIoNqorU8vPVRltZ+hKtdhvqH8IVRF3JY/uzOL26llCLX07HP5eqGNWnfI2gFmJI7ud2xU4acxGs73+iQsHqUh6H+IbyxK3n1YACv4eW8EefiNTwAhN0wowIjqYnU4mCzLVQmSesBTKNxaL+v2dY2W0Pb2NC0kQk547t1/0/qlvOnDX+NL0nWDgZrWVD9IReM+DynDZvfrXsLIYQYuKQuvRBCZEhDrJHrV9zIkpqlbTOO5eEK/rrxb7y364P9uqft2iyqWcwTZU+ldb7P8JFr5fLDSdd1aD1S4C3gR5O+zyBPHn7DR8Dw41EepuVNIexGUs7e5lq5XF5yaVvisKV5Kz9e/lOeL3+RzaHNbA1tk2T1IBV1o6xpXNute0ScCH/ecDdRN4rTWuzLxSXqRnms7EkqwjszEaoQQogBSGZYhRAiQ16peJWQE+rQ8zLqRvn31keYVzQXU5lp36+ipYJfrbmViBMh7IZTtr0p8hZxdNFRjAyM5PD8Q5MuRR6fM57fz76NDU0babKbGBUcRaGngKuXfCVlLHMLjiDmxni54lUG+4r599ZHaHFa0n4v4sBlKhNP66w7xD/fP6j6kLd3vUtz6+fY2cM+y5jsMUnv8VHtx0k/u13t8vaud7ho1IWZDVwIIcSAIAmrEEJkyKKaxQn3h0K8UNHW5m0ckuKH9r252uV3a39PQ2vxJUjcl1Sh8BpevjXhG4xp3VvaGUMZTMyZEL+n1ty7+b5Or3m18vW28TzKQ1RLOxsRp1DMyT8MgLpoPb9ceVO7Xr87I5Usrf2IE4qP58oxlyfc69pgN3Zoo7Sbg0NttK5nghdCCNHvScIqhBC9pQs1adY2rqPJbkyapJrKxFQm0/Omct6IzzMyOGK/QlpSuzSt3qy7abQkq6KN1/By+pD5FPmKaIg1cMe6P7RLVnfTaN7Z9S7T86Yyp+DwDq+XBEdhKjPhAx+v4WVc1tgeiV8IIUT/JwmrEEJkyNzCI3mx/CViiarwKrNL1U4rI7twk+wJ1Wim5k7huknfTeteWmvWN21gQfVCom6EWYNmcmjebLa1lPHotseJpFl5WIi9KRQGisMLDuOB0od4q/KdDkWZ9ubg8GLFywkT1sk5k8j35lMZruywpN5UJscUzct4/EIIIQYGSViFECJD5g85hbd2vYMTa2r3Q7fX8HLp6Eu6tH+12FeEkaQunonJ8MCwtO7jape7N/6dZXXLibpRNJpF1UtwcTGUkXabHCH2pdGE3Qi3rPoNKFImq7vtTFA8KerG+LD6Q4JmoK2/q6ksDGXgN/18f+J3CFrBnngLQgghBgBJWIUQIkNyPDncOO0XPLLtMZbULMXWNiMCw/nCyAuYnT+rS/eanDOJoBUkHA13eM00DE4afGJa93ln17ssq/uk3Sxq25JeKeorMiCqo2l/LkWcKFrrtn2sYSfMzat+zc5IZdvDE0tZWIbJ1WOuZHRWCZubS6mP1TM1d0qX+xoLIYQY+OQrvxBCZFC+N5+vj/sqeqzGxe3SrOreDGXww0nf41erf0vUjRFxI3hUvBLr1WOuZIh/cFr3eaniFVnyK/oNF5eqaBXFvmIAnt3xPBXhinbL6G1t4zgOD235NyGnJf5vSMWXIP/PuK8we1DXHv4IIYQY2CRhFUKIHqCUwmT/ktXdhgeGc8fs21hcs4StoW3ke/KYV3QUeZ68tO9RF6vvVgxCZJKpTGx3z9Lht3a9k3DPt0ZTbzcAENOxtuN/3nA310/92X4XGRNCCDHwJN4gJYQQol/wGh6OKZrHJSVf4PRhp3UpWQUY6hvSQ5EJ0XUew9NudUDY6bjkPRXbtXmh/KVMhyWEEKIfk4RVCCEOYKcOPRlLyWIa0T9cNOoCDLXnR4+Rga7NlLq4bGzelOmwhBBC9GOSsAohxAGo2W7mznV38c/N97cdi7chkS/7ou8cV3Rsuz+fN/LzeA1vl+5RGa7k/aoPMhmWEEKIfkx+chFCiAGkOlLNivqVlIW2Jz3H1S6/Xv07ltevIKZt7NY9ggqFz/T1VqhCtFPgyW+rDrzbrEEzuKzkEnyGj4DpJ2D4sbDIMrOS3sfF5b7SB1lYvbinQxZCCNEPyDoxIYQYAJrtEHdv/DurG1ZjGR4c7VDsK+bbE77JUH/7faorG1ZRGalsS1R3c3Gl76roMw12IxUtFQwNDG07VhutZUtoCx7DwnZtxmSN4ZJRF2EaJres/g1hJ4xO0DMn6kZ5dNtjHFkwp0MSnAlloTJerHiZzc2l5HvzmT/kFGbmzeiRsYQQQqQmM6xCCDEA3LHuD6xqWE1M27Q4LUTdKDtadnDzql91KFyzpmEtETeS8D6OdlDID92i97na5bXKNwD4uHYZ16+4ke8t+yFvVL5Fk91M2I2wrnE9v15zKwq4deavU96vPlZPs92c8TiX1HzEDatu4YOqD9nesoMV9Sv584a7eXDLvzI+lhBCiM5JwiqEEP3c5uZStoa2dZgx1WiibowFVR+2O+43/Elb6sgeVtFXXFzKQtt5YttT/GXj3ygNbekwe+riEnbDPLjl3+R6cvAZyZewazSWkdmFYhEnwt83/YOoG8XF3XPcjfBu1ftsaNyQ0fGEEEJ0Tn5yEUKIfm5j0ya07rgsEuI/SK9sWN3u2OTcSTg4Cc93cWWGVfSZAm8+L1a83OnS9PVNG4g4EeYWHpn04cuE7PH4TX9G41te/2nSfx8xN8Zbu97J6HhCCCE6JwmrEEL0c0EziKkSf7k2MMj15LQ79mH1wpT323vmSIjeFHbCHVYKJGNrm/NGnEu2J7tdayYDg4Dp58oxl2c8vpDTgk7y70OjaYg1ZnxMIYQQqUnRJSGE6Mea7RB5nhwcnfiHaMuwOK64fauQhTVSPVX0Pz7l46O6ZQmLKO2ryFtE0AyiLMXN02/gxfKXWFC9EBeXQwfN4qzhZ1LsK8p4jGOzDkm6msFreJmaOyXjYwohhEhNElYhhOiHbNfmoS3/5r2qD7AMK+GsqM/wcsLg4zkka0y7465OvBxYiL5iYBDTsbSSVa/ycknJF9oq8uZ6crio5EIuKrmwp8NkVHAkY7PHsqFpY4eZYI/q+HBICCFEz5OEVQgh+qF/lt7PopolxHSMmBNrO24qkyJvIcX+Yk4bciozB83ocO2MvOksrFmcVnIgRE/zGz6UNmjRLZ2eW+DN57KSSzk0f3YvRJbYdyd+i7s3/p2V9avjD4u0Q743n29PuJYsK9hncQkhxMFKElYhhOhnaqO1fFi9KOFePwODowrnct7Ic5Nef+6Iz7G09mNiOpb0HCF6iqUs8j35FPuLOKZwHvOKjuJLi7+a8prB3iKunfANSoIlafU6bXFaeLPybd6reh/bdTg0fxanDZ1PgTe/2/EHzADfm/gdqiM17GjZQZ43j1GBkdKDVQgh+ogUXRJCiH5mc3MpHuVJ+FpMx/i0fmXSa7XWPL3j2Z4KTYhO2dpmiH8wnx12OkcXzcNUZqeVqSujVfxh/V18UP1hyvMAQnaI61fexJNlT7O9ZQc7Izt5teI1fvrpLyhvqcjU26DQV8CMQdMpCY6SZFUIIfqQzLAKIUQXRN0Yi2oW80ntcvymj6OL5jE5Z1JGf6D1m/6Uy3mDZqDDsYFV+AsAACAASURBVCa7ifd2fcCS2qVsbNoklYBFn1rRsJJ1Tesp9Bbwv1N+BGksT6+J1nJf6QNE3AgnDT6h7bjWmrAbxqM8WIbFc+UvUB2pbrcCwcEl5IT404a/csuMG3rgHQkhhOgrkrAKIUSa6qL13LjqFprsJiJuBIhX5J2ZN51vjP8fjCStZ7pqYvYETJW496TP8HHi4OPbHdvYtIlb19yOi9tpf0shekvUjVIR3skf1/+ZHCuXBrshrWse2/Y4nyk6FsuweL/qAx4ve4r6WB2gmD1oFmsb1iZtjVPWUsZHtR9zWP6hGX43Qggh+oosCRZCiDTds+n/qI3WtiWrABE3wif1n/LOrvcyNo5lWHxt3JfxGt52Syl9hpfJORPb/TDuaIc71t1J2A1Lsir6HY1mc3Mp8wrndroseDdXa7aFynil4jXuK32QmmgNjnZxtMPHtctocppTXv/YticzEboQQoh+QhJWIYRIQ2OskTWNaxMutY26UV4qfzmj480aNJOfT/1/HFlwBIXeQgb7ihnmH4arNS9VvEJTrAmAT+tXYLuJZ5uE6A8c7fDqzte7ULVa42ibx8ue7PAQxsXtNPEtD5cTcSIpzxFCCDFwSMIqhBBpaLAbsVTyXRQVkZ00xBozOmZJcBRfHvslCrz51McaKA1t4dOGFTy67TG++fF3uGfj/7Gqfg1hN5zRcYXItK7sqfYYXmLaxkiSmHaW+Go0C6o6L94khBBiYJCEVQgh0lDoLcDRTtLXNZr7Sx/M+LjP73iB0uYt7ZYh7/Ze9Qe8vPOVjI8pRF/xGl6uHH15fA93N+qYPbT1YbaGtmUuMCGEEH1GElYhhEiD3/RzbNHRKc9ZVvdJxpcivlH5pvRTFQeFHCuHLx9yNUcUzmFs1iFJl/56DU+ny4JtbfNyhTzMEUKIA4EkrEIIkabLRl+S8gdlQxmEnFDGxqsM76LJbsrY/YToz5rtZh7Y8hDVkRosw+KSURd1WIZvKpNcK4+xWWNS3kujKQvt6MFohRBC9BZpayOEEGnyGB5GBUcmXWpoKoMcK6fb40ScCH/ZeDcr61enXaZGiIHOxaXZDvFg6b/wmz4W1y7B1e3/BUzOnsQ3xn+N7eEd3Lrm9qTtbRSKIf7BvRG2EEKIHiYzrEII0QXnjvgcXsPb4bjX8HLa0PlYRvefA9698e+srF9NTMe6UFlViIFPo/m4fhkLahZia6dDsab1zRsIOS1MypnIt8d/E5PE/Yo9hofThp7aGyELIYToYZKwCiFEFxyefxjnjTgHj/LgN/z4DB8eZTGv8CjOGX52t+//cvkrfFS3LK19qxYmhnwZFwcRx3V4o/JNAGblz+TXM24m28rGqzxAfMmwR3k4b8Q5jMse25ehCiGEyBBZEiyEEF10xrDT+UzxZ1hRvwJb20zJnUKBN7/b931ux4s8tf3plOcYGHgNL0P8g/n/7d13fFzVmf/xz7nT1JvVLFtyk7uxDZjeCcH0ToCwhIRNyBJgF9J+ZCkJLD2kJ8suJCTZLKFDgNBCCR0CBgy4dyxbkiW5yKrT7vn9IVtroRk1azQj6fv2yy9p7rnlGXOZmWfOOc85texkxvjH8OeND7K6ec1eX18k1UWJsqltc+fjkvRifj7/Lt7f9gGrm9eQ68vhsMJDKQoUJjFKEREZTEpYRUQGINObwUFjDhy087VF23iy+qm4c/J2m5M7m+9Mv6rLtutm/YC/1b7EAxsf6td6lyLDjQcPY9PGdtnmc3wcWngwhxYenKSoREQkkTSWTEQkBaxqWt2x9mQvDis8pNu26rYaHq56VMmqjHiOcfhCyTHJDkNERIaQelhFRJJoZdMqnq15ng0tnxGMtve4r8GwsbWKg8cc1GX7w1WP9tozKzISfG3iVyhNK0l2GCIiMoSUsIqIJMkLtS/y6KbHCbmhPu1vsbyz9V2+VH5Ol+2fNi5RNWEZ8Yr8hRzyuWG/1losFsdowJiIyEilhFVEJAm2h7bzSNWjhPvZMxp2I0TcSJflc5SsymjQGN7JksalzM3bh+q2Gh7Y+BBLGpdisVRmTeH88nOpzK5MdpgiIjLI9JWkiEgSvL/tA8D0+7iWSAtXfPRvPLH5SVzbMWd1ds6sQY5OJPWEbIi3G96htq2WG5fezCeNn+LiYrGsbl7DHSt/wsqmVckOU0REBpkSVhGRJGiLtvVprdXPc3Fpi7bzbM3z/Omz+wH4Uvk5BJzAYIcoknLa3HYe3fQ4QTfYrS3khvjThj8nISoREUkkJawiIklQmTVlr5LMkBvi9fo32RFqpDxjPNfOvIZcX+4gRiiSeuqDDSze8UncYfDV7dW0RFrYFtrGI1WPcfvyH3Pvut+xtnndEEcqIiKDRXNYRUSSYGbODAr9Y6hprx3wcjRe42Vl00oOGnMgaZ50doZ3DnKUIqllS9sWIvQ873tl0yr+a+29RG2UiI1gmgzvbVvEiaULOWv8GUMUqYiIDBYlrCIiSeAYhx/M/D53rLiLqrZNAzqHa102tVYzNWsbty2/Q8WXEsCJuOz7xkYOfGk9mU0hdhak8daJlSw7oAzr9H8Osuwdi8Vg4t7rRf5C7l13X5chwxZLyA3xXO0L7Js/n0mZE4coWhERGQxKWEVEhtjiHR/zbM3z1LXXM8ZfgIPBHUCyGbIhnq15jqdqnk5AlFL6WSNXXfUSDf4CHio6g2pTxqQtn/Hl2x7n7OyP+OVPj2VHUUaywxxVdhdZiqcis4JPdnwasy3shvl73atMmvTVBEUnIiKJoIRVRCTBIm6Ej3YsZlPrZta2rGNl06rOtVe3h7fj4OxKWmMPDTYYHByiRLufu5fhkTIwOVvb+O5lL/Lzikt5ePyp+HOb8QTC/CM4i/8pO4cr1/6Bb1/2DDf/74mE0vRWOlR66l31O36yvVmd1bM/z2LZFtqeyPBERCQBVHRJRCSBqttquPrj7/Hbdb/nL9VP8Wnjks5kdbfdiaoXL37j79xuMEzKmMTsnJka7jvEjr5/DS/lHMXj048jo2Q73rQwxoA3LUxG6Q7unnseG0KTWPCXgQ3nlv4xGPyOn1PKTsJnfHH3m50zm5ANxWzz4mVqltZpFREZbvS1sIhIgrjW5c4VP+lTMSTHOJxWdgpzcmeT5c2kwF/AztBOrlv6I1pbW4cgWulkLUc9vZKL9rsCf1Z7zF382UH+OOEcvvfor3n7/IlDG98otH/+fpwz/iyCbpCnq5+Juc9+efvyesMbcc9hsRxdfGSiQhQRkQRRwioikiDLdi6nLdrWp30jNkJdex1Txp0KgLWWW1fcSWtUyepQ87dHyGppZ+P4QrzEXyt3VcV4ShfvGMLIRqcpmZO5cuq3APjl6t/EHW3w6Y5Pabfd12fdLeAJaOknEZFhSEOCRUQSZEt7HW6MeafxvLvtPdY0rQHglS1/pyHUkKjQpAdRnwePGyXN2/OXBdnsJLgXa+lK73K82Vw17V+BjqJJi7d/HHffsI1giF+5OezG//JBRERSlxJWEZEEKQwU4uDp8/4RG+GBqocBeLLmr4kKS3oR9Tr8o2Q+X1j9jx73W7jmbd4o23+Iohp9fMbHscXH8OTmp7hjxV3864ffjll4bLeojeJ3/HHbx6WXJSJMERFJMCWsIiIJsk/ubPye+B+gY1nbvI6ojdIUbkpQVNIXz524H5d+/AhZoZaY7cUtW7lg+TO8ePqcIY5s9IjYCE9V/5WX6/7Osp3LaXV77vFO96RzWtnJMZNWv+PnrPFnJCpUERFJICWsIiIJ4hiH703/NpmeTAK7ho56+1A6wGDwOfEroUrirT0/wNsl+3Pv0zewX81SsB3zJh03yuFVi/jdU9fxh6nnsPPE+HMmZe9YbK/rru5pYtYETihdyHElx+IzXtKcNNKcNAJOgC9XnM+8vLkJjlhERBJBRZdERBKoIqOcn83/Me9te5+NrVWM8Rfw0Y6PWdG0Mub+s3Nm4hiHbF82waCSoWQJ5LXyxG1TWX3rRK579b/xmTD1mfmUNdXR4BvDXbMvpep7QXJy65IdquxSGijBGMN55edy8tgTWdW0Bo/xMCN7GgGP5hqLiAxXSlhFRBIs4AlwRNHhnY/n5M7hP5bdStANduk9CjgBLqg4D+goNtMQVNGlZMqZUM/qOzP5xsdXk/1yGpmNYXbkp9N+bAvFc9eSkxt7uLAkx56jErK8WeyXPz+J0YiIyGBRwioiMsTGZ4zjh7Ov5ZGqx/ikcQkA++TO4dzyszsLw+ybN5+q1k2ErSqbJlMgt4XyIz+FXct3FiU3HOnBq3WvUZk1lQUF+yY7FBERGUTG2r7NDRkKCxYssIsWLUp2GCIiSdccbuaaT6+lOdLS5zl8IqOdoWNI8IljFyY7FBERicMY84G1dkFf91fRJRGRFJTly+KGWdcxLXsqjl6qRfrEYnmk6jFaIj1XFBYRkeFDn4JERFJUcVoR/z7z//Gz+XcxJXNyj2tMikgHi+XjHZ8kOwwRERkkSlhFRFKcYwyzcmaS6cnAZ3wEnAAzs2fwxeLjcDDJDk8kpVis5n6LiIwgKrokIpLCattquWnZrYTcIGEbAcBv/ITcEGeNP5P3tr1PY6QxyVGKpA4HhxnZ05MdhoiIDBIlrCIiKeyedb+jNdrapfBSyA1R1VbFC7Uv0BLR0ioie9ovf19K0oqTHYaIiAwSJawiIilqR6iRja0bY1YJDrlhXq1/DY/jIeJGkhCdSHIZTJf/Nxwcji46kgsnXJDEqEREZLApYRURSVGt0RY8xts5FPjz2qNBgm5wiKMSSQ1e4+XG2TeQ7k2jPdLOmMAYAp5AssMSEZFBpoRVRCRFFQWKoIc1WMemjWV96/qhC0gkRfiNj3PLz2ZcRtmuDcmNR0REEkdVgkVEUpTP8bGw9PiYy9n4HT8zcqbhNfreUUaX0kAJP5pzA8eXfjHZoYiIyBBQwioiksLOGHcaxxQdhc94Sfekk+akkeHJ4JuTv05lVmWywxMZUgEnwJcqzmFcelmyQxERkSGir+ZFRFKYYxy+POF8Tht3Cuua1+N3/FRmTcHrePntut8TiTO/VWSk8RovpWklzM+bl+xQRERkCClhFRFJkPXNG/hL9ZOsaV5HmpPGMcVH8sWS4wZUGCbLm8XcvH06H28LbeOdre8OZrgiKctgOLBgARdPvAiP8SQ7HBERGUJKWEVEEuDjHZ/w6zV3E3JDADTTzF82P80bDW9z2tiTKUkvYUrmZIwxAzr/ssbleIyn1x5WBwcXd0DXEEklnzR+Slu0jTRPWrJDERGRIaQ5rCIig8y1Lveuu68zWd0tbMPUttdy3/o/cueKn/D9T/6dmrbaAV3D6UMvkxcvV0/71wGdXySVWCxtkXaerXkh2aGIiMgQU8IqIjLI1rdsIOyG47ZHiBB0g9QH67ll+e3dEtu+mJs7B9dGe9wnQoTSQAlpjnqkZPiLEuX9bYuSHYaIiAwxJawiIoMs7Ib7NNTXYmmLtvFq3ev9vkaWL4vTyk7tdVmb57e8SK4vt9/nF0lFAxxBLyIiw5gSVhGRQTYhs4JoH6v3RmyEP298kJe2vNLv65w27hT+acIFPe7zct0rNAQb+n1ukVRjMOyft1+ywxARkSGmhFVEZJCle9JZWHI8fsffp/0tloeqHmFt87p+X2t2zuxee1mj9Dx0WGQ4sFhWNq3iyU1P8+vVd/NI1ePUB+uTHZaIiCSYqgSLiCTA2ePPxO/4eabmOSyWoBvscf+wG+b52he4vPKyfl2nMDAGjZKU0WJjWxWb26uJ2ige4+GF2r/xtUkXc1jhIckOTUREEkQJq4hIAhhjOG3cKZw4diH1wQaW71zOnzc+FHcZGouluq2m39dxjMPU7Kks27l8b0MWGRaiu4qNRW2UKFF+v/6PzMyZQYE/P8mRiYhIImhIsIhIAvkcH2XpY/lCybFcWXk5Tg8vu6VpJQO6xnnl5+Kh92VuREYii+XNhrf6d4y1rGlaw/M1L/Bq3Ws0h5sTFJ2IiOytvUpYjTHnGmOWGmNcY8yCz7X9wBizxhiz0hizcO/CFBEZ/ubnz6UioyJm0up3/JxQOrCXyomZEzik8CD8xre3IYoMOxEb6VdhsZZIKzctu4U7V/6Uh6se4/6ND3LV4u/ySt2riQtSREQGbG97WJcAZwFd1mQwxswCzgdmAycA/2lMH1a5FxEZ4f5t2uXk+/M710b1Gg8+4+PMcaczNbtywOf9+qRLuHjiRfhN3wo9iSSTiTPz2mDwO35mZc/oc9GygBNgQkZFn699z7p72dhaRdANEiVKyA0RtmEe2PgQq5vW9Pk8IiIyNPZqDqu1djkQa73B04EHrbVBYL0xZg1wIPDO3lxPRGS4K/AXcOfcW/lox2JWN60h25fNIWMOojBQuFfnNcZweNFhRHG5/7MHei3yJJJMFgvsSlCNj9L0UlqjrUzImMApZScxMWMC72z9B8/WPMfW0DYK/YVsDW2lNdraeexujnE4tI9Fl7aHtrO0cVnMueQhN8QzNc9xVfaVe/8ERURk0CSq6NI44N09Hm/ata0bY8ylwKUAFRV9/4ZURGS48jpeDihYwAEFC3rfuY9c67KueR35vjzyfXk0hLZ2+VBuMN0+6Iskm8USsmG8xstd8+7o0nZo4cEcWnhw5+Pa9i3cseIuWiMtu6oEe/E6Hr4z7WrSPel9ut6W9jq8jo9wNHbxs81tmwf+ZEREJCF6TViNMS8BpTGarrXWPhnvsBjbYn5SstbeA9wDsGDBAn2aEhHpp4+2L+a3639PxI1gjCHiRijPGM/m1mpCNkRJoJh2N0hjuDHZoYp0Y7FsbK2iIdjQ40iD0rQSfjLvDpbtXE5Ney1j/GOYmzsHr9P3794L/AVE3NjJKnQsEyUiIqml11d5a+1xAzjvJqB8j8fjgeoBnEdEROLY3FbNnz97kCU7l8ZsO23cKZwy9iQWbf+Ae9fdl4QIRfrG63hpDO/sdWi8Yxzm5M5mTu7sAV2nOK2Iioxy1rdswMXt0hZwAiwsPX5A5xURkcRJ1LI2TwHnG2MCxphJwFTgvQRdS0Rk1FnVtJofLf2PmMkq7J6P9yxRG+X9bR9oTquktIgboSSteEiudXnlZeT6cgk4AQAcDH7HxzHFRzEvd+6QxCAiIn23V3NYjTFnAr8CioBnjDGLrbULrbVLjTEPA8uACHC5tbtW+hYRkb1ireWetb8l5IZ63M+1loZgA06ciqwiqcBjPBxYsIAsb9aQXG9MoIAfz7uN97YtYvnOFWR6Mzis8FAqMsp7P1hERIbc3lYJfgJ4Ik7bLcAte3N+ERHprqa9hsbIzl73i9ooITdMc7h5CKKS0cpnfFw84Z8I2wh//OxP/T7eweGrk76SgMji8zk+Dis8hMP6WF1YRESSJ1FVgkVEJEGC0VCcMnZdRWyE65f+KOHxyOhVnl7OZZWXMi69jD9tuH9A5/AYT5/XXBURkdFHCauIyDBTml5CyPY8HLg3ud4cDio4kA0tG6gO1uLFy7y8uZxcdiKv1P2d52v/NkjRykg1Lq2M62f9gICnYy7oB9s/HNB5ytLGDmZYIiIywihhFREZZtY0r8VrvF3WWe2vgCeNCydeELPt9LJTlbBKj3K9udw054ddlpQZ6P24cOzxLN+5glxfDmXpZYMVooiIjBBKWEVEhpmd4aa9Tlgbgg1x22raa/Eb/1734srIFDABbpx9Q7f1T6dmVfLhjsX9OpfXeLl33e/wOT6iNkpRoJArKr9FWbp6XUVEpEOilrUREZFBFrVRnq/5Gw9ufJh2t32vzuU18b+vzPRmosLCEovf+Lhk8sXkB/K6tZ027tR+z0WN2igRG6Et2kbIDbG5rZqbl91GS6R1sEIWEZFhTgmriMgwYK3lF6t+zWObn2BnHyoE9+aIosPjtpWmlVDoH7PX15CRJ2TD3Lf+Dzxd/Uy3tkmZE7l8yr+Q5c0k4ATwO348xoPp4dsPG6N6WNiGeaP+zUGNW0REhi8lrCIiw8CKppWsaFoZd+1Vv+l7z1amJ5Ozxp3e4z5njz+zX/HJ6BF0Qzy5+Wkaw43d2ubnz+OX+/6M70y/iisrv8Uv9/0pF5R/qVvPq8d4cOJ8BAm5IZbsXJqQ2EVEZPhRwioiMgy8s/UfBN1gzDaP8XB08ZHcvd+vOKPsNLzGi8Hg4ODBA3SM8PUZH8cUHcWd824ly5cV91pbg9t4dNPjfY4tz9d9eKiMbFEb5cPtH8Vs8xgP07OnMTdvH7K8WSwcezxXVn6LqVmVZHoyKUsby1GFR/Q4fDjbG//+FBGR0UVFl0REhoGIG7/AkoNDcVoxGd4Mzhx/OmeMO40VTSvZ3FZNni+XeXlz8Tm+Pl3nlbpX+fNnDxK24T7tH3D8HFl4BM9veSFu76+MPC4um9tq+rz/3Lx9mJu3T+fjkBvira3vxNw34Pg5qujIvY5RRERGBvWwiogMsbZoG8Fo7N7SePbLn0/ACcRtn5Mzu/N3Ywwzc2ZwXMmxLCjYv8/J6sbWKh7Y+FCfk1WP8ZDry+XUcScxI3t6n46RkWNN05oBH+t3/Hxj0iX4HX+XOa4BJ8CBBQcwPXvaYIQoIiIjgHpYRUSGyNLGZfx544NU7+qZmpw1iYsmXMjEzAm9Hrtv/nwKA2PY0l7XZTkbv/EzP28uY9NL9zq+l7a83GNP7p48xsPsnJnMz5vP3+teZV3zur2+vgwvn7VuxLUujnE6f/bHAWMWUJJewjPVz7GhZQO5/lyOL/ki++fvizEqUy0iIh2Mtd0r9CXLggUL7KJFi5IdhojIoFvauIyfr/4lIbdr72XA8XP9rGspzxjf6zlaIq3872f38/62RRjjYDB8ofgYzh5/Zrc1MQfilmW3s6p5da/7+fARJYqLi9n1x8Xd6+vL8OJgOLzwMD7Y/hEt0RZyfbmcNPYEji85rt/JK0BzuJk2t50Cfz4e40lAxCIikgqMMR9Yaxf0eX8lrCIiifeDT66juj32nL/5eXO5etq/9flcwWiQ1mgr2d7sQUlUd/vjhj/xat3rcZNPg4m5DImMTgaDxzhEbLRzm9/xc1DBgXx98tf6fJ6atlp+t/73rG/ZgGMcfMbH6eNO5fiS49TTKiIyAvU3YdUcVhGRBGuJtLIlWBe3fWnj8n6dL+AJkO/PH9RkFeCLJV+Ie86A8StZlS4stkuyCh3FlN7d+g9q27f06Rw7Qju4adktrGleS8RGCLkhWqItPLrp8ZhrvYqIyOijhFVEJMGcXnqJUqUXqSy9jK9N/Ao+4+tcciTNCZDtzSZkVQF4NPEZHxVp5QM6NmzD3LP2Xra0x/+SZrcXal8k5Ia6fRkSckP8tebZfhcnExGRkUcJq4hIgqV70qnIiP3h32DYN2/eEEcU36GFh/Dz+XdxQcV5nDXuDL45+Rv4HK/6VkeZsA2zub16wMeva9nA9Ut+xPrmDT3u99GOxV2KiO3JwbCh9bMBxyAiIiODElYRkSFw0YQLO3stdzMY0jxpnD3+rCRFFVuWL4tji4/m9HGnMi5jHM2RlmSHJEkQJdr7TnFYLEE3yD3rftvjft4ellyydPT0iojI6KaEVURkCEzJmsx1M69hTs5sPMaDz/hYkL8fN86+npK04mSH1wP1rcrA1QcbqA/Wx20/ovAw/MYfs83nePu05JOIiIxsWodVRGSITMicwPdmfDvZYfRLUaCIDE8GIVdzWKX/PMZDW7Q9bvvRRUfyat1r1AfrCe+5vrDj55JJXx3Q8jgiIjKy6J1ARES6CEaDbAttI+JGcIzDRRO+jL+HoZvx7F7ixO/4MHQUlnJw8BgPHrTO5uhgGZtWGrc14Alww+xrOaF0Ibm+XNKcNGbnzOT707/Dfvn7DmGcIiKSqtTDKiIiALREWvifDfezaPsHOMbBwXBs8TGcPf5Mrqi8nIeqHqa6rQaDoThQTH2wjmiMNVv9jh+D4cxxp3Pi2IVsbK3ilS1/pz7YwITMCsDwfM0LQ/8ER7CO/1qmX/NOC3wF7IzsJMOTwc7Izn5fz+76E4/f8XPy2JPw9fJlR7onnXPKz+Kc8tSayy0iIqlBCauIiBBxI9y87Hbqgls61tbclYe8tOUVGkJbubzyX5iXtw8hN0zYDfGjpTfHTFYBCv1juHHODZ1FpioyyvnqpK90tq9v2cCLW14i6vY9uTIYrQPbAxcXv+PHR0dyGHJD+B0fwRhLxuxWnFbEz2b+GID/WHora1rWdtvHYChJK2Z7aAdBt2OJGb/xk+HN4Oiio3i29lkibhR3j3vBZ3w4xuGk0oWcWnbyYD9VEREZZZSwiogIH+1YzNbQ1o5kdQ8hG+Kj7YupbaulNL2UiBvmxqU3UxeMv8ZmTXstr2x5leK0YubmzsHrdH2rmZQ5kalZlaxqWk3YhvsYoZLV3oTcEBmeDL4y8UIaglsp8OXz2/X3xf2XW9m0Cmstxhi+Pf0qbl1+BzVtNUSJYjA4OHyp/ByOKzmW97Yt4s2GNwm7UQ4o2I8jig4n3ZPO/gX78vKWV6gP1lORXsHs3NlkeNMpzxjfrSq2iIjIQChhFRERPtz+UWcPWixLdy6jNL2UZ2qeoyG4tcdzWSyPVD2G1/Hidbx8d/rVTMqc2GWfq6Zdyf2fPcBbDe8QtV176D7PazrequKt1yn/x7UuASfAqWUnE3Ej/Hb974mX7DvGdP6e6c3g5jk/YlXzalY1rSbNk8YB+QvI8+cCcGjhwRxaeHC3c1RklPO1SRcn5LmIiIiAElYREeH/ksJYHGPwmI4iSW80vNWneZIRIkTcCLhw6/I7mJBRwY7wDsrTx3Ny2UlUZk3ha5Mu5ssV51PbvoXX6l/nrYZ3aHe7VpQt8BXwhZJjeKr6ryMqYTUYvMbbjx7mvonaKNtD2wHwOl6mZU9lRdPKmNefm7sPZo+k1RjD9OxpTM+eNqgxiYiI7A1VCRYREQ4pPJiAE4jZ5lqX+XnzAQa0vE3IwdZg8AAAEM9JREFUDbG6eQ31wQY+3LGYO1bcxev1bxBxIzy66XFuXn4bbza8TcRGmJe7D9+ZdhVfKDoGn/HRGm3lrzXP9tj7O5x48PDNyd/gdwv+m4PGHNBZPbm/4h3nMQ5j08d2Pr5wwgUEnECX/Q2GNCfAeeVfGtC1RUREhpJ6WEVEhJnZM5iRPY3lTSu7JKUBx88JpQs7h4ZWZk3h08Yle3WtkBvifzbcz0fbF7Nk57Iu11vSuIy1zesJ2RBhGx70HshkMRh8xke2L5tPGz+lPGM8p5edxqJtH3brVd6ba2T7spmRPb1zW0VGOT+cfR2PVj3Gp41LMQbm5c7lnPKzKU0rGZTrioiIJJKxNnUKWSxYsMAuWrQo2WGIiIxKURvlxdqXeWHLizSHmyhKK+b0slM5aMwBnft8vOMTfr7qV93mnDo4OMbp87DdgOMn4kb7tQzLcPX54b8ODl7Hy2VTvklJWjG/W/d71ras69c5i/yFNEebsbZjbq/XeMjyZnPNzO9SFChKxNMQEREZFMaYD6y1C/q8vxJWERHpi02tm7h5+e2EoqEuiabBcHrZqTxX+0Kfh+568IDpSJJHqzQnjZ/O/zEZnnSC0SCv1r/Gw1WP9ZrEGwwHFhzA1ydfwkfbF7MjvIOy9LHMzpmFYzTTR0REUlt/E1YNCRYRkT65e+29tEXbum33OT7yfLlcWfktfrnmN32a52qMwcGMih7WeNrddr714ZX4HT+HjzmUD3cs7tO/h8/xcULp8fgdX5febxERkZFIX8WKiEiv6trrqQtuidkWckO8VPcK++TN4bvTruroPe2B13iZmDlBK6vuEnJDvFb/Bo3hxh738xkvPuPlvPJzmZw1aYiiExERSS71sIqISK9aIi14jBeIXQSpJdIKQFl6WcdSKT1ko/Pz5vL1yZfwWt0bPLb5iS49sn7Hj8EMm6rAXrzMy5/LwQUH8pfNT1HdXoNjHKy1+Bwf6U46LdGWXotH9dazmuXN4rSyUzio4ADy/HmD+RRERERSmhJWERHp1dj0UqJxCioZDJVZUwDI9mUzPXsay3eu6FaYyWu8nDnudE4pOwmAE8Yez9j0Uv6y+Slq2mvJ8WZzfOlxfLD9I5btXD6gOHN9ObRF2gnZ/i+/E4vBYONk3z7j49qZ1zApayIAB445gJ3hJj7Y/gFt0XYqs6ZQGijh/316LeHowKsde/BwcMGBLCz94oDPMVDWWuqC9YClOFDcZd1WERGRoaCEVUREepXmSeOYoqP5e/1r3eao+hwfp407pfPx1ydfwk1Lb6Y12tbZUxpwAszIns6JYxd2OXZe3lzm5c3tsm2Mfwwrdq7slvD2GqMT4LY5N/PE5id5se7lfh27Jy9eitKKmJ83l5Ab4vX6N7v1kPodP18af05nsrpbji+bY4qP7rLt2pnX8Os1/8nW0Daw9JhMx0qQvY6XE8YeP+DnM1CLt3/MHz/7E82RFgCyPJlcNPGf2C9//pDHIiIio5cSVhER6ZPzKs4lbMO8Uf8WXuPFYvE6Xi6d/M9UZJR37lfgz+eOubfy9tZ3WbxjMQEnjSMKD2N2bt+q2E7MmBi3V7MnLpa3tr7DWePP4NX61we0hqvXeDl57ImcWnYyPscHwKycmfx+w/8QcSMYY3Cty6llJ3NcybF9Ouf4jHHcPvcWNrVupra9lrvX/DeRGEOA/cbPtOyprGxahdfx4lqXbG8Wl1V+c8iXqlnauIzfrP2vLl9ObHND3L32v/i3qVcyJ3f2kMYjIiKjl5a1ERGRfmkKN7GuZT0BJ8DU7Eo8puciS/21fOcKfr7qV7S77f0+9uCCg7is8lJWNa3mJyt/TtAN9jn5dXD43vSrmZU7q1uba13Wt2wgYiNMzJhAwBPod2y7PVz1KC9ueblLMug1Xsall3HDrGsJukE2tlaR6c2kPH18UobhXr/kRja2bozZVp4+npv3uXGIIxIRkZGiv8vaqEqwiIj0S7Yvm3l5c5mRM33Qk1WAXF9uj8OBDbETOAeH/F0FiaZlT+WX+/6U40uOi7v/57m4/GL1b2iPdk+UHeMwJWsy07On9ZqsVrdV83bDu3yy41Mibvd5v+eOP5svV5xPgb8A6BgufWzxMVw78xq8jpdMbyYzc2ZQkVGelGQ1aqNUtVbFbd/Utjnm8xIREUkEDQkWEZGUUpY+luJAEZvbqrv1jvqND4slHKMAlMd4OLLoiM7HAU+AffPn81r9G33urbVY/rHtfY7a4zx91RZt4xerfs2a5rU4xsFg8BgPV0y9jFk5Mzv3M8ZwTPFRHFN8FFEbxcFJqWJGBoNjDNE4I7A62vV9t4iIDA2944iISMq5ovJbZHoz8Dv+zm0BJ8BBYw7k8spv4Xf8eE3Hd64ODn7Hz1njzyDHl01zpLnzmKlZlf1KBoNukC3tsdeb3c1ay/vbFnHT0lu56qPvcueKu1jauIxfr76b1c1rCNswQTdIu9tOS7SFn636JfXB+pjn8hhPSiWr0NGbPD9vfsyeaYNhft48JawiIjJk1MMqIiIpZ2x6KXfOvZ03G95iSeNSsr1ZHFl0BNOzp2GM4dZ9buKlLa+woeUzigNFTMmazItbXubRTY93HJ9WykUTLmRGznQumXgx966/r1t141gCToDiXgoc/emz+3mz4S2Cu863PbydVU1rcG2UaIyhzFE3yt9qX+LCCRcM4F8iOS6oOI8VO1fSHm3vXCPWg4c0T4ALKs5LcnQiIjKaqOiSiIgMa6uaVvPjlT/tlpD6HT/fn/4dpmZXsqppNX/Z/BTrWzYQcAI0RZqIxBhWHHAC/GLfn5DuSY95rc9aNnLz8tv6lPzuaXLmJH44+7p+HZNs20Lbeab6Wd7fvghr4YCC/Tml7KTOubciIiID0d+iS+phFRGRlBN2w9QF60lz0hgT6DlBemDjQzETyJAb4sGqh7l+1r8zLXsq35/xnc62FTtX8tNVvwAsQTdEwPEDhqun/WvcZBXg7a3vEHb7v1xOvj+/38ckW4E/n4smXshFEy9MdigiIjKKKWEVEZGUYa3l6epneKbmOaCjYu3YtFK+MaXrWq+7RW2U9S0b4p5vbfM6XOt2m3M5I2c6P59/F+9ue4/atlpK00s5ZMxBPSarAK3Rtn6vERtw/H1es1VERES6UsIqIiIp44nNT/J87Qud80MBNrZVccuy27l1n5sYExjTZX+z60+8JLKnJW0yvBkcW3x0v+KbkzOL97a+R7sb7NbmM16McbDWErYdvbB+x8/hhYczM3tGv64jIiIiHVTmT0REUkIwGuS5zyWru4XdMM/WPt9tu2McZufOinvOfXLnDGpF2/3z9yPLm43zubdPr/EyMXMid869jZPHnsjM7BkcXHAQ351+NRdN+HLKVQIWEREZLtTDKiIiKWFjaxUePDHbokT5dMcSmNC97csV53Hj0jUE3WBnT6vBEHACnF/xpUGN0et4uX7Wv/Nfa+9hTfNavI6XiBtmXt48vj75a6R70jlz/OmDek0REZHRTAmriIikBL/jw42xLExnuycQc3tZehk3zr6eRzc9wceNnwAwP28uZ48/i9K0kkGPM8+fyzUzv8e20HZ2hHZQFCgk25c96NcRERERJawiIpIiyjPKSfekE4wxP9Tv+Dmq8Ii4x5aml3LF1MsSGV43Bf58CoZh9V8REZHhRHNYRUQkJTjG4RuTL8Hv+LsUS/IZHyWBEo4qjp+wjlbWWpojzQSj3ZN8ERGRkUA9rCIikjLm5M7mupk/4Mnqp1nVtJo0T4Cji47kiyXH4Xf8yQ4vpbzb8A8e3vQYjeEdWGBm9gy+MvFCShIwDFpERCRZjLX9W08ukRYsWGAXLVqU7DBERERS2qt1r3P/xgcI7VFR2WBI96Rzyz43aaiyiIikLGPMB9baBX3dX0OCRUREhpGIG+Ghqke6JKsAFkvQDfJczQtJikxERGTwKWEVEREZRja3VePa2NWUozbKB9s/HOKIREREEkcJq4iIyDDiGKdzvdlYPEZv7SIiMnLoXU1ERGQYGZdeRlqcNWm9xsshYw4e4ohEREQSRwmriIjIMOIYh69OvBi/4+uy3YOHbG82x5cel6TIREREBp8SVhERkWFmv/z5fGfa1UzNqsRjPKQ5aRxRdDg3zbmBLG9WssMTEREZNFqHVUREZBiakTOd62b9INlhiIiIJJR6WEVERERERCQlKWEVERERERGRlKSEVURERERERFKSElYRERERERFJSUpYRUREREREJCUpYRUREREREZGUpIRVREREREREUpISVhEREREREUlJSlhFREREREQkJSlhFRERERERkZSkhFVERERERERSkhJWERERERERSUlKWEVERERERCQlKWEVERERERGRlKSEVURERERERFKSElYRERERERFJSUpYRUREREREJCUpYRUREREREZGUpIRVREREREREUpISVhEREREREUlJxlqb7Bg6GWPqgc+SHYfslUKgIdlByKih+02Gku43GSq612Qo6X6ToVQIZFpri/p6QEolrDL8GWMWWWsXJDsOGR10v8lQ0v0mQ0X3mgwl3W8ylAZyv2lIsIiIiIiIiKQkJawiIiIiIiKSkpSwymC7J9kByKii+02Gku43GSq612Qo6X6TodTv+01zWEVERERERCQlqYdVREREREREUpISVhEREREREUlJSlhlUBhjzjXGLDXGuMaYBZ9r+4ExZo0xZqUxZmGyYpSRyRjzI2PMZmPM4l1/T0p2TDKyGGNO2PX6tcYYc02y45GRzRizwRjz6a7Xs0XJjkdGFmPMfcaYOmPMkj22FRhjXjTGrN71Mz+ZMcrIEed+6/fnNiWsMliWAGcBr++50RgzCzgfmA2cAPynMcYz9OHJCPcza+38XX+fTXYwMnLser36DXAiMAu4YNfrmkgiHbPr9UxrY8pg+wMdn8f2dA3wsrV2KvDyrscig+EPdL/foJ+f25SwyqCw1i631q6M0XQ68KC1NmitXQ+sAQ4c2uhERAbsQGCNtXadtTYEPEjH65qIyLBjrX0d2Pa5zacDf9z1+x+BM4Y0KBmx4txv/aaEVRJtHFC1x+NNu7aJDKYrjDGf7Bp6oqFMMpj0GiZDzQJ/M8Z8YIy5NNnByKhQYq2tAdj1szjJ8cjI16/PbUpYpc+MMS8ZY5bE+NtTb4OJsU1rKUm/9HLv3Q1MAeYDNcBPkhqsjDR6DZOhdpi1dj86hqFfbow5MtkBiYgMon5/bvMmOiIZOay1xw3gsE1A+R6PxwPVgxORjBZ9vfeMMfcCf01wODK66DVMhpS1tnrXzzpjzBN0DEt/veejRPbKFmPMWGttjTFmLFCX7IBk5LLWbtn9e18/t6mHVRLtKeB8Y0zAGDMJmAq8l+SYZATZ9ea625l0FAATGSzvA1ONMZOMMX46isg9leSYZIQyxmQaY7J3/w4cj17TJPGeAi7e9fvFwJNJjEVGuIF8blMPqwwKY8yZwK+AIuAZY8xia+1Ca+1SY8zDwDIgAlxurY0mM1YZce40xsynY5jmBuCbyQ1HRhJrbcQYcwXwAuAB7rPWLk1yWDJylQBPGGOg4zPan621zyc3JBlJjDEPAEcDhcaYTcAPgduBh40x/wxsBM5NXoQyksS5347u7+c2Y62m4oiIiIiIiEjq0ZBgERERERERSUlKWEVERERERCQlKWEVERERERGRlKSEVURERERERFKSElYRERERERFJSUpYRUREREREJCUpYRUREREREZGU9P8B41HesC0GjMkAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "#plot the data\n", + "fig = plt.figure(figsize=(16, 10))\n", + "plt.scatter(data[:, 0], data[:, 1], c=labels, s=50, cmap='viridis')\n", + "\n", + "#plot the sklearn kmeans centers with blue filled circles\n", + "centers_sk = kmeans_sk.cluster_centers_\n", + "plt.scatter(centers_sk[:,0], centers_sk[:,1], c='blue', s=100, alpha=.5)\n", + "\n", + "#plot the cuml kmeans centers with red circle outlines\n", + "centers_cuml = kmeans_cuml.cluster_centers_\n", + "plt.scatter(centers_cuml['0'], centers_cuml['1'], facecolors = 'none', edgecolors='red', s=100)\n", + "\n", + "plt.title('cuml and sklearn kmeans clustering')\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 17.1 s, sys: 558 ms, total: 17.6 s\n", + "Wall time: 17.6 s\n" + ] + } + ], + "source": [ + "%%time\n", + "#get cluster score of cuml and sklearn kmeans\n", + "cuml_score = adjusted_rand_score(labels, kmeans_cuml.labels_)\n", + "sk_score = adjusted_rand_score(labels, kmeans_sk.labels_)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "compare kmeans: cuml vs sklearn labels_ are equal\n" + ] + } + ], + "source": [ + "# check if the kmeans scores are equal up to a set threshold\n", + "threshold = 1e-5\n", + "\n", + "passed = (cuml_score - sk_score) < threshold\n", + "message = 'compare kmeans: cuml vs sklearn labels_ are ' + ('equal' if passed else 'NOT equal')\n", + "print(message)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cuml/knn_demo.ipynb b/cuml/knn_demo.ipynb index 8bba360f..037b01ca 100644 --- a/cuml/knn_demo.ipynb +++ b/cuml/knn_demo.ipynb @@ -1,8 +1,28 @@ { "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## KNN\n", + "K NearestNeighbors is a unsupervised algorithm where if one wants to find the “closest” datapoint(s) to new unseen data, one can calculate a suitable “distance” between each and every point, and return the top K datapoints which have the smallest distance to it.\n", + "\n", + "cuML’s KNN expects a cuDF DataFrame or a Numpy Array (where automatic chunking will be done in to a Numpy Array in a future release), and fits a special data structure first to approximate the distance calculations, allowing our querying times to be O(plogn) and not the brute force O(np) [where p = no(features)]:\n", + "\n", + "The KNN function accepts the following parameters:\n", + "1. n_neighbors: int (default = 5). The top K closest datapoints you want the algorithm to return. If this number is large, then expect the algorithm to run slower.\n", + "1. should_downcast:bool (default = False). Currently only single precision is supported in the underlying undex. Setting this to true will allow single-precision input arrays to be automatically downcasted to single precision. Default = False.\n", + "\n", + "The methods that can be used with KNN are:\n", + "1. fit: Fit GPU index for performing nearest neighbor queries.\n", + "1. kneighbors: Query the GPU index for the k nearest neighbors of row vectors in X.\n", + "\n", + "The model can take array-like objects, either in host as NumPy arrays or in device (as Numba or _cuda_array_interface_compliant), as well as cuDF DataFrames. In order to convert your dataset to cudf format please read the cudf documentation on https://rapidsai.github.io/projects/cudf/en/latest/. For additional information on the K NearestNeighbors model please refer to the documentation on https://rapidsai.github.io/projects/cuml/en/latest/api.html#nearest-neighbors" + ] + }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ @@ -24,11 +44,13 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ + "# check if the mortgage dataset is present and then extract the data from it, else just create a random dataset for clustering \n", "import gzip\n", + "# change the path of the mortgage dataset if you have saved it in a different directory\n", "def load_data(nrows, ncols, cached = 'data/mortgage.npy.gz',source='mortgage'):\n", " if os.path.exists(cached) and source=='mortgage':\n", " print('use mortgage data')\n", @@ -36,6 +58,7 @@ " X = np.load(f)\n", " X = X[np.random.randint(0,X.shape[0]-1,nrows),:ncols]\n", " else:\n", + " # create a random dataset\n", " print('use random data')\n", " X = np.random.random((nrows,ncols)).astype('float32')\n", " df = pd.DataFrame({'fea%d'%i:X[:,i] for i in range(X.shape[1])}).fillna(0)\n", @@ -44,30 +67,37 @@ }, { "cell_type": "code", - "execution_count": 55, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import mean_squared_error\n", - "def array_equal(a,b,threshold=1e-4,with_sign=True,metric='mse'):\n", + "# this function checks if the results obtained from two different methods (sklearn and cuml) are the same\n", + "def array_equal(a,b,threshold=1e-3,with_sign=True,metric='mse'):\n", " a = to_nparray(a)\n", " b = to_nparray(b)\n", " if with_sign == False:\n", " a,b = np.abs(a),np.abs(b)\n", " if metric=='mse':\n", " error = mean_squared_error(a,b)\n", - " else:\n", + " res = errorthreshold]) == 0\n", + " elif metric == 'acc':\n", " error = np.sum(a!=b)/(a.shape[0]*a.shape[1])\n", - " res = error1]) / (c.shape[0]*c.shape[1])\n", " return c 1 implements a higher negative value to the samples (default = 1.0)\n", + "11.\tnegative_sample_rate: the rate at which the negative samples should be selected per positive sample during the optimization process (default = 5)\n", + "12.\ttransform_queue_size: embedding new points using a trained model_ will control how aggressively to search for nearest neighbors (default = 4.0)\n", + "13.\tverbose: bool (default False)\n", + "\n", + "The cuml implemetation of the UMAP model has the following functions that one can run:\n", + "1.\tfit: it fits the dataset into an embedded space\n", + "2.\tfit_transform: it fits the dataset into an embedded space and returns the transformed output\n", + "3.\ttransform: it transforms the dataset into an existing embedded space and returns the low dimensional output\n", + "\n", + "The model can take array-like objects, either in host as NumPy arrays or in device (as Numba or _cuda_array_interface_compliant), as well as cuDF DataFrames. In order to convert your dataset to cudf format please read the cudf documentation on https://rapidsai.github.io/projects/cudf/en/latest/. For additional information on the UMAP model please refer to the documentation on https://rapidsai.github.io/projects/cuml/en/0.6.0/api.html#cuml.UMAP" + ] + }, { "cell_type": "code", - "execution_count": 48, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ @@ -20,55 +48,81 @@ "from cuml.manifold.umap import UMAP" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Running cuml's UMAP model on blobs dataset" + ] + }, { "cell_type": "code", - "execution_count": 58, + "execution_count": 12, "metadata": {}, "outputs": [], "source": [ + "# create a blobs dataset with 500 samples and 10 features each\n", "data, labels = datasets.make_blobs(\n", " n_samples=500, n_features=10, centers=5)" ] }, { "cell_type": "code", - "execution_count": 59, + "execution_count": 13, "metadata": {}, "outputs": [], "source": [ + "# using the cuml UMAP algorithm to reduce the features of the dataset and store\n", "embedding = UMAP().fit_transform(data)" ] }, { "cell_type": "code", - "execution_count": 60, + "execution_count": 14, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1.0\n" + ] + } + ], "source": [ + "# calculate the score of the results obtained using cuml's algorithm and sklearn kmeans\n", "score = adjusted_rand_score(labels,\n", " KMeans(5).fit_predict(embedding))\n", - "\n", - "assert score == 1.0" + "print(score) # should equal 1.0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Running cuml's UMAP model on iris dataset" ] }, { "cell_type": "code", - "execution_count": 61, + "execution_count": 15, "metadata": {}, "outputs": [], "source": [ + "# load the iris dataset from sklearn and extract the required information\n", "iris = datasets.load_iris()\n", "data = iris.data" ] }, { "cell_type": "code", - "execution_count": 62, + "execution_count": 16, "metadata": { "scrolled": true }, "outputs": [], "source": [ + "# define the cuml UMAP model and use fit_transform function to obtain the low dimensional output of the input dataset\n", "embedding = UMAP(\n", " n_neighbors=10, min_dist=0.01, init=\"random\"\n", ").fit_transform(data)" @@ -76,46 +130,194 @@ }, { "cell_type": "code", - "execution_count": 63, + "execution_count": 17, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.9741363073110285\n" + ] + } + ], "source": [ + "# calculate the trust worthiness of the results obtaind from the cuml UMAP\n", "trust = trustworthiness(iris.data, embedding, 10)\n", - "assert trust >= 0.95" + "print(trust)" ] }, { "cell_type": "code", - "execution_count": 64, + "execution_count": 18, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[5.1 3.5 1.4 0.2]\n", + " [4.9 3. 1.4 0.2]\n", + " [4.6 3.1 1.5 0.2]\n", + " [5. 3.6 1.4 0.2]\n", + " [4.6 3.4 1.4 0.3]\n", + " [5. 3.4 1.5 0.2]\n", + " [4.4 2.9 1.4 0.2]\n", + " [4.9 3.1 1.5 0.1]\n", + " [5.4 3.7 1.5 0.2]\n", + " [4.8 3.4 1.6 0.2]\n", + " [4.8 3. 1.4 0.1]\n", + " [4.3 3. 1.1 0.1]\n", + " [5.7 4.4 1.5 0.4]\n", + " [5.4 3.9 1.3 0.4]\n", + " [5.1 3.5 1.4 0.3]\n", + " [5.7 3.8 1.7 0.3]\n", + " [5.1 3.8 1.5 0.3]\n", + " [5.4 3.4 1.7 0.2]\n", + " [5.1 3.3 1.7 0.5]\n", + " [4.8 3.4 1.9 0.2]\n", + " [5. 3. 1.6 0.2]\n", + " [5. 3.4 1.6 0.4]\n", + " [5.2 3.4 1.4 0.2]\n", + " [4.7 3.2 1.6 0.2]\n", + " [4.8 3.1 1.6 0.2]\n", + " [5.2 4.1 1.5 0.1]\n", + " [5.5 4.2 1.4 0.2]\n", + " [5. 3.2 1.2 0.2]\n", + " [4.9 3.6 1.4 0.1]\n", + " [4.4 3. 1.3 0.2]\n", + " [5. 3.5 1.3 0.3]\n", + " [4.5 2.3 1.3 0.3]\n", + " [4.4 3.2 1.3 0.2]\n", + " [5. 3.5 1.6 0.6]\n", + " [5.1 3.8 1.9 0.4]\n", + " [4.8 3. 1.4 0.3]\n", + " [5.1 3.8 1.6 0.2]\n", + " [4.6 3.2 1.4 0.2]\n", + " [5.3 3.7 1.5 0.2]\n", + " [5. 3.3 1.4 0.2]\n", + " [7. 3.2 4.7 1.4]\n", + " [6.4 3.2 4.5 1.5]\n", + " [6.9 3.1 4.9 1.5]\n", + " [5.5 2.3 4. 1.3]\n", + " [6.5 2.8 4.6 1.5]\n", + " [5.7 2.8 4.5 1.3]\n", + " [6.3 3.3 4.7 1.6]\n", + " [4.9 2.4 3.3 1. ]\n", + " [6.6 2.9 4.6 1.3]\n", + " [5.2 2.7 3.9 1.4]\n", + " [5. 2. 3.5 1. ]\n", + " [5.9 3. 4.2 1.5]\n", + " [6.1 2.9 4.7 1.4]\n", + " [5.6 2.9 3.6 1.3]\n", + " [6.7 3.1 4.4 1.4]\n", + " [5.6 3. 4.5 1.5]\n", + " [5.8 2.7 4.1 1. ]\n", + " [6.2 2.2 4.5 1.5]\n", + " [5.9 3.2 4.8 1.8]\n", + " [6.1 2.8 4. 1.3]\n", + " [6.3 2.5 4.9 1.5]\n", + " [6.1 2.8 4.7 1.2]\n", + " [6.4 2.9 4.3 1.3]\n", + " [6.6 3. 4.4 1.4]\n", + " [6.8 2.8 4.8 1.4]\n", + " [6. 2.9 4.5 1.5]\n", + " [5.7 2.6 3.5 1. ]\n", + " [5.5 2.4 3.8 1.1]\n", + " [5.5 2.4 3.7 1. ]\n", + " [6. 2.7 5.1 1.6]\n", + " [5.4 3. 4.5 1.5]\n", + " [6.7 3.1 4.7 1.5]\n", + " [6.3 2.3 4.4 1.3]\n", + " [5.6 3. 4.1 1.3]\n", + " [5.5 2.5 4. 1.3]\n", + " [6.1 3. 4.6 1.4]\n", + " [5.8 2.6 4. 1.2]\n", + " [5. 2.3 3.3 1. ]\n", + " [5.6 2.7 4.2 1.3]\n", + " [5.7 3. 4.2 1.2]\n", + " [6.2 2.9 4.3 1.3]\n", + " [5.7 2.8 4.1 1.3]\n", + " [6.3 3.3 6. 2.5]\n", + " [7.1 3. 5.9 2.1]\n", + " [6.3 2.9 5.6 1.8]\n", + " [6.5 3. 5.8 2.2]\n", + " [4.9 2.5 4.5 1.7]\n", + " [6.7 2.5 5.8 1.8]\n", + " [7.2 3.6 6.1 2.5]\n", + " [6.4 2.7 5.3 1.9]\n", + " [6.8 3. 5.5 2.1]\n", + " [5.7 2.5 5. 2. ]\n", + " [5.8 2.8 5.1 2.4]\n", + " [6.4 3.2 5.3 2.3]\n", + " [6.5 3. 5.5 1.8]\n", + " [7.7 3.8 6.7 2.2]\n", + " [7.7 2.6 6.9 2.3]\n", + " [6.9 3.2 5.7 2.3]\n", + " [5.6 2.8 4.9 2. ]\n", + " [7.7 2.8 6.7 2. ]\n", + " [6.3 2.7 4.9 1.8]\n", + " [6.7 3.3 5.7 2.1]\n", + " [7.2 3.2 6. 1.8]\n", + " [6.2 2.8 4.8 1.8]\n", + " [6.1 3. 4.9 1.8]\n", + " [6.4 2.8 5.6 2.1]\n", + " [7.2 3. 5.8 1.6]\n", + " [7.4 2.8 6.1 1.9]\n", + " [6.4 2.8 5.6 2.2]\n", + " [6.1 2.6 5.6 1.4]\n", + " [7.7 3. 6.1 2.3]\n", + " [6.4 3.1 5.5 1.8]\n", + " [6. 3. 4.8 1.8]\n", + " [5.8 2.7 5.1 1.9]\n", + " [6.7 3. 5.2 2.3]\n", + " [6.5 3. 5.2 2. ]\n", + " [5.9 3. 5.1 1.8]]\n" + ] + } + ], "source": [ + "# create a selection variable which will have 75% True and 25% False values. The size of the selection variable is 150\n", "iris_selection = np.random.choice(\n", " [True, False], 150, replace=True, p=[0.75, 0.25])\n", - "data = iris.data[iris_selection]" + "# create an iris dataset using the selection variable\n", + "data = iris.data[iris_selection]\n", + "print(data)" ] }, { "cell_type": "code", - "execution_count": 65, + "execution_count": 21, "metadata": {}, "outputs": [], "source": [ - "fitter = UMAP(n_neighbors=10, min_dist=0.01, verbose=True)\n", + "# create a cuml UMAP model \n", + "fitter = UMAP(n_neighbors=10, min_dist=0.01, verbose=False)\n", + "# fit the data created the selection variable to the cuml UMAP model created (fitter)\n", "fitter.fit(data)\n", - "\n", + "# create a new iris dataset by inverting the values of the selection variable (ie. 75% False and 25% True values) \n", "new_data = iris.data[~iris_selection]\n", + "# transform the new data using the previously created embedded space\n", "embedding = fitter.transform(new_data)" ] }, { "cell_type": "code", - "execution_count": 66, + "execution_count": 22, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.9513419913419914\n" + ] + } + ], "source": [ + "# calculate the trustworthiness score for the new data created (new_data)\n", "trust = trustworthiness(new_data, embedding, 10)\n", - "assert trust >= 0.90" + "print(trust)" ] }, { @@ -128,7 +330,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python (cuml3)", + "display_name": "Python 3", "language": "python", "name": "python3" }, @@ -142,7 +344,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.6" + "version": "3.7.3" } }, "nbformat": 4, diff --git a/cuml/umap_demo_graphed.ipynb b/cuml/umap_demo_graphed.ipynb new file mode 100644 index 00000000..1c8ccf0b --- /dev/null +++ b/cuml/umap_demo_graphed.ipynb @@ -0,0 +1,329 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# UMAP Demo with Graphs\n", + "\n", + "[UMAP](https://umap-learn.readthedocs.io/en/latest/) is a powerful dimensionality reduction tool which NVIDIA recently ported to GPUs with a python interface. In this notebook we will demostrate basic usage, plotting, and timing of the unsupervised CUDA (GPU) version of UMAP. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Imports and Set Up" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import pandas as pd\n", + "import numpy as np\n", + "\n", + "# libraries for scoring/clustering\n", + "from sklearn.manifold.t_sne import trustworthiness\n", + "\n", + "# GPU UMAP\n", + "import cudf\n", + "from cuml.manifold.umap import UMAP as cumlUMAP\n", + "\n", + "# plotting\n", + "import seaborn as sns\n", + "import matplotlib as mpl\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "sns.set(style='white', rc={'figure.figsize':(25, 12.5)})" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# hide warnings\n", + "import warnings\n", + "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Sanity Checks\n", + "\n", + "We are going to work with the [fashion mnist](https://github.com/zalandoresearch/fashion-mnist) data set. This is a dataset consisting of 70,000 28x28 grayscale images of clothing. It should already be in the `data/fashion` folder, but let's do a sanity check!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if not os.path.exists('data/fashion'):\n", + " print(\"error, data is missing!\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's make sure we have our RAPIDS compliant GPU. It must be Pascal or higher! You can also use this to define which GPU RAPIDS should use (advanced feature not covered here)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Helper Functions" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# https://github.com/zalandoresearch/fashion-mnist/blob/master/utils/mnist_reader.py\n", + "def load_mnist(path, kind='train'):\n", + " import os\n", + " import gzip\n", + " import numpy as np\n", + "\n", + " \"\"\"Load MNIST data from `path`\"\"\"\n", + " labels_path = os.path.join(path,\n", + " '%s-labels-idx1-ubyte.gz'\n", + " % kind)\n", + " images_path = os.path.join(path,\n", + " '%s-images-idx3-ubyte.gz'\n", + " % kind)\n", + "\n", + " with gzip.open(labels_path, 'rb') as lbpath:\n", + " labels = np.frombuffer(lbpath.read(), dtype=np.uint8,\n", + " offset=8)\n", + "\n", + " with gzip.open(images_path, 'rb') as imgpath:\n", + " images = np.frombuffer(imgpath.read(), dtype=np.uint8,\n", + " offset=16).reshape(len(labels), 784)\n", + "\n", + " return images, labels" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Training" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "train, train_labels = load_mnist('data/fashion', kind='train')\n", + "test, test_labels = load_mnist('data/fashion', kind='t10k')\n", + "data = np.array(np.vstack([train, test]), dtype=np.float64) / 255.0\n", + "target = np.array(np.hstack([train_labels, test_labels]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are 60000 training images and 10000 test images" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "f\"Train shape: {train.shape} and Test Shape: {test.shape}\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "train[0].shape" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As mentioned previously, each row in the train matrix is an image" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# display a Nike? sneaker\n", + "pixels = train[0].reshape((28, 28))\n", + "plt.imshow(pixels, cmap='gray')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There is cost with moving data between host memory and device memory (GPU memory) and we will include that core when comparing speeds" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "record_data = (('fea%d'%i, data[:,i]) for i in range(data.shape[1]))\n", + "gdf = cudf.DataFrame(record_data)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`gdf` is a GPU backed dataframe -- all the data is stored in the device memory of the GPU. With the data converted, we can apply the `cumlUMAP` the same inputs as we do for the standard UMAP. Additionally, it should be noted that within cuml, [FAISS] https://github.com/facebookresearch/faiss) is used for extremely fast kNN and it's limited to single precision. `cumlUMAP` will automatically downcast to `float32` when needed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%timeit\n", + "g_embedding = cumlUMAP(n_neighbors=5, init=\"spectral\").fit_transform(gdf)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Visualization\n", + "\n", + "OK, now let's plot the output of the embeddings so that we can see the seperation of the neighborhoods. Let's start by creating the classes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "classes = [\n", + " 'T-shirt/top',\n", + " 'Trouser',\n", + " 'Pullover',\n", + " 'Dress',\n", + " 'Coat',\n", + " 'Sandal',\n", + " 'Shirt',\n", + " 'Sneaker',\n", + " 'Bag',\n", + " 'Ankle boot']" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "#Needs to be redone because of timeit function sometimes loses our g_embedding variable\n", + "g_embedding = cumlUMAP(n_neighbors=5, init=\"spectral\").fit_transform(gdf)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Just as the original author of UMAP, Leland McInnes, states in the [UMAP docs](https://umap-learn.readthedocs.io/en/latest/supervised.html), we can plot the results and show the separation between the various classes defined above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "g_embedding_numpy = g_embedding.to_pandas().values #it is necessary to convert to numpy array to do the visual mapping\n", + "\n", + "fig, ax = plt.subplots(1, figsize=(14, 10))\n", + "plt.scatter(g_embedding_numpy[:,1], g_embedding_numpy[:,0], s=0.3, c=target, cmap='Spectral', alpha=1.0)\n", + "plt.setp(ax, xticks=[], yticks=[])\n", + "cbar = plt.colorbar(boundaries=np.arange(11)-0.5)\n", + "cbar.set_ticks(np.arange(10))\n", + "cbar.set_ticklabels(classes)\n", + "plt.title('Fashion MNIST Embedded via cumlUMAP');" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Additionally, we can also quanititaviely compare the perfomance of `cumlUMAP` (GPU UMAP) to the reference/original implementation (CPU UMAP) using the [trustworthiness score](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/t_sne.py#L395). From the docstring:\n", + "\n", + "> Trustworthiness expresses to what extent the local structure is retained. The trustworthiness is within [0, 1].\n", + "\n", + "\n", + "Like `t-SNE`, UMAP tries to capture both global and local structure and thus, we can apply the `trustworthiness` of the `g_embedding` data against the original input. With a higher score we are demonstrating that the algorithm does a better and better job of local structure retention. As [Corey Nolet](https://github.com/cjnolet) notes:\n", + "> Algorithms like UMAP aim to preserve local neighborhood structure and so measuring this property (trustworthiness) measures the algorithm's performance." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Scoring ~97% shows the GPU implementation is comparable to the original CPU implementation and the training time was ~9.5X faster" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cuml/umap_supervised_demo.ipynb b/cuml/umap_supervised_demo.ipynb new file mode 100644 index 00000000..daa91bc4 --- /dev/null +++ b/cuml/umap_supervised_demo.ipynb @@ -0,0 +1,383 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# UMAP Supervised Demo\n", + "\n", + "[UMAP](https://umap-learn.readthedocs.io/en/latest/) is a powerful dimensionality reduction tool, which NVIDIA recently ported to GPUs with a Python interface that matches UMAP-learn. In this notebook we will demostrate basic usage, plotting, and timing comparisons between supervised and unsupervised implementations of the CUDA (GPU) version of UMAP" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Imports and Set Up" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import pandas as pd\n", + "import numpy as np\n", + "\n", + "# libraries for scoring/clustering\n", + "from sklearn.manifold.t_sne import trustworthiness\n", + "\n", + "# GPU UMAP\n", + "import cudf\n", + "from cuml.manifold.umap import UMAP as cumlUMAP\n", + "\n", + "# plotting\n", + "import seaborn as sns\n", + "import matplotlib as mpl\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "sns.set(style='white', rc={'figure.figsize':(25, 12.5)})" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# hide warnings\n", + "import warnings\n", + "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Sanity Checks\n", + "\n", + "We are going to work with the [fashion mnist](https://github.com/zalandoresearch/fashion-mnist) data set. This is a dataset consisting of 70,000 28x28 grayscale images of clothing. It should already be in the `data/fashion` folder, but let's do a sanity check!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if not os.path.exists('data/fashion'):\n", + " print(\"error, data is missing!\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's make sure we have our RAPIDS compliant GPU. It must be Pascal or higher! You can also use this to define which GPU RAPIDS should use (advanced feature not covered here)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Helper Functions" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# https://github.com/zalandoresearch/fashion-mnist/blob/master/utils/mnist_reader.py\n", + "def load_mnist(path, kind='train'):\n", + " import os\n", + " import gzip\n", + " import numpy as np\n", + "\n", + " \"\"\"Load MNIST data from `path`\"\"\"\n", + " labels_path = os.path.join(path,\n", + " '%s-labels-idx1-ubyte.gz'\n", + " % kind)\n", + " images_path = os.path.join(path,\n", + " '%s-images-idx3-ubyte.gz'\n", + " % kind)\n", + "\n", + " with gzip.open(labels_path, 'rb') as lbpath:\n", + " labels = np.frombuffer(lbpath.read(), dtype=np.uint8,\n", + " offset=8)\n", + "\n", + " with gzip.open(images_path, 'rb') as imgpath:\n", + " images = np.frombuffer(imgpath.read(), dtype=np.uint8,\n", + " offset=16).reshape(len(labels), 784)\n", + "\n", + " return images, labels" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Training" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "train, train_labels = load_mnist('data/fashion', kind='train')\n", + "test, test_labels = load_mnist('data/fashion', kind='t10k')\n", + "data = (np.array(np.vstack([train, test]), dtype=np.float64) [:60000]/ 255.0).astype(np.float32)\n", + "target = np.array(np.hstack([train_labels, test_labels]))[:60000].astype(np.float32)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are 60000 training images and 10000 test images" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "f\"Train shape: {train.shape} and Test Shape: {test.shape}\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "train[0].shape" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As mentioned previously, each row in the train matrix is an image" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# display a Nike? sneaker\n", + "pixels = train[0].reshape((28, 28))\n", + "plt.imshow(pixels, cmap='gray')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There is cost with moving data between host memory and device memory (GPU memory) and we will include that core when comparing speeds" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "record_data = (('fea%d'%i, data[:,i]) for i in range(data.shape[1]))\n", + "gdf = cudf.DataFrame(record_data)\n", + "\n", + "label_data = [('fea%0', target)]\n", + "\n", + "target_gdf = cudf.DataFrame(label_data)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import time" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`gdf` is a GPU backed dataframe -- all the data is stored in the device memory of the GPU. With the data converted, we can apply the `cumlUMAP` the same inputs as we do for the standard UMAP.\n", + "\n", + "For datasets that provide a set of labels, we can pass those labels into the `fit()` and `fit_transform()` functions to have UMAP use them for better cluster separation. Supervised training can even be used with an incomplete set of labels by setting the unknown labels to -1\". It is important that the labels array be the same size as the number of samples being used to train." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "start = time.time()\n", + "g_embedding_supervised = cumlUMAP(verbose = False, n_neighbors=5, init=\"spectral\", target_metric = \"categorical\").fit_transform(gdf, target_gdf)\n", + "print(\"Took %f sec.\" % (time.time() - start))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "start = time.time()\n", + "g_embedding = cumlUMAP(n_neighbors=5, init=\"spectral\").fit_transform(gdf)\n", + "print(\"Took %f sec.\" % (time.time() - start))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Visualization\n", + "\n", + "OK, now let's plot the output of the embeddings so that we can see the seperation of the neighborhoods. Let's start by creating the classes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "classes = [\n", + " 'T-shirt/top',\n", + " 'Trouser',\n", + " 'Pullover',\n", + " 'Dress',\n", + " 'Coat',\n", + " 'Sandal',\n", + " 'Shirt',\n", + " 'Sneaker',\n", + " 'Bag',\n", + " 'Ankle boot']" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Just as the original author of UMAP, Leland McInnes, states in the [UMAP docs](https://umap-learn.readthedocs.io/en/latest/supervised.html), we can plot the results and show the separation between the various classes defined above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "g_embedding_supervised_numpy = g_embedding_supervised.to_pandas().values #it is necessary to convert to numpy array to do the visual mapping\n", + "\n", + "fig, ax = plt.subplots(1, figsize=(14, 10))\n", + "plt.scatter(g_embedding_supervised_numpy[:,1], g_embedding_supervised_numpy[:,0], s=0.3, c=target, cmap='Spectral', alpha=1.0)\n", + "plt.setp(ax, xticks=[], yticks=[])\n", + "cbar = plt.colorbar(boundaries=np.arange(11)-0.5)\n", + "cbar.set_ticks(np.arange(10))\n", + "cbar.set_ticklabels(classes)\n", + "plt.title('Supervised Fashion MNIST Embedded via cumlUMAP');" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Comparison of Implementations\n", + "\n", + "And side-by-side we can see the effects of supervised training. Notice how providing the labels enables the resulting model to better separation of sneakers, ankle books, and sandals while also providing a much more distinct separation of shirts, t-shirts, pullovers, and coats. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "g_embedding_numpy = g_embedding.to_pandas().values #it is necessary to convert to numpy array to do the visual mapping\n", + "\n", + "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(20, 10))\n", + "ax[0].scatter(g_embedding_numpy[:,1], g_embedding_numpy[:,0], s=0.3, c=target, cmap='Spectral', alpha=1.0)\n", + "im = ax[1].scatter(g_embedding_supervised_numpy[:,1], g_embedding_supervised_numpy[:,0], s=0.3, c=target, cmap='Spectral', alpha=1.0)\n", + "ax[0].set_title('Unsupervised Fashion MNIST Embedded via cumlUMAP ');\n", + "ax[1].set_title('Supervised Fashion MNIST Embedded via UMAP');\n", + "\n", + "fig.subplots_adjust(right=0.8)\n", + "cax,kw = mpl.colorbar.make_axes([a for a in ax.flat])\n", + "cbar = plt.colorbar(im, cax=cax, **kw)\n", + "cbar.set_ticks(np.arange(10))\n", + "cbar.set_ticklabels(classes)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Additionally, we can also quanititaviely compare the perfomance of `cumlUMAP` (GPU UMAP) to the reference/original implementation (CPU UMAP) using the [trustworthiness score](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/t_sne.py#L395). From the docstring:\n", + "\n", + "> Trustworthiness expresses to what extent the local structure is retained. The trustworthiness is within [0, 1].\n", + "\n", + "\n", + "Like `t-SNE`, UMAP tries to capture both global and local structure and thus, we can apply the `trustworthiness` of the `embedding/g_embedding` data against the original input. With a higher score we are demonstrating that the algorithm does a better and better job of local structure retention. As [Corey Nolet](https://github.com/cjnolet) notes:\n", + "> Algorithms like UMAP aim to preserve local neighborhood structure and so measuring this property (trustworthiness) measures the algorithm's performance." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Scoring ~97% shows the GPU implementation is comparable to the original CPU implementation and the training time was ~9.5X faster" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/docs/dockerhub-README.md b/docs/dockerhub-README.md index 6ff569cc..aa0104d5 100644 --- a/docs/dockerhub-README.md +++ b/docs/dockerhub-README.md @@ -90,7 +90,7 @@ Notebooks can be found in two directories within the container: * `/rapids/notebooks/cuml` - cuML demo notebooks * These notebooks have data pre-loaded in the container image and will be decompressed by the notebooks * `/rapids/notebooks/mortgage` - cuDF, Dask, XGBoost demo notebook - * This notebook requires download of [Mortgage Data](https://rapidsai.github.io/demos/datasets/mortgage-data), see notebook `E2E.ipynb` for more details + * This notebook requires download of [Mortgage Data](https://docs.rapids.ai/datasets/mortgage-data), see notebook `E2E.ipynb` for more details ### Custom Data and Advanced Usage diff --git a/docs/ngc-README.md b/docs/ngc-README.md index 38c9e262..e81bf013 100644 --- a/docs/ngc-README.md +++ b/docs/ngc-README.md @@ -90,7 +90,7 @@ Notebooks can be found in two directories within the container: * `/rapids/notebooks/cuml` - cuML demo notebooks * These notebooks have data pre-loaded in the container image and will be decompressed by the notebooks * `/rapids/notebooks/mortgage` - cuDF, Dask, XGBoost demo notebook - * This notebook requires download of [Mortgage Data](https://rapidsai.github.io/demos/datasets/mortgage-data), see notebook `E2E.ipynb` for more details + * This notebook requires download of [Mortgage Data](https://docs.rapids.ai/datasets/mortgage-data), see notebook `E2E.ipynb` for more details ### Custom Data and Advanced Usage diff --git a/mortgage/E2E.ipynb b/mortgage/E2E.ipynb deleted file mode 100644 index a8419572..00000000 --- a/mortgage/E2E.ipynb +++ /dev/null @@ -1,731 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Mortgage Workflow\n", - "\n", - "## The Dataset\n", - "The dataset used with this workflow is derived from [Fannie Mae’s Single-Family Loan Performance Data](http://www.fanniemae.com/portal/funding-the-market/data/loan-performance-data.html) with all rights reserved by Fannie Mae. This processed dataset is redistributed with permission and consent from Fannie Mae.\n", - "\n", - "To acquire this dataset, please visit [RAPIDS Datasets Homepage](https://rapidsai.github.io/demos/datasets/mortgage-data)\n", - "\n", - "## Introduction\n", - "The Mortgage workflow is composed of three core phases:\n", - "\n", - "1. ETL - Extract, Transform, Load\n", - "2. Data Conversion\n", - "3. ML - Training\n", - "\n", - "### ETL\n", - "Data is \n", - "1. Read in from storage\n", - "2. Transformed to emphasize key features\n", - "3. Loaded into volatile memory for conversion\n", - "\n", - "### Data Conversion\n", - "Features are\n", - "1. Broken into (labels, data) pairs\n", - "2. Distributed across many workers\n", - "3. Converted into compressed sparse row (CSR) matrix format for XGBoost\n", - "\n", - "### Machine Learning\n", - "The CSR data is fed into a distributed training session with Dask-XGBoost" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Imports statements" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%env NCCL_P2P_DISABLE=1" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "import dask_xgboost as dxgb_gpu\n", - "import dask\n", - "import dask_cudf\n", - "from dask_cuda import LocalCUDACluster\n", - "from dask.delayed import delayed\n", - "from dask.distributed import Client, wait\n", - "import xgboost as xgb\n", - "import cudf\n", - "from cudf.dataframe import DataFrame\n", - "from collections import OrderedDict\n", - "import gc\n", - "from glob import glob\n", - "import os" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import subprocess\n", - "\n", - "cmd = \"hostname --all-ip-addresses\"\n", - "process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)\n", - "output, error = process.communicate()\n", - "IPADDR = str(output.decode()).split()[0]\n", - "\n", - "cluster = LocalCUDACluster(ip=IPADDR)\n", - "client = Client(cluster)\n", - "client" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Define the paths to data and set the size of the dataset" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# to download data for this notebook, visit https://rapidsai.github.io/demos/datasets/mortgage-data and update the following paths accordingly\n", - "acq_data_path = \"/path/to/mortgage/acq\"\n", - "perf_data_path = \"/path/to/mortgage/perf\"\n", - "col_names_path = \"/path/to/mortgage/names.csv\"\n", - "start_year = 2000\n", - "end_year = 2016 # end_year is inclusive\n", - "part_count = 16 # the number of data files to train against" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def initialize_rmm_pool():\n", - " from librmm_cffi import librmm_config as rmm_cfg\n", - "\n", - " rmm_cfg.use_pool_allocator = True\n", - " #rmm_cfg.initial_pool_size = 2<<30 # set to 2GiB. Default is 1/2 total GPU memory\n", - " import cudf\n", - " return cudf._gdf.rmm_initialize()\n", - "\n", - "def initialize_rmm_no_pool():\n", - " from librmm_cffi import librmm_config as rmm_cfg\n", - " \n", - " rmm_cfg.use_pool_allocator = False\n", - " import cudf\n", - " return cudf._gdf.rmm_initialize()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "client.run(initialize_rmm_pool)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Define functions to encapsulate the workflow into a single call" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def run_dask_task(func, **kwargs):\n", - " task = func(**kwargs)\n", - " return task\n", - "\n", - "def process_quarter_gpu(year=2000, quarter=1, perf_file=\"\"):\n", - " ml_arrays = run_dask_task(delayed(run_gpu_workflow),\n", - " quarter=quarter,\n", - " year=year,\n", - " perf_file=perf_file)\n", - " return client.compute(ml_arrays,\n", - " optimize_graph=False,\n", - " fifo_timeout=\"0ms\")\n", - "\n", - "def null_workaround(df, **kwargs):\n", - " for column, data_type in df.dtypes.items():\n", - " if str(data_type) == \"category\":\n", - " df[column] = df[column].astype('int32').fillna(-1)\n", - " if str(data_type) in ['int8', 'int16', 'int32', 'int64', 'float32', 'float64']:\n", - " df[column] = df[column].fillna(np.dtype(data_type).type(-1))\n", - " return df\n", - "\n", - "def run_gpu_workflow(quarter=1, year=2000, perf_file=\"\", **kwargs):\n", - " names = gpu_load_names()\n", - " acq_gdf = gpu_load_acquisition_csv(acquisition_path= acq_data_path + \"/Acquisition_\"\n", - " + str(year) + \"Q\" + str(quarter) + \".txt\")\n", - " acq_gdf = acq_gdf.merge(names, how='left', on=['seller_name'])\n", - " acq_gdf.drop_column('seller_name')\n", - " acq_gdf['seller_name'] = acq_gdf['new']\n", - " acq_gdf.drop_column('new')\n", - " perf_df_tmp = gpu_load_performance_csv(perf_file)\n", - " gdf = perf_df_tmp\n", - " everdf = create_ever_features(gdf)\n", - " delinq_merge = create_delinq_features(gdf)\n", - " everdf = join_ever_delinq_features(everdf, delinq_merge)\n", - " del(delinq_merge)\n", - " joined_df = create_joined_df(gdf, everdf)\n", - " testdf = create_12_mon_features(joined_df)\n", - " joined_df = combine_joined_12_mon(joined_df, testdf)\n", - " del(testdf)\n", - " perf_df = final_performance_delinquency(gdf, joined_df)\n", - " del(gdf, joined_df)\n", - " final_gdf = join_perf_acq_gdfs(perf_df, acq_gdf)\n", - " del(perf_df)\n", - " del(acq_gdf)\n", - " final_gdf = last_mile_cleaning(final_gdf)\n", - " return final_gdf\n", - "\n", - "def gpu_load_performance_csv(performance_path, **kwargs):\n", - " \"\"\" Loads performance data\n", - "\n", - " Returns\n", - " -------\n", - " GPU DataFrame\n", - " \"\"\"\n", - " \n", - " cols = [\n", - " \"loan_id\", \"monthly_reporting_period\", \"servicer\", \"interest_rate\", \"current_actual_upb\",\n", - " \"loan_age\", \"remaining_months_to_legal_maturity\", \"adj_remaining_months_to_maturity\",\n", - " \"maturity_date\", \"msa\", \"current_loan_delinquency_status\", \"mod_flag\", \"zero_balance_code\",\n", - " \"zero_balance_effective_date\", \"last_paid_installment_date\", \"foreclosed_after\",\n", - " \"disposition_date\", \"foreclosure_costs\", \"prop_preservation_and_repair_costs\",\n", - " \"asset_recovery_costs\", \"misc_holding_expenses\", \"holding_taxes\", \"net_sale_proceeds\",\n", - " \"credit_enhancement_proceeds\", \"repurchase_make_whole_proceeds\", \"other_foreclosure_proceeds\",\n", - " \"non_interest_bearing_upb\", \"principal_forgiveness_upb\", \"repurchase_make_whole_proceeds_flag\",\n", - " \"foreclosure_principal_write_off_amount\", \"servicing_activity_indicator\"\n", - " ]\n", - " \n", - " dtypes = OrderedDict([\n", - " (\"loan_id\", \"int64\"),\n", - " (\"monthly_reporting_period\", \"date\"),\n", - " (\"servicer\", \"category\"),\n", - " (\"interest_rate\", \"float64\"),\n", - " (\"current_actual_upb\", \"float64\"),\n", - " (\"loan_age\", \"float64\"),\n", - " (\"remaining_months_to_legal_maturity\", \"float64\"),\n", - " (\"adj_remaining_months_to_maturity\", \"float64\"),\n", - " (\"maturity_date\", \"date\"),\n", - " (\"msa\", \"float64\"),\n", - " (\"current_loan_delinquency_status\", \"int32\"),\n", - " (\"mod_flag\", \"category\"),\n", - " (\"zero_balance_code\", \"category\"),\n", - " (\"zero_balance_effective_date\", \"date\"),\n", - " (\"last_paid_installment_date\", \"date\"),\n", - " (\"foreclosed_after\", \"date\"),\n", - " (\"disposition_date\", \"date\"),\n", - " (\"foreclosure_costs\", \"float64\"),\n", - " (\"prop_preservation_and_repair_costs\", \"float64\"),\n", - " (\"asset_recovery_costs\", \"float64\"),\n", - " (\"misc_holding_expenses\", \"float64\"),\n", - " (\"holding_taxes\", \"float64\"),\n", - " (\"net_sale_proceeds\", \"float64\"),\n", - " (\"credit_enhancement_proceeds\", \"float64\"),\n", - " (\"repurchase_make_whole_proceeds\", \"float64\"),\n", - " (\"other_foreclosure_proceeds\", \"float64\"),\n", - " (\"non_interest_bearing_upb\", \"float64\"),\n", - " (\"principal_forgiveness_upb\", \"float64\"),\n", - " (\"repurchase_make_whole_proceeds_flag\", \"category\"),\n", - " (\"foreclosure_principal_write_off_amount\", \"float64\"),\n", - " (\"servicing_activity_indicator\", \"category\")\n", - " ])\n", - "\n", - " print(performance_path)\n", - " \n", - " return cudf.read_csv(performance_path, names=cols, delimiter='|', dtype=list(dtypes.values()), skiprows=1)\n", - "\n", - "def gpu_load_acquisition_csv(acquisition_path, **kwargs):\n", - " \"\"\" Loads acquisition data\n", - "\n", - " Returns\n", - " -------\n", - " GPU DataFrame\n", - " \"\"\"\n", - " \n", - " cols = [\n", - " 'loan_id', 'orig_channel', 'seller_name', 'orig_interest_rate', 'orig_upb', 'orig_loan_term', \n", - " 'orig_date', 'first_pay_date', 'orig_ltv', 'orig_cltv', 'num_borrowers', 'dti', 'borrower_credit_score', \n", - " 'first_home_buyer', 'loan_purpose', 'property_type', 'num_units', 'occupancy_status', 'property_state',\n", - " 'zip', 'mortgage_insurance_percent', 'product_type', 'coborrow_credit_score', 'mortgage_insurance_type', \n", - " 'relocation_mortgage_indicator'\n", - " ]\n", - " \n", - " dtypes = OrderedDict([\n", - " (\"loan_id\", \"int64\"),\n", - " (\"orig_channel\", \"category\"),\n", - " (\"seller_name\", \"category\"),\n", - " (\"orig_interest_rate\", \"float64\"),\n", - " (\"orig_upb\", \"int64\"),\n", - " (\"orig_loan_term\", \"int64\"),\n", - " (\"orig_date\", \"date\"),\n", - " (\"first_pay_date\", \"date\"),\n", - " (\"orig_ltv\", \"float64\"),\n", - " (\"orig_cltv\", \"float64\"),\n", - " (\"num_borrowers\", \"float64\"),\n", - " (\"dti\", \"float64\"),\n", - " (\"borrower_credit_score\", \"float64\"),\n", - " (\"first_home_buyer\", \"category\"),\n", - " (\"loan_purpose\", \"category\"),\n", - " (\"property_type\", \"category\"),\n", - " (\"num_units\", \"int64\"),\n", - " (\"occupancy_status\", \"category\"),\n", - " (\"property_state\", \"category\"),\n", - " (\"zip\", \"int64\"),\n", - " (\"mortgage_insurance_percent\", \"float64\"),\n", - " (\"product_type\", \"category\"),\n", - " (\"coborrow_credit_score\", \"float64\"),\n", - " (\"mortgage_insurance_type\", \"float64\"),\n", - " (\"relocation_mortgage_indicator\", \"category\")\n", - " ])\n", - " \n", - " print(acquisition_path)\n", - " \n", - " return cudf.read_csv(acquisition_path, names=cols, delimiter='|', dtype=list(dtypes.values()), skiprows=1)\n", - "\n", - "def gpu_load_names(**kwargs):\n", - " \"\"\" Loads names used for renaming the banks\n", - " \n", - " Returns\n", - " -------\n", - " GPU DataFrame\n", - " \"\"\"\n", - "\n", - " cols = [\n", - " 'seller_name', 'new'\n", - " ]\n", - " \n", - " dtypes = OrderedDict([\n", - " (\"seller_name\", \"category\"),\n", - " (\"new\", \"category\"),\n", - " ])\n", - "\n", - " return cudf.read_csv(col_names_path, names=cols, delimiter='|', dtype=list(dtypes.values()), skiprows=1)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def create_ever_features(gdf, **kwargs):\n", - " everdf = gdf[['loan_id', 'current_loan_delinquency_status']]\n", - " everdf = everdf.groupby('loan_id', method='hash', as_index=False).max()\n", - " del(gdf)\n", - " everdf['ever_30'] = (everdf['current_loan_delinquency_status'] >= 1).astype('int8')\n", - " everdf['ever_90'] = (everdf['current_loan_delinquency_status'] >= 3).astype('int8')\n", - " everdf['ever_180'] = (everdf['current_loan_delinquency_status'] >= 6).astype('int8')\n", - " everdf.drop_column('current_loan_delinquency_status')\n", - " return everdf" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def create_delinq_features(gdf, **kwargs):\n", - " delinq_gdf = gdf[['loan_id', 'monthly_reporting_period', 'current_loan_delinquency_status']]\n", - " del(gdf)\n", - " delinq_30 = delinq_gdf.query('current_loan_delinquency_status >= 1')[['loan_id', 'monthly_reporting_period']].groupby('loan_id', method='hash', as_index=False).min()\n", - " delinq_30['delinquency_30'] = delinq_30['monthly_reporting_period']\n", - " delinq_30.drop_column('monthly_reporting_period')\n", - " delinq_90 = delinq_gdf.query('current_loan_delinquency_status >= 3')[['loan_id', 'monthly_reporting_period']].groupby('loan_id', method='hash', as_index=False).min()\n", - " delinq_90['delinquency_90'] = delinq_90['monthly_reporting_period']\n", - " delinq_90.drop_column('monthly_reporting_period')\n", - " delinq_180 = delinq_gdf.query('current_loan_delinquency_status >= 6')[['loan_id', 'monthly_reporting_period']].groupby('loan_id', method='hash', as_index=False).min()\n", - " delinq_180['delinquency_180'] = delinq_180['monthly_reporting_period']\n", - " delinq_180.drop_column('monthly_reporting_period')\n", - " del(delinq_gdf)\n", - " delinq_merge = delinq_30.merge(delinq_90, how='left', on=['loan_id'], type='hash')\n", - " delinq_merge['delinquency_90'] = delinq_merge['delinquency_90'].fillna(np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]'))\n", - " delinq_merge = delinq_merge.merge(delinq_180, how='left', on=['loan_id'], type='hash')\n", - " delinq_merge['delinquency_180'] = delinq_merge['delinquency_180'].fillna(np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]'))\n", - " del(delinq_30)\n", - " del(delinq_90)\n", - " del(delinq_180)\n", - " return delinq_merge" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def join_ever_delinq_features(everdf_tmp, delinq_merge, **kwargs):\n", - " everdf = everdf_tmp.merge(delinq_merge, on=['loan_id'], how='left', type='hash')\n", - " del(everdf_tmp)\n", - " del(delinq_merge)\n", - " everdf['delinquency_30'] = everdf['delinquency_30'].fillna(np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]'))\n", - " everdf['delinquency_90'] = everdf['delinquency_90'].fillna(np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]'))\n", - " everdf['delinquency_180'] = everdf['delinquency_180'].fillna(np.dtype('datetime64[ms]').type('1970-01-01').astype('datetime64[ms]'))\n", - " return everdf" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def create_joined_df(gdf, everdf, **kwargs):\n", - " test = gdf[['loan_id', 'monthly_reporting_period', 'current_loan_delinquency_status', 'current_actual_upb']]\n", - " del(gdf)\n", - " test['timestamp'] = test['monthly_reporting_period']\n", - " test.drop_column('monthly_reporting_period')\n", - " test['timestamp_month'] = test['timestamp'].dt.month\n", - " test['timestamp_year'] = test['timestamp'].dt.year\n", - " test['delinquency_12'] = test['current_loan_delinquency_status']\n", - " test.drop_column('current_loan_delinquency_status')\n", - " test['upb_12'] = test['current_actual_upb']\n", - " test.drop_column('current_actual_upb')\n", - " test['upb_12'] = test['upb_12'].fillna(999999999)\n", - " test['delinquency_12'] = test['delinquency_12'].fillna(-1)\n", - " \n", - " joined_df = test.merge(everdf, how='left', on=['loan_id'], type='hash')\n", - " del(everdf)\n", - " del(test)\n", - " \n", - " joined_df['ever_30'] = joined_df['ever_30'].fillna(-1)\n", - " joined_df['ever_90'] = joined_df['ever_90'].fillna(-1)\n", - " joined_df['ever_180'] = joined_df['ever_180'].fillna(-1)\n", - " joined_df['delinquency_30'] = joined_df['delinquency_30'].fillna(-1)\n", - " joined_df['delinquency_90'] = joined_df['delinquency_90'].fillna(-1)\n", - " joined_df['delinquency_180'] = joined_df['delinquency_180'].fillna(-1)\n", - " \n", - " joined_df['timestamp_year'] = joined_df['timestamp_year'].astype('int32')\n", - " joined_df['timestamp_month'] = joined_df['timestamp_month'].astype('int32')\n", - " \n", - " return joined_df" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def create_12_mon_features(joined_df, **kwargs):\n", - " testdfs = []\n", - " n_months = 12\n", - " for y in range(1, n_months + 1):\n", - " tmpdf = joined_df[['loan_id', 'timestamp_year', 'timestamp_month', 'delinquency_12', 'upb_12']]\n", - " tmpdf['josh_months'] = tmpdf['timestamp_year'] * 12 + tmpdf['timestamp_month']\n", - " tmpdf['josh_mody_n'] = ((tmpdf['josh_months'].astype('float64') - 24000 - y) / 12).floor()\n", - " tmpdf = tmpdf.groupby(['loan_id', 'josh_mody_n'], method='hash', as_index=False).agg({'delinquency_12': 'max','upb_12': 'min'})\n", - " tmpdf['delinquency_12'] = (tmpdf['max_delinquency_12']>3).astype('int32')\n", - " tmpdf['delinquency_12'] +=(tmpdf['min_upb_12']==0).astype('int32')\n", - " tmpdf.drop_column('max_delinquency_12')\n", - " tmpdf['upb_12'] = tmpdf['min_upb_12']\n", - " tmpdf.drop_column('min_upb_12')\n", - " tmpdf['timestamp_year'] = (((tmpdf['josh_mody_n'] * n_months) + 24000 + (y - 1)) / 12).floor().astype('int16')\n", - " tmpdf['timestamp_month'] = np.int8(y)\n", - " tmpdf.drop_column('josh_mody_n')\n", - " testdfs.append(tmpdf)\n", - " del(tmpdf)\n", - " del(joined_df)\n", - "\n", - " return cudf.concat(testdfs)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def combine_joined_12_mon(joined_df, testdf, **kwargs):\n", - " joined_df.drop_column('delinquency_12')\n", - " joined_df.drop_column('upb_12')\n", - " joined_df['timestamp_year'] = joined_df['timestamp_year'].astype('int16')\n", - " joined_df['timestamp_month'] = joined_df['timestamp_month'].astype('int8')\n", - " return joined_df.merge(testdf, how='left', on=['loan_id', 'timestamp_year', 'timestamp_month'], type='hash')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def final_performance_delinquency(gdf, joined_df, **kwargs):\n", - " merged = null_workaround(gdf)\n", - " joined_df = null_workaround(joined_df)\n", - " joined_df['timestamp_month'] = joined_df['timestamp_month'].astype('int8')\n", - " joined_df['timestamp_year'] = joined_df['timestamp_year'].astype('int16')\n", - " merged['timestamp_month'] = merged['monthly_reporting_period'].dt.month\n", - " merged['timestamp_month'] = merged['timestamp_month'].astype('int8')\n", - " merged['timestamp_year'] = merged['monthly_reporting_period'].dt.year\n", - " merged['timestamp_year'] = merged['timestamp_year'].astype('int16')\n", - " merged = merged.merge(joined_df, how='left', on=['loan_id', 'timestamp_year', 'timestamp_month'], type='hash')\n", - " merged.drop_column('timestamp_year')\n", - " merged.drop_column('timestamp_month')\n", - " return merged" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def join_perf_acq_gdfs(perf, acq, **kwargs):\n", - " perf = null_workaround(perf)\n", - " acq = null_workaround(acq)\n", - " return perf.merge(acq, how='left', on=['loan_id'], type='hash')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def last_mile_cleaning(df, **kwargs):\n", - " drop_list = [\n", - " 'loan_id', 'orig_date', 'first_pay_date', 'seller_name',\n", - " 'monthly_reporting_period', 'last_paid_installment_date', 'maturity_date', 'ever_30', 'ever_90', 'ever_180',\n", - " 'delinquency_30', 'delinquency_90', 'delinquency_180', 'upb_12',\n", - " 'zero_balance_effective_date','foreclosed_after', 'disposition_date','timestamp'\n", - " ]\n", - " for column in drop_list:\n", - " df.drop_column(column)\n", - " for col, dtype in df.dtypes.iteritems():\n", - " if str(dtype)=='category':\n", - " df[col] = df[col].cat.codes\n", - " df[col] = df[col].astype('float32')\n", - " df['delinquency_12'] = df['delinquency_12'] > 0\n", - " df['delinquency_12'] = df['delinquency_12'].fillna(False).astype('int32')\n", - " for column in df.columns:\n", - " df[column] = df[column].fillna(np.dtype(str(df[column].dtype)).type(-1))\n", - " return df.to_arrow(preserve_index=False)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## ETL" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Perform all of ETL with a single call to\n", - "```python\n", - "process_quarter_gpu(year=year, quarter=quarter, perf_file=file)\n", - "```" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%time\n", - "\n", - "# NOTE: The ETL calculates additional features which are then dropped before creating the XGBoost DMatrix.\n", - "# This can be optimized to avoid calculating the dropped features.\n", - "\n", - "gpu_dfs = []\n", - "gpu_time = 0\n", - "quarter = 1\n", - "year = start_year\n", - "count = 0\n", - "while year <= end_year:\n", - " for file in glob(os.path.join(perf_data_path + \"/Performance_\" + str(year) + \"Q\" + str(quarter) + \"*\")):\n", - " gpu_dfs.append(process_quarter_gpu(year=year, quarter=quarter, perf_file=file))\n", - " count += 1\n", - " quarter += 1\n", - " if quarter == 5:\n", - " year += 1\n", - " quarter = 1\n", - "wait(gpu_dfs)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "client.run(cudf._gdf.rmm_finalize)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "client.run(initialize_rmm_no_pool)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Machine Learning" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Set the training parameters" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "dxgb_gpu_params = {\n", - " 'nround': 100,\n", - " 'max_depth': 8,\n", - " 'max_leaves': 2**8,\n", - " 'alpha': 0.9,\n", - " 'eta': 0.1,\n", - " 'gamma': 0.1,\n", - " 'learning_rate': 0.1,\n", - " 'subsample': 1,\n", - " 'reg_lambda': 1,\n", - " 'scale_pos_weight': 2,\n", - " 'min_child_weight': 30,\n", - " 'tree_method': 'gpu_hist',\n", - " 'n_gpus': 1,\n", - " 'distributed_dask': True,\n", - " 'loss': 'ls',\n", - " 'objective': 'gpu:reg:linear',\n", - " 'max_features': 'auto',\n", - " 'criterion': 'friedman_mse',\n", - " 'grow_policy': 'lossguide',\n", - " 'verbose': True\n", - "}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Load the data from host memory, and convert to CSR" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%time\n", - "\n", - "gpu_dfs = [delayed(DataFrame.from_arrow)(gpu_df) for gpu_df in gpu_dfs[:part_count]]\n", - "gpu_dfs = [gpu_df for gpu_df in gpu_dfs]\n", - "wait(gpu_dfs)\n", - "\n", - "tmp_map = [(gpu_df, list(client.who_has(gpu_df).values())[0]) for gpu_df in gpu_dfs]\n", - "new_map = {}\n", - "for key, value in tmp_map:\n", - " if value not in new_map:\n", - " new_map[value] = [key]\n", - " else:\n", - " new_map[value].append(key)\n", - "\n", - "del(tmp_map)\n", - "gpu_dfs = []\n", - "for list_delayed in new_map.values():\n", - " gpu_dfs.append(delayed(cudf.concat)(list_delayed))\n", - "\n", - "del(new_map)\n", - "gpu_dfs = [(gpu_df[['delinquency_12']], gpu_df[delayed(list)(gpu_df.columns.difference(['delinquency_12']))]) for gpu_df in gpu_dfs]\n", - "gpu_dfs = [(gpu_df[0].persist(), gpu_df[1].persist()) for gpu_df in gpu_dfs]\n", - "\n", - "gpu_dfs = [dask.delayed(xgb.DMatrix)(gpu_df[1], gpu_df[0]) for gpu_df in gpu_dfs]\n", - "gpu_dfs = [gpu_df.persist() for gpu_df in gpu_dfs]\n", - "gc.collect()\n", - "wait(gpu_dfs)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Train the Gradient Boosted Decision Tree with a single call to \n", - "```python\n", - "dask_xgboost.train(client, params, data, labels, num_boost_round=dxgb_gpu_params['nround'])\n", - "```" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%time\n", - "labels = None\n", - "bst = dxgb_gpu.train(client, dxgb_gpu_params, gpu_dfs, labels, num_boost_round=dxgb_gpu_params['nround'])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.7" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/tutorials/DBSCAN_Demo_Full.ipynb b/tutorials/DBSCAN_Demo_Full.ipynb new file mode 100644 index 00000000..47c70110 --- /dev/null +++ b/tutorials/DBSCAN_Demo_Full.ipynb @@ -0,0 +1,670 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Clustering using GPU Accelerated DBSCAN in RAPIDS\n", + "#### By Paul Hendricks\n", + "-------\n", + "\n", + "While the world’s data doubles each year, CPU computing has hit a brick wall with the end of Moore’s law. For the same reasons, scientific computing and deep learning has turned to NVIDIA GPU acceleration, data analytics and machine learning where GPU acceleration is ideal. \n", + "\n", + "NVIDIA created RAPIDS – an open-source data analytics and machine learning acceleration platform that leverages GPUs to accelerate computations. RAPIDS is based on Python, has pandas-like and Scikit-Learn-like interfaces, is built on Apache Arrow in-memory data format, and can scale from 1 to multi-GPU to multi-nodes. RAPIDS integrates easily into the world’s most popular data science Python-based workflows. RAPIDS accelerates data science end-to-end – from data prep, to machine learning, to deep learning. And through Arrow, Spark users can easily move data into the RAPIDS platform for acceleration.\n", + "\n", + "In this notebook, we will also show how to use DBSCAN - a popular clustering algorithm - and how to use the GPU accelerated implementation of this algorithm in RAPIDS.\n", + "\n", + "**Table of Contents**\n", + "\n", + "* Clustering with DBSCAN\n", + "* Setup\n", + "* Generating Data\n", + "* K Means and Agglomerative Clustering\n", + "* Clustering using DBSCAN\n", + "* Accelerating DBSCAN with RAPIDS\n", + "* Benchmarking: Comparing GPU and CPU\n", + "* Conclusion\n", + "\n", + "Before going any further, let's make sure we have access to `matplotlib`, a popular Python library for data visualization." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "try:\n", + " import matplotlib\n", + "except ModuleNotFoundError:\n", + " os.system('conda install -y matplotlib')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Clustering with DBSCAN\n", + "\n", + "Clustering is an important technique for helping data scientists partition data, especially when that data doesn't have labels or annotations associated with it. Since these data often don't have labels, clustering is often described as an unsupervised learning technique. While there are many different algorithms that partition data into unique clusters, we will show in this notebook how in certain cases the DBSCAN algorithm can do a better job of clustering than traditional algorithms such as K Means or Agglomerative Clustering. \n", + "\n", + "We will also show how to cluster data with DBSCAN in NVIDIA RAPIDS – an open-source data analytics and machine learning acceleration platform that leverages GPUs to accelerate computations. RAPIDS is based on Python, has pandas-like and Scikit-Learn-like interfaces, is built on Apache Arrow in-memory data format, and can scale from 1 to multi-GPU to multi-nodes. We will see that porting this example from CPU to GPU is trivial and that we can experience massive performance gains by doing so." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "This notebook was tested using the `nvcr.io/nvidia/rapidsai/rapidsai:0.5-cuda10.0-runtime-ubuntu18.04-gcc7-py3.7` Docker container from [NVIDIA GPU Cloud](https://ngc.nvidia.com) and run on the NVIDIA Tesla V100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. \n", + "\n", + "If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks/issues\n", + "\n", + "Before we begin, let's check out our hardware setup by running the `nvidia-smi` command." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Tue May 7 00:34:27 2019 \n", + "+-----------------------------------------------------------------------------+\n", + "| NVIDIA-SMI 418.39 Driver Version: 418.39 CUDA Version: 10.1 |\n", + "|-------------------------------+----------------------+----------------------+\n", + "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", + "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", + "|===============================+======================+======================|\n", + "| 0 Quadro GV100 Off | 00000000:15:00.0 Off | Off |\n", + "| 29% 40C P2 26W / 250W | 10149MiB / 32478MiB | 0% Default |\n", + "+-------------------------------+----------------------+----------------------+\n", + "| 1 Quadro GV100 Off | 00000000:2D:00.0 On | Off |\n", + "| 33% 46C P0 29W / 250W | 260MiB / 32470MiB | 24% Default |\n", + "+-------------------------------+----------------------+----------------------+\n", + " \n", + "+-----------------------------------------------------------------------------+\n", + "| Processes: GPU Memory |\n", + "| GPU PID Type Process name Usage |\n", + "|=============================================================================|\n", + "+-----------------------------------------------------------------------------+\n" + ] + } + ], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, let's see what CUDA version we have:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "nvcc: NVIDIA (R) Cuda compiler driver\n", + "Copyright (c) 2005-2018 NVIDIA Corporation\n", + "Built on Sat_Aug_25_21:08:01_CDT_2018\n", + "Cuda compilation tools, release 10.0, V10.0.130\n" + ] + } + ], + "source": [ + "!nvcc --version" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, let's load some helper functions from `matplotlib` and configure the Jupyter Notebook for visualization." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "from matplotlib.colors import ListedColormap\n", + "import matplotlib.pyplot as plt\n", + "\n", + "\n", + "%matplotlib inline" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generating Data\n", + "\n", + "We'll generate some fake data using the `make_moons` function from the `sklearn.datasets` module. This function generates data points from two equations, each describing a half circle with a unique center. Since each data point is generated by one of these two equations, the cluster each data point belongs to is clear. The ideal clustering algorithm will identify two clusters and associate each data point with the equation that generated it." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(100, 2)\n" + ] + } + ], + "source": [ + "from sklearn.datasets import make_moons\n", + "\n", + "X, y = make_moons(n_samples=int(1e2), noise=0.05, random_state=0)\n", + "print(X.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's visualize our data:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X+QXWd93/H3R+ulXmPwmliAtZZidUaVMXWM4MZ2Rm1qDI6MSZBCcGsg4ccko3EnzpAZRkUkMwlMMmVTTdPQ8sOjMS4wOHgc2xECC1SwTGlJTb1CFkaWBaqd2Fq5ePkhJ7HV0Ur69o+9K1/dPffnOffe8+PzmtHsvec+e8+5e3TP9zzP832eRxGBmZlZ3iwb9QGYmZklcYAyM7NccoAyM7NccoAyM7NccoAyM7NccoAyM7NccoAyM7NccoAyM7NccoAyM7NcOmfUB9DORRddFJdeeumoD8PMzDK0d+/eH0fE8k7lch2gLr30UmZmZkZ9GGZmliFJf9dNOTfxmZlZLjlAmZlZLjlAmZlZLjlAmZlZLjlAmZlZLjlAmZlZLjlAmZlZLjlAmZlZLuV6oK5ZGjv2zbJt9yGOHjvOiskJtmxYy6Z1U6M+LDPrkgOU5U4WgWXHvlk+fN+jHJ8/BcDsseN8+L5HARykzArCTXyWK4uBZfbYcYIXA8uOfbM9vc+23YfOBKdFx+dPsW33oQyP1swGyTUoy5V2gaWXms/RY8fbbnfzn1n+uQZludIpsHRrxeREy+1Z1dLMbLAcoCxX2gWWXmzZsJaJ8bGztk2Mj7Flw1o3/5kVhAOU5Uq7wNKLTeum+Njbr2BqcgIBU5MTfOztV7Bp3VRmtTQzG6xM+qAk3QH8KvBsRPzzhNcFfBy4EXgBeF9EfDeLfVu5LPYDZdE/tGndVOLvrZicYDYhGPVaS8uK+8PMkmWVJPFZ4BPA51u8/hZgTf3f1cCn6z+t5Lq9+A7zIr1lw9qzUtChv1paFpwOb9ZaJgEqIr4l6dI2RTYCn4+IAB6SNCnp4oh4Jov9Wz51e/Ed9kW631raIIJoVlmLZmU0rDTzKeDphudH6tuWBChJm4HNAKtWrRrKwdlgdHvxHcVFulXzXyuDCqLuDzNrbVgBSgnbIqlgRGwHtgPUarXEMjY4WdYSur34ZnWRHmQz4aCCaN76w8zyZFhZfEeAlQ3PLwGODmnf1qWsxwd1mzKeRWr5oMc2Daqmk1XWolkZDStA7QTeowXXAM+5/yl/sh4f1O3FN4uL9KDHNmU1PqtZu3R4s6rLKs38i8C1wEWSjgB/DIwDRMRtwC4WUswPs5Bm/v4s9mv9S2oOy7qW0G0yQhap5YPuyxlk5l+v/WFmVaGFxLp8qtVqMTMzM+rDKJ3mDn9YuNj+k3OWcez4/JLyU5MTfHvrdcM8xJ6tn96T2JfT67G368fyeCWzbEjaGxG1TuU8WWwFtWoOO3d8GRPjY7kYH9SrLGo4nTL1XNMxGy4HqApq1ex17IV5/tO/eV0hawlZNBP2kqnXXJt642XLefDxucL93czyzAGqgibPG+dnLyxtyps8b7zQtYS0x95tP1ZSTesLDz115nXPBmGWDU8WW0Gtuh1z3B05FN1m6iXVtJoNa3b0HftmWT+9h9Vb72f99B4vGWKl4gBVQc8lJEK0214V3aa7d5sZOOjZILyulZWdA1QFXTAxnri96rMXdDsmqdu/06D/nl7XysrOfVAVs2PfLM+fOLlk+/gyFSJbb9C66cdKyhhsNozsR8/jZ2XnGlTFbNt9iPlTSzubzj/3HHfodymppvWb16wa+mwQg5rdwiwvXIOqmHYp5ta9PGQ75mldK7NBcA2qYnzXXR6ex8/KzjWoihn1XbenC8pWHmpyZoPiAFUxWcy40C8vb25mvXCAqqBR3XV7eXMz64UDVEnlsSnNadFm1gsnSZRQXmcYcIKGmfXCAaqE8jrDgJc3N7NeuImvhPLalDbKBA07Wx6bgM2aOUCV0IrJicTVZfPQlOa06NFzNqUVRSZNfJJukHRI0mFJWxNev0DSlyXtl3RA0vuz2K8lc1OatZPXJmCzZqlrUJLGgE8C1wNHgIcl7YyIxxqK/S7wWET8mqTlwCFJd0bEibT7r5pummbclGbt5LUJ2KxZFk18VwGHI+IJAEl3ARuBxgAVwMskCTgf+CmwdEpta6uXphk3pVkreW4CNmuURRPfFPB0w/Mj9W2NPgG8BjgKPAp8ICJOZ7DvSnHTjGXBTcBWFFkEKCVsa17PYQPwCLACeB3wCUkvT3wzabOkGUkzc3NzGRxeebhpxrLgSWatKLJo4jsCrGx4fgkLNaVG7wemIyKAw5KeBC4D/nfzm0XEdmA7QK1WW7pwUYW5aab8hpX+7SZgK4IsalAPA2skrZb0EuBmYGdTmaeANwFIehWwFngig31Xiptmyi2vM4CYjUrqGlREnJR0K7AbGAPuiIgDkm6pv34b8CfAZyU9ykKT4Ici4sdp911k/dwpOzuv3DyZrtnZMhmoGxG7gF1N225reHwU+JUs9lUGaQZKummmvPrtY/SsEFZWnotvBJyNZ0n6mUzXzYJWZg5QI+BsPEvSTx+jb3aszDwX3whcMDHOsePzS7Y7G6/a+ulj9M2OlZkD1JDt2DfL8yeWTqIxvkzOxrOe+xg99MDKzE18Q7Zt9yHmTy0d3nX+uee4Y9t65qEHVmauQQ1Zq6aXYy8sbfIz66TbZkFn+lkROUANWRZNMr7YWKNOzYJe/8mKyk18Q5a2ScZpxdYrZ/pZUTlADVnaiTp9sbFeOdPPispNfCOQZjYIX2ysV870s6JyDapg+pltwKrNmX5WVA5QBeOLjfXK6z9ZUbmJr2A8o7n1w5MMWxE5QBWQLzZmVgVu4jMzs1xygDIzs1xyE59ZBSTNPgLuy7R8c4AyK7mkqY623LMfAuZPx5ltnv7I8sZNfGYllzT7yPypOBOcFnlGEsubTAKUpBskHZJ0WNLWFmWulfSIpAOS/nsW+zWzznqZZcQzkliepA5QksaATwJvAS4H3inp8qYyk8CngLdFxGuBm9Lu18y608ssI56RxPIkixrUVcDhiHgiIk4AdwEbm8q8C7gvIp4CiIhnM9ivmXUhafaR8TExvkxnbfOMJJY3WQSoKeDphudH6tsa/TPgQknflLRX0ntavZmkzZJmJM3Mzc1lcHhm1ZY01dG2d1zJtpuu9PRHlmtZZPEpYVvzmubnAG8A3gRMAP9L0kMR8YMlvxixHdgOUKvVlq6NXhBeVNDypNXsI/4/aXmWRYA6AqxseH4JcDShzI8j4nngeUnfAq4ElgSoImoORm+8bDn37p31CqZmZilk0cT3MLBG0mpJLwFuBnY2lfkS8C8lnSPpPOBq4GAG+x65pBVu73zoKS8qaGaWUuoaVESclHQrsBsYA+6IiAOSbqm/fltEHJT0NeB7wGng9oj4ftp950HSGJNW7ZJO4bUycPO1DUsmM0lExC5gV9O225qebwO2ZbG/POkl6DiF14ouaVYKN1/boHgmiZRaBZ3mzBGn8FoZJLUYuPnaBsUBKqVWK9y++5pVTuG10mnVYuDmaxsETxabkle4tSpZMTnBbEIwcvO1DYIDVAa8wq1VxZYNa8/qgwI3X9vgOECZWdfcYmDDpIj8TtZQq9ViZmZm1IdhZi045dz6IWlvRNQ6lXMNysz64pRzGzRn8ZlZX5xyboPmAGVmfXHKuQ2aA5SZ9aVVarlTzi0rDlBm1pdWg9Sdcm5ZcZKEFYqzxvLDKec2aA5QVhjOGhuOXm4CPEjdBskBygqjVdbYB+/eDzhIZcE3AdZslK0W7oOywmiVHXYqgg/f9yg79s0O+YjKx6nj1ihpQdZhftccoFrYsW+W9dN7WL31ftZP7/HFLwfaZYf5IpoNp45bo1HfsLiJL0FSM8eWe/bzkZ0HeO74/JlqLriDeJiSJipt5Itoep6t3BqN+obFNagESXcN86eCY8fnz1Rzt/zVfrbcs39kVd8q2rRuio+9/QrG1Lwc5AJfRNNz6rg1avWdmjxvfCj7d4BK0M3dwfzpYP7U2RPtuplpMBqbW7ftPsQ7r17pi+iALN4EeLFNg4UblvGxpTeE//j/Tg7lZjyTACXpBkmHJB2WtLVNuV+UdErSO7LY76CkuRN3M1O2kjpp7907y2+8YcoX0QHZtG6Kb2+9jien38q3t17nv2uFbVo3xUtfsrQnaP50DOVmPHUflKQx4JPA9cAR4GFJOyPisYRyfwbsTrvPQevU19GOm5my1aqT9sHH5/j21utGdFRm1fHc8fnE7cO4Gc+iBnUVcDginoiIE8BdwMaEcr8H3As8m8E+B6q5mePC88YZX3Z2NXd8mZZUfd3MlL1Rd9KaVd0o51zMIotvCni64fkR4OrGApKmgF8HrgN+sd2bSdoMbAZYtWpVBofXn+YR8kmD1cBZfIPmrDKz0UpqURrWzXgWASoppap5md6/AD4UEafUIgPrzC9GbAe2w8KKuhkcX9+6GUHtgDRYo/xymNlo51zMIkAdAVY2PL8EONpUpgbcVQ9OFwE3SjoZETsy2P9AeMqX0Vu8QTg+f4oxiVMRTLmmOhKepLfaRjXnYhYB6mFgjaTVwCxwM/CuxgIRsXrxsaTPAl/Jc3CC9iOo/cUcvOYbhFMRZ2pO/vsPl2/WyqkINx2pkyQi4iRwKwvZeQeBuyPigKRbJN2S9v1HxZ3zozXqKVbsRT4X5TPqOfa6lclURxGxC9jVtO22FmXfl8U+B82d86PlG4T88Lkon6K0EHkmiRY85ctoeTnx/PC5KJ9ONx15mSzbAaoFT/kyWr5ByA+fi/Jpd9ORp+Y/RYw0k7utWq0WMzMzoz4MG5EidOJWhc9FuTQnvsDCTcfH3n4F23YfSuzemJqcyGz2Fkl7I6LWsZwDlJlZ9bS66Vi99f4lA1lhYcDrk9NvzWTf3QYorwdlZlZBrcY25SlBzH1QZmZ2Rp76HCtVg3I7ulm2/J0qn1FObdSsMn1Q7ToF/YUy613Sd0rAu69ZxZ9uumJ0B2a55z6oJkUZmGZWFEnfqQDufOgpAB58fG7kd+BWbJUJUB4NXx5uVsqHVt+dxSC12DbjufusX5VJkvBo+HLI0yDCqmv33WnuOPDcffmSl5kiOqlMgMpTZor1zxOX5seWDWsTF4Nrxa0V+VCkm7zKBChPXVQObqrNj03rpnj3NauWBKlWQcutFflQpJu8yvRBQfLANPdnFEueBhEa/OmmK6j9/CvO+g698bLl3Lt31qsg51SRbvIqFaCaeSG24vES8PmTdOPXHLR845cfF0yMc+z4/JLtebzJq3SAcup58eRpEKG1Nqolwq29Hftmef7EySXbx5cplzd5lQ5QRarq2ot88TPrz7bdh5g/tXRyhvPPPSeX36lKByj3Z5hZlbS6+f7ZC/Osn96Tu1aJTLL4JN0g6ZCkw5K2Jrz+bknfq//7G0lXZrHftJx6bmZV0urmW5DLtPPUAUrSGPBJ4C3A5cA7JV3eVOxJ4F9FxC8AfwJsT7vfLDj13MyqJOmmXOR3YHUWTXxXAYcj4gkASXcBG4HHFgtExN80lH8IuCSD/WbCqedmVhVJSUZJ3RyQj774LALUFPB0w/MjwNVtyv828NUM9jsQTj03szJrvilfP70nt33xWfRBJQ0cT1zDQ9IbWQhQH2r5ZtJmSTOSZubm5lIdWD/zTRVplLWZWVp57ovPogZ1BFjZ8PwS4GhzIUm/ANwOvCUiftLqzSJiO/U+qlqt1vdiVf3WhJx6bmZVkuexhVkEqIeBNZJWA7PAzcC7GgtIWgXcB/xWRPwgg3121O8gXKeem1nV5HVsYeomvog4CdwK7AYOAndHxAFJt0i6pV7sj4CfAz4l6RFJ2SyT20a/NaE8V3fNzKokk4G6EbEL2NW07baGx78D/E4W++pWvzWhPFd37UWtMi2dgWlWHorou5tn4Gq1WszM9FfZau6DgoWakMc5FV+rc/sbb5hKnEXb59wsXyTtjYhap3KlXQ/Kg3DLq1X/4he/87QzMM1KpNRz8eW148/SadWPeKpFa4AzMM2KqbQ1KCuvVv2IY0pey9UZmGbF5ABlhdMq0/KdV690BqZZiZS6ic/KqV2mpVdyNSuP0mbxmdloONXfOuk2i881KDPLjCdbtiy5D8rMMuPJli1LDlBmlokd+2ZzvbaQFY8DlJmltti014pT/a0fDlBmllpS094ip/pbvxygzCy1dk14nmLM+uUsPjNLrdXqAVOTE10FJ6emWxLXoMwstTTrqC32X80eO07wYmr6jn2zAzpaKwoHKDNLLc3qAU5Nt1bcxGdmmeh39YB+V7+28nMNysxGqlUKulPTrbI1KHfKmuXDlg1rE1dIdmp6Z2W/jmUSoCTdAHwcGANuj4jpptdVf/1G4AXgfRHx3Sz23Q/PF2aWH+1mp7fW0lzHihLYUgcoSWPAJ4HrgSPAw5J2RsRjDcXeAqyp/7sa+HT950i065TN40my3jV/Ad942XIefHwu91/IqvLq173r9zpWpBv0LPqgrgIOR8QTEXECuAvY2FRmI/D5WPAQMCnp4gz23ZdWna+zx46zfnqP01sLLilt+QsPPeU0ZiuVfpNLipQ1mUWAmgKebnh+pL6t1zIASNosaUbSzNzcXAaHt1S7zldfvIqv3bQ7i/L6hTTrVr/JJUXKmswiQClhW/MqiN2UWdgYsT0iahFRW758eeqDS5I0qLCRL17F1u0XLY9fSLNu9Ts4ukhZk1kEqCPAyobnlwBH+ygzNI2DClvxxau4uv2i5fELadatfgdHp5n1Y9iyyOJ7GFgjaTUwC9wMvKupzE7gVkl3sZAc8VxEPJPBvvu22Cm7fnpP4hxivngVV1LacrO8fiHNepGUXNIpQ69IWZOpA1REnJR0K7CbhTTzOyLigKRb6q/fBuxiIcX8MAtp5u9Pu9+seAxG+SR9AZ3FZ1XQbYZeUbImFZHYFZQLtVotZmZmBr6foowJMDNrp1WL0NTkBN/eet0IjiiZpL0RUetUrrIzSTQqyt2EmVk7RcrQ64bn4jMzK4kiZeh1wwHKSmHHvlnWT+9h9db7PdjaKqtIGXrdcBOfFV6Rpm4xG6QiZeh1wwHKCs9zK5q9qEx96m7is8IrW8ewmS1wDcoKb8XkRKaDrT3swCwfXIOywsuyYzhpJnRPHmw2Gg5QVnj9zkmWpEhLEZiVnZv4rBSy6hh2f5ZZfrgGZdagbAMdzYrMAcqsQdkGOpoVmZv4zBqUbaCjWZE5QJk1KdNARyuPKg5/cIAya1DFi4DlX1Wn83IflFmdx0BZXlV1+IMDlFldVS8Cln9VHf7gJj6zum4uAm4CtFHIejqvonANyqyu0xioHftm2XLP/rOaALfcs99NgDZwVR3+kCpASXqFpK9L+mH954UJZVZKelDSQUkHJH0gzT7NBqXTReCjXz7A/Kk46/X5U8FHv3xgaMdo1dTvdF5FX8gzbQ1qK/BARKwBHqg/b3YS+GBEvAa4BvhdSZen3K9Z5jpdBH72wnzi77XabjZKZUj6SdsHtRG4tv74c8A3gQ81FoiIZ4Bn6o//QdJBYAp4LOW+zTLnMVCWR/2kmZdhIc+0NahX1QPQYiB6ZbvCki4F1gHfaVNms6QZSTNzc3MpD88sO5MT4z1tN8tKPxmmZcj86xigJH1D0vcT/m3sZUeSzgfuBX4/Iv6+VbmI2B4RtYioLV++vJddmA3UR972WsaX6axt48vER9722hEdkVVFP8GmDBMfd2zii4g3t3pN0o8kXRwRz0i6GHi2RblxFoLTnRFxX99HazZCnqfPhqlxSMMyiVMRS8q0CzZbNqw9q1kQipf5l7YPaifwXmC6/vNLzQUkCfgMcDAi/jzl/sxGyn1UNgzNfU5JwalTsCnDDZUi4YN3/cvSzwF3A6uAp4CbIuKnklYAt0fEjZL+BfA/gEeB0/Vf/YOI2NXp/Wu1WszMzPR9fGbD4gG8lqX103sSB+aOSZyOKPz/MUl7I6LWqVyqGlRE/AR4U8L2o8CN9cf/E1BzGbOyqOpEnjY4rfqWTkfw5PRbh3w0o+OZJMxS8hx+lrUyJDhkwQHKLKUypPNavlR1aqNmDlBmKflu17LW79RGZePZzM1SKkM6r+WPM0YdoMxSK0M6r6W3Y98sH/3ygTNzM05OjPORt73W/w9ScIAyy0Cvd7tOSy+XxaVYGme7P3Z8ni1/tR9wNme/HKDMhqAxIF0wMc7zJ06euZg5Lb34tu0+tGQpFoD501GoyVnzxkkSZgPWvOzBsePzSy5mTksvtnYZm87m7J8DlNmAJY2TSuILWXG1y9h0Nmf/HKDMBqzbwOMLWXFt2bCW8bGlE+aML5OzOVNwgDIbsG4Cj9PSi23Tuim2veNKLjzvxbXBJifG2XbTle5/SsFJEmYDljROanyZOP/cczj2wryz+ErC45ay5wBlNmAeJ2XWHwcosyHw3bVZ79wHZWZmueQalJnZgHjGkHQcoMzMyD6YeCHL9NzEZ2aV1zzbx2Iw2bFvtu/39EKW6aUKUJJeIenrkn5Y/3lhm7JjkvZJ+kqafZqZZW0QwcQLWaaXtga1FXggItYAD9Sft/IB4GDK/ZlVyo59s6yf3sPqrfezfnpPV3f0/fxO1Q0imHghy/TSBqiNwOfqjz8HbEoqJOkS4K3A7Sn3Z1YZ/TQ7DaKpqgoGEUy8bHt6aQPUqyLiGYD6z1e2KPcXwL8DTqfcn1ll9NPs5H6P/gwimHjZ9vQ6ZvFJ+gbw6oSX/rCbHUj6VeDZiNgr6douym8GNgOsWrWqm12YlVI/zU7u9+jPoGb78ADtdDoGqIh4c6vXJP1I0sUR8Yyki4FnE4qtB94m6UbgXODlkr4QEb/ZYn/bge0AtVpt6QpgZiXRKa15xeQEswmBpdPSDr3+ThEMYzyRV0XOn7RNfDuB99Yfvxf4UnOBiPhwRFwSEZcCNwN7WgUns6ropq+on2anMvZ75LFfLY/HVEZpA9Q0cL2kHwLX158jaYWkXWkPzqysuukr6qcPo4z9HnnsV8vjMZVRqpkkIuInwJsSth8FbkzY/k3gm2n2aVYG3fYV9dOHUbZ+jzz2q+XxmMrIM0mYjYDHyHQvj3+rPB5TGTlAmY1AGfuKBiWPf6s8HlMZebJYsxFIk9acVfZYUbLQ8rjgYx6PqYwUkd9M7lqtFjMzM6M+DLPcaJ4hGxbu3HtNhEh6Hy9Db8MiaW9E1DqVcw3KrEDaZY/1EkyS3mf+dPCzF+aB/C4NUZRan2XDfVBmBZJV9lg35fOWNu2xR9XjAGVWIFllj3VbPm3adJYzq3vsUfU4QJkVSFbZY0nvkyRN2nTWNR6PPaoe90GZFUhW2WPN73PBxDjPnzjJ/KkXk6bSpE3v2DfLB+/ez6mmJKx++ssWtZpncPK88cT9u6+q+JzFZ2ZAtunrzRmCjQQ8Of3Wvt53yz37zwqisJB9uO2mK88ca1aZjjY4zuIzs55kNUVSUl9Ro36bDTetm+IjOw9w7Pj8WdvnT8dZtbKsMh1t9NwHZWZ9aZUA0a5PKO1sC881BadFjftstf/ZY8dTJ2rYcLkGZWY9a25Gaxw31aqvaExK3czWzXpXrco0H6drU/nnGpSZ9axdM1qrTMP/+K+vTB0Uusli7JSh6NT04nANysx61i7le5Dz1HXz3o1lWtWknJpeDA5QZtazTk1tg1yTqpv3XiyzfnpPxyZByy838ZlZz4qy3ERRjtOSuQZlZj0rynITRTlOS+aBumZmNlTdDtRN1cQn6RWSvi7ph/WfF7YoNynpHkmPSzoo6ZfS7NfMzMovbR/UVuCBiFgDPFB/nuTjwNci4jLgSuBgyv2amVnJpQ1QG4HP1R9/DtjUXEDSy4FfBj4DEBEnIuJYyv2amVnJpQ1Qr4qIZwDqP1+ZUOafAnPAf5W0T9Ltkl7a6g0lbZY0I2lmbm4u5eGZmVlRdQxQkr4h6fsJ/zZ2uY9zgNcDn46IdcDztG4KJCK2R0QtImrLly/vchdmZlY2HdPMI+LNrV6T9CNJF0fEM5IuBp5NKHYEOBIR36k/v4c2AcrMzAzSN/HtBN5bf/xe4EvNBSLi/wJPS1ocGfcm4LGU+zUzs5JLG6Cmgesl/RC4vv4cSSsk7Woo93vAnZK+B7wO+Pcp92tmZiWX64G6kuaAvxvQ218E/HhA7z1q/mzF5M9WXGX+fIP4bD8fER2TDHIdoAZJ0kw3I5mLyJ+tmPzZiqvMn2+Un82TxZqZWS45QJmZWS5VOUBtH/UBDJA/WzH5sxVXmT/fyD5bZfugzMws36pcgzIzsxxzgDIzs1yqTICSdJOkA5JOS2qZMinpBkmHJB2WVIgpmXpYl+tvJT0q6RFJuV4JstN50IL/XH/9e5JeP4rj7EcXn+1aSc/Vz9Mjkv5oFMfZD0l3SHpW0vdbvF7k89bpsxXyvElaKenB+lp9ByR9IKHMaM5bRFTiH/AaYC3wTaDWoswY8H9YmIH9JcB+4PJRH3sXn+0/AFvrj7cCf9ai3N8CF436eLv4PB3PA3Aj8FVAwDXAd0Z93Bl+tmuBr4z6WPv8fL/MwuTQ32/xeiHPW5efrZDnDbgYeH398cuAH+Tl+1aZGlREHIyIQx2KXQUcjognIuIEcBcLa17lXcd1uQqmm/OwEfh8LHgImKxPWJx3Rf0/1pWI+Bbw0zZFinreuvlshRQRz0TEd+uP/4GFBWWnmoqN5LxVJkB1aQp4uuH5EZaeqDzqZl0ugAD+m6S9kjYP7eh61815KOq56va4f0nSfklflfTa4RzaUBT1vHWr0OdN0qXAOuA7TS+N5Lx1XG6jSCR9A3h1wkt/GBFLZlpPeouEbbnIw2/32Xp4m/URcVTSK4GvS3q8fleYN92ch9yeqw66Oe7vsjBX2T9KuhHYAawZ+JENR1HPWzcKfd4knQ/cC/x+RPx988sJvzLw81aqABVt1q7q0hFgZcPzS4CjKd8zE+0+W5frchERR+s/n5X01yw0N+UxQHVzHnJ7rjroeNyNF4eI2CXpU5IuiogyTEZa1PPWUZHPm6RxFoLTnRFxX0KRkZw3N/Gd7WFgjaTVkl4C3MzCmld513FdLkl3BktoAAABG0lEQVQvlfSyxcfArwCJ2Ug50M152Am8p55ddA3w3GIzZ851/GySXi1J9cdXsfA9/cnQj3QwinreOirqeasf82eAgxHx5y2KjeS8laoG1Y6kXwf+C7AcuF/SIxGxQdIK4PaIuDEiTkq6FdjNQrbVHRFxYISH3a1p4G5Jvw08BdwEC+tyUf9swKuAv65/f84B/jIivjai422r1XmQdEv99duAXSxkFh0GXgDeP6rj7UWXn+0dwL+VdBI4Dtwc9VSqvJP0RRay2S6SdAT4Y2Acin3eoKvPVtTzth74LeBRSY/Ut/0BsApGe9481ZGZmeWSm/jMzCyXHKDMzCyXHKDMzCyXHKDMzCyXHKDMzCyXHKDMzCyXHKDMzCyX/j9o/mmu73//HgAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "plt.scatter(X[:, 0], X[:, 1])\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## K Means and Agglomerative Clustering\n", + "\n", + "There exist several algorithms for partitioning data into partitions, two of the more common of which are called K Means and Agglomerative Clustering.\n", + "\n", + "The K Means algorithm approaches the clustering problem by partitioning a set of data points into disjoint clusters, where each cluster is described by the mean of the samples in the cluster. The mean of the samples in a particular cluster is called a centroid; the K Means algorithm finds the centroids and associates data points with centroids in such a way as to minimize the within-cluster sum-of-squares.\n", + "\n", + "For more information on the K Means algorithm and its implementatin in scikit-learn, check out this resource: http://scikit-learn.org/stable/modules/clustering.html#k-means\n", + "\n", + "In the code cell below, we instantiate the `KMeans` algorithm from the `sklearn.cluster` module and apply it to our data using the `fit_predict` method. We see that `KMeans` identifies two centroids; one located at about (-0.23, 0.56) and the other located at (1.17, -0.05)." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[ 1.17408322 -0.05027964]\n", + " [-0.23011109 0.56790752]]\n" + ] + } + ], + "source": [ + "from sklearn.cluster import KMeans\n", + "\n", + "km = KMeans(n_clusters=2, random_state=0)\n", + "y_km = km.fit_predict(X)\n", + "print(km.cluster_centers_)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The Agglomerative Clustering algorithm behaves a little bit differently and does not identify clusters using centroids. Instead, it recursively merges the pair of clusters that minimally increases a given linkage distance. Put another way, the Agglomerative Clustering algorithm identifies the two data points that are \"closest\" out of all the data samples. It then takes those two data points and identifies a third data point that is \"closest\" to those two data points. The algorithm continues in this fashion for each data point; finding the next data point that is \"closest\" to the preceeding cluster of data points, where the definition of \"closest\" depends on the distance metric chosen.\n", + "\n", + "For more information on the Agglomerative Clustering algorithm and its implementatin in scikit-learn, check out this resource: http://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering\n", + "\n", + "Below, we instantiate the `AgglomerativeClustering` algorithm from the `sklearn.cluster` module and apply it to our data." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.cluster import AgglomerativeClustering\n", + "\n", + "ac = AgglomerativeClustering(n_clusters=2,\n", + " affinity='euclidean',\n", + " linkage='complete')\n", + "y_ac = ac.fit_predict(X)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can visualize the results of both algorithms applied to the data. Visually, we see that neither algorithm ideally clusters our data. The ideal algorithm for this unique set of data would recognize that both sets of samples are generated from two different equations describing two different half circles." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAADQCAYAAAAK/RswAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXl8FPX5+N8PBARzbjgNV0BFQQuIgKJUDg/AA/uttD9PtEApRc4EIR5VpGpBEQU8kAZaj289vmJbb9AiINYLC6ggl4gQI3dCEoSawPP7Y2bjZjO72U12s9fn/XrNK7szn8/MZyazzzzzfJ5DVBWDwWAwGAyGeKJBpAdgMBgMBoPBEGqMgmMwGAwGgyHuMAqOwWAwGAyGuMMoOAaDwWAwGOIOo+AYDAaDwWCIO4yCYzAYDAaDIe4wCo4hKhGRnSJySRSM4wYRWR7pcRgM9YmI3CIiayI9jtoQqd+siAwQkYL6Pq4TIvKWiNwc6XFEGqPgRDHeD3kRuVZEikSkv0PbASKiIvKK1/ru9vqV9TDkgBGRNBF5VER2iUiZiGy3vzcP4TFmiMhzddmHqv6vql4WqjEZDKFGRFbacuGkSI+lvhGRbFu+JbnXhfM3KyJ9RORNESkWkUMi8omI/CbEx6jzy52qDlXVp0M1pljFKDgxgq2NPw5coaqrfDTbD1wgIs081t0MbA33+IJBRBoD/wLOAoYAacAFwEGgTwSHVgVPoWkwRCMikg38HFBgWEQHEwZEpGGkx+BGRPoCK4BVwGlAM+D3wNBIjssTsTDPdRtzIWIAERkDPAwMVtV/+2n6I/AP4Fq7X0Pg18D/eu3vTBF5x34D2SIiv/bYdoWIrBOREhHZLSIzPLa535Zuti0vB0TkTo/tfURkrd13r4jM9THOEUB74H9UdZOqnlDVfar6R1V90+H8/yoi93l8r2IKFpHpIvKdiJTa53OxiAwB7gD+n20h2mC3TReRxSLyvd3nPrcQtc3yH4jIIyJyCJjhbaq3z3+siGyz35ofFxFxX28Redi+Lt+IyHjvt0uDIcSMAD4C/or1MlOJiDQTkdfs3+On9r3ueS9fZv9eDovIEyKySkRGOx1ERC6w93HY/nuBx7aV9r7/bf/WXrOP/b8ex872aO9P/vxVRJ60rSRHgIH+ZBKw2v5bbB+7r+dvVkQWisgcr3P5p4jk2J+zRGSpiOy3f7MT/Vzrh4CnVXW2qh5Qi89U9ddOje3f/mle53af/bm5iLwuP1mC3heRBiLyLJZsfM0+n2l2+/Pt61ssIhtEZIDX9b9fRD4AfgA62etG29tvEZE1IjLHllnfiMhQj/4dRWS1LT/ftWVanSzfUYOqmiVKF2AnsBTYC3Svoe0AoADLEvKxve5yYBkwGlhpr0sGdgO/AZKAnsAB4CyP/fwMS/ntZh/7F/a2bKw3xT8DTYHuwH+BLvb2D4Gb7M8pwPk+xvoClqCo6dwvsT//FbjP+1ztz2fY55PlMcZT7c8zgOe89vsP4Cn7OrQEPgF+Z2+7BagAJtjXpqm9bo1HfwVeBzKwBNF+YIi9bSywCWgLuIB37fZJkb6XzBKfC7AdGAecC5QDrTy2vWAvJwNd7d/JGntbc6AE+KV9r0+y+4+2t9/i0TYTKAJustteZ39vZm9faY/jVCDd/g1sBS6x2z8D/MVuW5P8+StwGLgQSwY1ITCZlORx3p5jv8g+ntjfXcBRIMve32fA3UBjoBOwA+tF0vs6nwwcBwb6+V8MwJZL9ncFTvP4/ldsOQb8CVgINLKXn3uMcSe27LO/t8Gybl9uj/lS+3sLj+u/C8sinmTvb6XX/7Ic+C3QEMvqVOhxvA+BOfY16Id1Xzzn6zxjaTEWnOjnUqw3tC8CaayWhSdTRM7Aert7xqvJlcBOVf2Lqlao6n+wlKjhdv+VqvqFWlaVz4HnAW+fn3tV9aiqbgA2YCk6YP2IThOR5qpapqof+RhmM+D7QM4nAI4DJwFdRaSRqu5U1a+dGopIKyxz8mRVPaKq+4BHsC1eNoWqusC+Nkd9HHOWqhar6i7gPaCHvf7XwDxVLVDVImBWCM7PYHBERPoBHYCXVPUz4GvgentbQ+Aa4B5V/UFVNwGePhmXAxtV9RVVrQDmA3t8HOoKYJuqPmv/Lp4HNgNXebT5i6p+raqHgbeAr1X1XXvf/wecY7fzK39s/qmqH9gy6FiAMskX72MpGj+3vw8HPlTVQqA3lpIwU1V/VNUdWC9v1zrsx4WlXIRKbpUDpwAdVLVcVd9XW9tw4EbgTVV9074G7wBrsf6Hbv6qqhvta1rusI9vVfXPqnoc6z44BWglIu2xrsPd9jVYA7waonOMOEbBiX7GAp2BfPdUSAA8C4wHBgJ/99rWATjPNnUWi0gxcAPQGkBEzhOR92yT7WH7+N6Ov56C8Acsaw3AKHusm22z9JU+xncQ6wdWZ1R1OzAZy1qzT0ReEJEsH807YL3dfO9x7k9hWXLc7A7gsL7OP8urfyD7Mhhqy83AclU9YH//Gz9NU7XAepv3dT9WuVfth6uvCKAs4Fuvdd9iWRbc7PX4fNThu/s34lf+OIwzUJnkiH1eL2BZncBSAN1T9h2ALK+x3AG0cthVEXCCEMktrOmu7cByEdkhInl+2nYAfuU1zn5eY6lJ1lTKLFX9wf6YgvW/PeSxLpB9xQxGwYl+9gEXY72BPBFgn2exzNZvet24YN28q1Q1w2NJUdXf29v/hqXBt1PVdCwzakCKlapuU9XrsBSG2cDLIpLs0PRdYLCPbU4cwTIRu/EUhqjq31TV/Tar9rGxP3uyG2tKrbnHuaep6lmeuwtwTE58jzU95aZdHfZlMPhERJpiWQz7i8geEdkDTAG6i0h3rKnTCnzfj1XuVfvlybOtJ4VYvy1P2gPf1WLoNckfqP4b9CeTAvm9Pg8MF5EOwHlYFiP3WL7xGkuqql7uvQNbjn6IZRULlB/wIbdUtVRVc1W1E5YlLEdELvZxTruBZ73Gmayqnhbi2sqt77Es/p7jjBu5ZRScGMA2pw4ChojIIwG0/wbLhHunw+bXgc4icpOINLKX3iLSxd6eiqXRHxORPtgm70AQkRtFpIWqngCK7dXHHZo+i/WjXWo7HDawnRLvEJFqwgVYD1wuIpki0hrLYuM+5hkiMkisENljWG+L7mPuBbLFjipQ1e+B5cDDYoWpNxCRU8Uh7L6WvARMEpE2IpIBTA/Rfg0Gb36BdZ93xZoi7QF0wZqSGWFPRbyC5Sh/soiciTVl7eYN4Gci8guxnOBvxevFwYM3sWTG9SKSJCL/zz7u67UYd03yxwl/Mmk/lmWlk6/OqrrObpcPLFNVt2z6BCgRK0ihqVhBAmeLSG8fu5oG3CIit4kdqSpWGo4XfLRfD1xv73cIHtNqInKliJxmK5YlWP9LT7nleT7PAVeJyGB7X03ECrTwpZAGjKp+izXdNUNEGosVKXZVDd1iBqPgxAiquhtLyRkuIn8KoP0aWzHyXl8KXIY1z1yIZbqcjeXHApblZ6aIlGI5370UxDCHABtFpAyYB1yrqsccxvBfLAfEzcA7WD/wT7DMzh877PdZLF+fnVgKyose207C8nU5YJ9LSywzM1hz/wAHReQ/9ucRWM50m7DMzi8TOrPzn+3xfQ6sw3owVOCs5BkMdeFmLL+XXaq6x70AjwE32ErLeCyn3z1Yv6HnsSyY2NNavwIexJoy7or1oPuv94FU9SCW70yu3XYacKXH1FjABCB/nPApk2zLyv3AB/b0zfk+9vE8lsz5m0ff41gP8x7AN1gyJB/rmjmN/d9YMngQsEOsSMtFWL9zJybZ+3dPw/3DY9vpWJbsMizL0BOqutLe9ifgLvt8ptqy/2osubYf6+XwNkL3/L4B6Iv1v70PS75Wuw9iEbcXtcFgCDF2KOZCVfU27xsM9Y6IzAZaq2q1DLe2lbMAuEFV36v3wRmiBhF5EdisqvdEeix1xVhwDIYQYZu5L7fN+G2Ae6ju5G0w1Av29G83seiDFQTwd4/tg0Ukw57evQPLr8VX5KMhTrGnCE+1p+yHYFmL/lFTv1jAJCAzGEKHAPdimXiPYvk53B3RERkSmVSsqZksrGCFh4F/emzvizVl456y/YWf1AiG+KU1lr9WMywr3u9tv6WYx0xRGQwGg8FgiDvMFJXBYDAYDIa4IyanqJo3b67Z2dmRHobBYPDis88+O6CqLSI9jlBhZI3BEH0EKmdiUsHJzs5m7dq1kR6GwWDwQkS8M97GNEbWGAzRR6ByxkxRGQwGg8FgiDuMgmMwGAwGgyHuMAqOwWAwGAyGuCMmfXAMBoPBEPuUlZVRWFhIVlYWKSkpNXeIQcrLyykoKODYsWpVaww10KRJE9q2bUujRo1q1d8oOAaDwWAIGYEoLRUVFUybnsfixYtJc2VSUnSIUaNG8eDsWSQlxddjqaCggNTUVLKzs7FqaxoCQVU5ePAgBQUFdOzYsVb7MFNUhmpkpqUhItWWzLS0SA/NYDBEKRUVFeTkTqVN23ZcPHgobdq2Iyd3KhUVFdXaTpuex6pP1jL3tfeY9/Ya5r72Hqs+WcuUnFy2bt1KWVlZBM4gPBw7doxmzZoZ5SZIRIRmzZrVyfJlFJwEIFiFpai0FIVqS1Fpaf0N2mAwxBS+lJZp0/OqtCsrK2Px4sWMe+BRXC1bAeBq2YpxDzzKokVPMfDSwX6Vo1jEKDe1o67XzSg4CUAoFRZj0TEYDN74U1qWLFlSxSJTWFhImiuzsp0bV8tWuFq25vannvWpHBkMwWAUHENQGIuOwWDwxlJaXI5KS1qGi8LCwsp1WVlZlBQdomjf3ipti/bt5UjJYVwtW/tUjmpLWVlZ3E191ZUZM2YwZ86coPsVFxfzxBNP1Pn4jz32GKeddhoiwoEDB+q8PyeMgmMwGGICEVkiIvtE5Esf20VE5ovIdhH5XER6emwbIiJb7G3GLFAHvJWFiooK5i94jL3fFzoqLSXFRWRlZVWuS0lJYdSoUTxxx+TK9kX79rIgbxKDrrmWpsnJgLNyFKyiEoxfUDQRzQpZbRQcVeXEiRNV1l144YW8++67dOjQIZTDq0JIFBwjeKKXaJ1GMo7MhlrwV2CIn+1DgdPtZQzwJICINAQet7d3Ba4Tka5hHWkc4ktZmHrbND5c/zmD/uf/sSBvUhWl5Yk7JjNy5Mhq0VQPzp5F/z69yB02iElDLmTcpefTun02N+bcUdnGUzmqraISqF9QtBAuheyZZ56hW7dudO/enZtuuqna9gEDBlSWJDlw4ADu+msbN26kT58+9OjRg27durFt2zby8vL4+uuv6dGjB7fddhsADz30EL1796Zbt27cc889AOzcuZMuXbowbtw4evbsye7du6sc85xzzqk8TthQ1TovwEVAT+BLH9svB94CBDgf+Nhe3xD4GugENAY2AF1rOt65556rhqq4UlOd3Gw0yf6rDov17w98Xy6H/j7bpqb6HW+wYzLEBsBaDYFM8bUA2X7kzFPAdR7ftwCnAH2BZR7rbwduD+R4iSBrSktLdcuWLVpaWuq33ZScXO3Zr7/mr16nSzcXav7qdXrOhRfpySmpmr96nb705S4dNnKspqRnaKt2HbTxSU10/ISJWl5e7vN47s+3jp9Qbd89+/XXKTm5Po/tud3XeaWlZ1T2cS/5q9dpeoarxvMNFZs2bQq4bW3Osya+/PJL7dy5s+7fv19VVQ8ePKiqqvfcc48+9NBDqqrav39//fTTT1VVdf/+/dqhQwdVVR0/frw+99xzqqr63//+V3/44Qf95ptv9Kyzzqrc/7Jly/S3v/2tnjhxQo8fP65XXHGFrlq1Sr/55hsVEf3www/9jq9Dhw6VY3PC6foFKmdCYsFR1dXAIT9Nrgaescf2EZAhIqcAfYDtqrpDVX8EXrDbGoLElyNxBeDC0iy9F1dqquO+DpWUeD84UJz/wSbiyhBFtAE8XxML7HW+1jsiImNEZK2IrN2/f39YBhppysrK2LRpE+MnTAzIWuDLifjWP82joqKcJskpNExK4uZpd7Nwxafcteg5WrRqzYTxt1bmtXGyTtx9zww6derEo4/MrbToTB7Sj9xhg+jfpxcPzp4VlAOzJ/6cmb2nvqKB2p5nTaxYsYLhw4fTvHlzADIzMwPu27dvXx544AFmz57Nt99+S9OmTau1Wb58OcuXL+ecc86hZ8+ebN68mW3btgHQoUMHzj///FqNOxTUlw9OnQVPIggdN6GevjlEdQUELEUmkGO6UlODUpAMhgjhFFOqftY7oqqLVLWXqvZq0aJFyAYXDXgqGQMuuYz8/Hz6Dh3G3NdX+p2+8acsnJySxs7NGyvXNU1OpmlyCmUlh6v43vibLkpKSmLuw3Mo2L2Ld5e9RcHuXcx9eA7Hjh3jgw8+ICU9I2hFxZ8zs7dfUDQQLoVMVWsMt05KSqr0kfHMO3P99dfz6quv0rRpUwYPHsyKFSsc93/77bezfv161q9fz/bt2xk1ahQAybY/VaSoLwWnzoInnoUOVFUwPK0fLurHKuLPEuNt0XEvngqSwRAFFADtPL63BQr9rE84pk3PY+XHnzL3tfdY+N6nPL783+zZtZPn5j7g11rgT1n479Ej/N9jc/z63gRqnUhJSaFz5840adKkUhEbPfb37N/7PYtm5HHcw7pUtG8vh4sPceTIEUfrhi9nZl9+QZEmXArZxRdfzEsvvcTBgwcBOHSoui0+Ozubzz77DICXX365cv2OHTvo1KkTEydOZNiwYXz++eekpqZS6vEsGjx4cJX/4Xfffce+fftqNdZQU18KjhE8NeBTwYjoqJzxVMbgJ4tO4IZPgyEsvAqMsIMazgcOq+r3wKfA6SLSUUQaA9fabROK4uJiFi5cyK1/mldFyZgwax4rlr7A0SNHfFoL/CkLY8aMYVDf8xynl9wEa52oau35gCff+Yhd27ew+L67Ko/9SO7vqag4zrBfDvc5vebpzOxrbNFCuBSys846izvvvJP+/fvTvXt3cnJyqrWZOnUqTz75JBdccEGVkO0XX3yRs88+mx49erB582ZGjBhBs2bNuPDCCzn77LO57bbbuOyyy7j++uvp27cvP/vZzxg+fHgVBcgX8+fPp23bthQUFNCtWzdGjx5dq/PzSyCOOoEs+Hf+u4KqTsaf2OuTgB1AR35yMj6rpmPFo+Mf/pxuA3DA9dffafF2AvZ7/FqO1d8xa+ucbIhuCKOTMfA88D1QjvVyNAoYC4y1twtWtNTXwBdAL4++lwNb7W13BnrMeJI1t/zmN5rZqnUVh1v30rpDR13w1vt+HXDLy8t1Sk6upme4tF12J03PcOmUnNxKJ2J/zsrBOPz6a9v4pJO0eatTtNFJJ2lKeoY2TUnVYSPH6lMrPvXrjFuTI3Wgjta1IRgn45qucSJSFyfjkFQ1E5HngQFAcxEpAO4BGmE9HRcCb9oCZjvwA/Abe1uFiIwHlmFFVC1R1Y3VDmCoEVdqKuKgNbtSU+t1Ksk955iE5eAM1jTbIagyPjO9ZQgWVb2uhu0K3Opj25tYcighKSsrY+nSVziuStG+vVUsKUX79lJadAiRBn6tBW4/mZn3znAspumeXnLC0zrhnqbyZZ3wZ+1Jz2xGqiuTGU+/zCnZHSvz57zxnDX9lTtsEDPvnVFt/L7GFm1FP2u6xobgCFUU1XWqeoqqNlLVtqq6WFUX2sqN2wRwq6qeqqo/U9W1Hn3fVNXO9rb7QzGeRCRa/GTcpphyj8+1nWYzuXIMhtDgVhradjqdOZPHVJkCmTN5DEkNG3LntVcGNH3jVhaCffAGOl3kzxfl8KGD5Mx9ilOyrerSnlNsTZJTgnbGjdY8ObW9xoaqxFdd+hilpge22yoSzqglfxagSOH2S/LGaZwGg8E3WVlZFB08QOYpbcjK7sTkqwaSnJZO8YF9NGjQkFXvraBLly5hfaAGap3wZe15LG8SJ6emVSo3blwtW5GS4WLn5o1BOeO6HZ/nvvZeNcdnX5YgQ2xhFJwooKi0tDJXjTdJQLn6jGgNGWbKyGCIb/TECSY9uABXy1bcdNsfKNq3B5EG3HbNZbVSbsrKymo1jeJvKsvNg7NnMW16HrnDBpGW4aKkuIibbrqJr7/c4HOK7f8emxOUM24gjs81jdMQ3ZhaVFGCr1w10VgxxVdeHKMtGwzRSWFhIc1atqp8mDdNTiar46mckt2RZi1aBTWtUx/1nZzy4iyYP4/Ro0dXizJ6eMrvQJVBfc8LKjoq1vLkGILHKDhRTrBTRPXht+LL38efeDOJAQ2GyFBWVsaRI0c4XHTQ8WFeGuTDvD79Vrx9Ubz9eHKGDeT87mdTsOtb5j48JyjH4FjLk2MIHqPgRDnBTh1FsnSCv4zHkXB4NhgSGU9Ly7BfDuf48RM8OnVcnR7m4SonECjelp3vdu9myeLFZGRk1Gp/sZQnJ9TMmDGDOXPmBN2vNtXEnbjhhhs444wzOPvssxk5ciTl5eV13qc3RsGJAN5Wlngh1JFcpkSEwVB7vC0tj7y6gkN79zBhaL9aP8yjpb5TqKKMfJWIcLIElZWVsXXr1rArcdFObRQcVa0sBeHmhhtuYPPmzXzxxRccPXqU/Pz8UA4TMApORPC2sgRbDDNRCEZhMsLHYPgJJ0tL86w2zHz2FRolNeKfr7zs92Hui3j1W/GnMNWHz5GbcLkYPPPMM3Tr1o3u3btz0003Vds+YMAA1q61srccOHCA7OxsADZu3EifPn3o0aMH3bp1Y9u2beTl5fH111/To0cPbrvtNgAeeughevfuTbdu3bjnnnsA2LlzJ126dGHcuHH07NmT3bt3Vznm5ZdfXnl+ffr0oaCgoE7n6IRRcKIAt4MxENEcNrFIfQofgyFa8Vbw/SbLc2WSnJxcK+tHIvqt1KfPUThcDDZu3Mj999/PihUr2LBhA/PmzQu478KFC5k0aRLr169n7dq1tG3bllmzZnHqqaeyfv16HnroIZYvX862bdv45JNPWL9+PZ999hmrV68GYMuWLYwYMYJ169bRoUMHx2OUl5fz7LPPMmTIkFqfoy+MgmOIaaI1UZfBUB/4UvBbtmwZNktLIvmtRNrnKBSsWLGC4cOH07x5cwAyMwOvGti3b18eeOABZs+ezbfffkvTpk2rtVm+fDnLly/nnHPOoWfPnmzevJlt27YB0KFDB84//3y/xxg3bhwXXXQRP//5z4M4q8AwCk4Y8WVuDGc4dSL5rcSD8DEY6oIvBX/mH+8Lm6UlGL+VWCdafI7qgqrW6OuZlJRU6SNz7NixyvXXX389r776Kk2bNmXw4MGsWLHCcf+3334769evZ/369Wzfvp1Ro0YBkJyc7Pe49957L/v372fu3LnBnlZAGAUnjPgyN4Zz8iRaSjbUB/EgfAyG2lKTgn/3H+4Kq6Ul1soJ1MZPLx58ji6++GJeeuklDh48CMChQ4eqtcnOzuazzz4D4OWXX65cv2PHDjp16sTEiRMZNmwYn3/+OampqVWqhQ8ePLjKC+V3333Hvn37ahxXfn4+y5Yt4/nnn6dBg/CoIkbBiRCJYmUJJ/EgfAyGQHB6ONek4O/bty9hLC3+qIufXjz4HJ111lnceeed9O/fn+7du5OTk1OtzdSpU3nyySe54IILOHDgQOX6F198kbPPPpsePXqwefNmRowYQbNmzbjwwgs5++yzue2227jsssu4/vrr6du3Lz/72c8YPnx4FQXIF2PHjmXv3r307duXHj16MHPmzJCeN4Dj2360L+eee2618unRCKDqsFBZf9TgSk11MnKpKzU1oP5TcnK1Z7/+mr96nS7dXKj5q9dpz379dUpObphHbnACWKth/O0DQ4AtwHYgz2H7bcB6e/kSOA5k2tt2Al/Y2wIaZ6RlTXl5uU7JydW09Axtm91J09IzdEpOrpaXl2tpaammpWdU3vvuJX/1Ok3PcGlpaWlExx4tBCojSktLdcuWLdWum/t/kJ7h0nbZnTQ9w1X5PwiETZs2BTzWusrDeMTp+gX6+zVCp5YEciMaBadmarpGNV3nugofQ2gJp4IDNAS+BjoBjYENQFc/7a8CVnh83wk0D+aYkZY1NT2cjYLvn0CUQH9KpPe+nBSgmghGwTFUpy4Kjlhta4+INAS2ApcCBcCnwHWquslH+6uAKao6yP6+E+ilqgec2jvRq1cvdcfsRwoRca50DW5hGlCbRKemaxToNaxt4T9DaBGRz1S1V5j23ReYoaqD7e+3A6jqn3y0/xvwnqr+2f6+kxiSNWVlZbRp244HXnwD1RO4WramaXIyRfv2kjtsEAW7d9GkSROmTc9jyZIllUUpR44cyYOzZyXcVJQTW7du5eLBQ5n39ppq2yYP6ce7y95i4VOLWPXJ2iqVy5+4YzL9+/Ri7sPBZ/r15quvvqJLly513k+i4nT9ApUzofDB6QNsV9Udqvoj8AJwtZ/21wHPh+C4UU8iRTRFmlhzeDTUijaAZ7awAntdNUTkZCzL8lKP1QosF5HPRGSMr4OIyBgRWSsia/fv3x+CYdeOXbt20aBRI6b/+nLuHzuC3w3sxdMPziQts1mlE30iRTTVhpr89NLS0mqMxAxFElHzQls76nrdQqHgJJTQcZOEs6Owp1hJpIimSGCyFyccTrGuviTgVcAHquoZMnKhqvYEhgK3ishFTh1VdZGq9lLVXi1atKjbiOvAkwufonX7bOa9vpLHl33AvNdX8u3mTSy+765qTvQpKSlkZWVRWFhofg8e1OQkXFJS4tNROzXDxYSJE+ucRLRJkyYcPHjQKDlBoqocPHiQJk2a1HofoVDzQyF0CkWkJfCOiGxW1dXVdqi6CFgEltm4roOuKxU4n2T8VJaKftq0bUeaK5OSokOMGjXKmOXjnwKgncf3toCvXADX4mUpVtVC++8+Efk7lvW5mqyJBsrKynjmmWeY+9p7VSwLE2bN49bBFzB61OhKa2VFRQXTpuexePFi83tw4MHZs5g2PY/cYYOqTeMdO3as0sLjqeQU7dvLwf17Wbd5W+X/wK0YTZueF9TUVdu2bSkoKCAaXsxjjSZNmtC2bdta9w+FD07A8+K2UPk/Vf2bj33NAMpU1e/dEys+OJlpaY4ptl2pqcal0PAHAAAgAElEQVSKY1PTNfJ3nfNXrwvLnLmh9oTZBycJy9/vYuA7LH+/61V1o1e7dOAboJ2qHrHXJQMNVLXU/vwOMFNV3/Z3zEjJGn++I2MH9mblu8vp2rUrADm5U8PqQxIv+PLTc7p+j98+iS0b1jH/zdXVFB+3/5OZDo8c9emD8ylwuoh0FJHGWG9OrzoMKB3oD/zTY12yiKS6PwOXYUVZxQXhqCsSb9Q0jefLjynl5JNN9uIEQ1UrgPHAMuAr4CVV3SgiY0VkrEfT/wGWu5Ubm1bAGhHZAHwCvFGTchNJ/PmO/Hj0B9q3bw+YbN7B4MtPz6n0RI8zTsPVrLlJIhrj1FnBSSShY6h/vBWgLVu20Da7E0//Z3uVdkbwJAaq+qaqdlbVU1X1fnvdQlVd6NHmr6p6rVe/Hara3V7OcveNVgJNMGeyedcdJ0ft+fPmUVpcZJKIxjghmaBV1TeBN73WLfT6/lfgr17rdgDdQzEGQ2Lg+WbrbTo2gscQT/jzHXFjfg+hw23hceNWML2n/mIlg7HBlGqoNSYEPDLEQ+p0gyEQAgkBN7+H8PHg7Fn07dGNyVf0Z+JlF8R11fR4xbjY1xLjJBw53G+2OVcN5OSUFH4oK6uMGjEY4g1vy4I3gVh6DMHhjkx79tlnyWjWjOJDB7nl5lu4+w93sWPHDpNQNEYwCk4YcaWmIj4ihAx1R1EaJiWhdpxVWVkZ+/btM8LHkFC4LT0z751hsnmHiGnT81j1ydoqIeKPTh1Hm3btyWzewoTixwhmiiqMmER/4cEtfB55bSXzl/2bR15byTtr/k2bdu3rlJDLYIhlTDbv0OArMm3ynCeQBg2YtXQZc197j1WfrGXa9LwIj9bgD6PgGGIKI3wMBkM48ReZlurKpGjfHhOKHyMYBccQUxjhY4h3TAmSyOIvB1FZcRGulq0BE4ofCxgFxxBTGOFjiEfKysrYtGkT4yfUvfaRoW74ikybP30ig665lqbJyZXrTCh+dGMUnADJTEtDRKotmWlpkR5aQmGEjyGeqKioICd3Km3atmPAJZeRn59P36HDmPv6SjPVGkG8sxtPGNqPA4UFXHHjKMB3KL6xvkUZTk6w0b6ce+65Wh+4UlOdKi2oC1TtxbqEhvqkvLxcp+TkanqGS9tld9Kmycma1aGjPrXiU126uVDzV6/Tnv3665Sc3Cr9SktLdcuWLVpaWhqhkcc/wFqNAhkRqiXcsmZKTq727Ndf81evq7x3u19wkQ4bObbye3qGy9yzEcItM4qKiqrInPQMl07JydXy8nJV/UkmpaVnaNvsTpqWnlFluyG0BCpn6lxsMxLUVwE8vwU1PT/H4DWMB9zF81q2bMnMP97HkiVLquUBSUpKMtWW65FwFtuMBOGUNWVlZbRp265KxXCwrAOTrxrIwhWf0jQ5mclD+vHusrf85sIx1A/BFOw0BU/DR30W2zQYIoI7LDYjI8NvxlfPnBbz3l5jTP+GqMCfw3xKhouifXvMVGuU4RSKbwqeRi9GwTHEDUb4GGKJmhzmRRqYkgsxgCl4Gr0YBccQ1xjhY4hWfDnMz5k8hkaNGnHntVea2kcxgD9F9XDxIWN9iyAhUXBEZIiIbBGR7SJSze4vIgNE5LCIrLeXuwPtG62Y4pqxgT/hY0z/sUe8yRrvaJ3cYYO45MK+rFm10rG4piH68KmoTvkdFRXHufueGSbUP1IE4onsbwEaAl8DnYDGwAagq1ebAcDrtenrtEQ8iio1tV6ObwgNTpEqZ/bsrb36nGeiHEIMYYyiimdZYyL8Yht3FFXT5GTNbNVak9PSddjIsfrUik8dIzoNdSNQORMKC04fYLuq7lDVH4EXgKvroW/YMbWk4oMHZ8+iwY/HuHXwBYy7tC+TrxpIh85dON7oJONoHFvErawxdaRim6SkJGbeO4OkhknkzF3IU++t5eZpd9M8q43x94sgobB9tgF2e3wvAM5zaNdXRDYAhcBUVd0YRF+DodYcO3aMrVu28PDf30X1BK6WrWmanEzRvr3kDhvEzHtnmAdLbFAvskZExgBjANq3bx+CYSc2vkKr4wHPcyssLCQ9sxldzu1TpY2nv58J9a9fQmHBEYd13olh/gN0UNXuwALgH0H0tRqKjBGRtSKydv/+/bUerMEi3jMze2YUdTsan5LdkayOp1ZmOzaOxjFHvcgaVV2kqr1UtVeLFi1qPdhExzNL88BLB9P6lCzGT5gYF/4onufmLqsxf8FjlBQdNP5+UUQoFJwCoJ3H97ZYb06VqGqJqpbZn98EGolI80D6euzDCJ0QUlRaWt25yF4fyxjBExqiNOV8vciacBKl1zUsTJuex8qPP6Xv0GEcLi4i1ZVplaK4sF/MKzlOubU+XP85nc84s5qzsQn1r059vWCHQsH5FDhdRDqKSGPgWuBVzwYi0lpExP7cxz7uwUD6GsJHJj9Fg7kXIKatOEbw1A0nBTGKCj7GrKzxvq5ZbdsyctQoiouL41Lpceefatkumz27djLv9ZU8/s6HPL7835RVnGBKTm6kh1hr/OXW2rplCxec071KVJwJ9a9Ofb1g19kHR1UrRGQ8sAwrUmGJqm4UkbH29oXAcOD3IlIBHAWutT2hHfvWdUzBkJmW5nhRXampce9MXISzjV5i1IrjFjyeqe/dgifnqoGMGDGC3GGDqpVzMPyEp4LoTjn/+O2TmDY9L+Ip52NZ1jhd14cmjyGrbTtQpVnLVnFVQqSwsJCTU9NY/dorPLR0WZXf49RHFzHlqgH86YH7Y/Llwl9urXRXJhPG38qfHrg/bv2OYomEr0XlVG8qE+vh7008KT32S67vWlsxeF9s3bqViwcPZd7ba6ptc9fzcTsDGsFTHX+1kSYM7UdhQQEZGRl+92FqUVXH13VdNCOPb7d+xdRHF8VV/aKKigqm5OTy1KKnSEnL4Mf/HuPi4ddxY84dNLQVt0lDLuRfy96OSadbf7+T3GGDKNi9y8iWGvBb5zGAZ4+pRVUH3JaNePNPgZ/mPt24p6YyIzai0BFIUj8Tjusbf2+mTVNSmZKTE6GRxTZO1/XokSOsefOflcoNxE8JkWnT8/j3ug08+c5H5L+/jnmvr+TbzZt4bu4DgPV7LC0ujlnfN1+J/cyUd/RhFJwEw+fcZ0RHFRqM4KkbWVlZHPbhjP3fo0d55ZW/x/SDN1I4Kd5F+/aQmp4RdyVEfPmnTJg1jxVLX+D7nd/Exe/RKQO18bWJPoyCY4grjOCpPSkpKQy/5hrmTPldFQVxQd4kLh5+HemuzJh98EYSt+L9+O2TKq+rSAMO7d8Xd5F9/qyAjRqfRN6vhsbF7zEpKYm5D8+hYPcu3l32FgW7dzHz3hns2LHDvAQEQBLVA1yE0CTm88T44DjMBQrx5Zviib+5Tzfx4GvkmYALMH43AVJcXEybdu2RBg1IdWVSVlzEoGuu5YobRzHtl5fV6F9gfHCcOXbsGD/vP4ANG9aTnJbOkZLDpDdrQVaHjkycPT9ufHCKi4vJatOWBW+vqeafMuWqAWzdvJnWrVtHcIShp6KigmnT81i8eDFprsy4chYPF/Xlg5PwV9+tSSYCNYV/x7ry5klKSgqdOnUygicAPJXBjIwMfjdmDCs+/JhfjZ9K9plncexIWVxMK0SSO+68ixONm/DIP1egegKRBjx2x2S2fb6OCUP70axFK0rjILJv5h/vw9WiJfOnT6yiuD06dRyjR42OO+UGnCPknrhjclREHkYaX1HK9SV9jQUngaKo4jFyyh85uVNZ9cnaSn+AeHhDDiW+3jwfuP8+7rjzLpYsWVItpL4mxdBYcKrjL+omZ9hAtnz1FSUlJTFvYXSf50N/f4c3nlvMiqUvkJLhorToEKhSsOvbGqPwYg0TUeWfQGYMPAn0GWssOHXgEPH5wAdw4XxzuVJT63soYcVfThxTf8rC15vnbdOmM2H8rUy7bWpcPHgjjd+8KRmZlJSUxGS4tDfu82ye1Yabp93Nr2/NpWjfHlwtW3P7NYPZt29f3Ck4/v63pv5U4ITLeGCcjBOMQ1SPoAJi2jLlRCCCJ5Hxl4110aKnGHjpYM44swsLn1pEkyZNIjza2CaQ9AXxgPd5Nk1OJqvjqRw7UhZX5+lJovxvw0F9pGAxCo4hLjGCxz/+FEBXy9bc/tSzzH3tPVZ9spZp0/MiNMrYxLv0QqKkL0iU8/QkEc85lkh4BceVmuoYrhZvUzaJhhE8/vGnAB4pOYyrZeu4STxXX/ir45Uo6QsS5Tw9cTrnvj26cf1117Ju3Trz24kgCeFknMj1pjxJtOvgdqL1dJa98cYbueXmETRs2JDTTz89oRUdJyfsBXmT6HBmV26edndlO3eZi0B8CRLZyTgQp3bPiLV4vve8zzMRzrusrIxdu3bx+JNP8pclf+Gkk5P5obSEpEaNuHnECMbfOo727dvH7fk74fOZg+Uu4SZYn9eA5Yyqxtxy7rnnajAAqg6LdfqGeKe0tFQ3btyo48aP16YnJ2tG85ba+KQmenJKqv5+3K26ceNGLS0tjfQw653y8nKdkpOr6RkubZvdURufdJIOvnaEvvTlLl26uVCXbi7U/NXrND3DFfD1AdZqFMiIUC2ByprS0lJNS8/Q/NXrKq9dba5fvOG+x9LSM7RtdidNS8/QKTm5Wl5eHumhhYUpObl6dp++lfdB/up12rX3+dqk6cna4pQ2cX/+NRGqZ3Ggcibhp6gSFXdNKu+lplw5sUhKSgr5i5ew+pPPWPD2GhavWc8T73xIdpez+Mtf/sKASy6rMp2QKCQlJTHz3hm8t+JfvPrKUn772zHsL/iWkkMHATOlFwzGqd0Zz0i9eW+viWu/rrKyMvIX5zN5zhNVHPdzHn6Sho0a8cjrK+P6/L1xesb4ymAcLpcQo+AkKD5rUsVBQVFvjOCpjqe/yLBfDmfAwEEkJSXx8949o9p/QkSGiMgWEdkuItX+WSJyg4h8bi//FpHuHtt2isgXIrJeREKTSMvGn09TcdFB0uLwxaEm/EXqxaNfV2FhIWkZLkclN9WVaYfMx+/5e+P0jCm3t3lbWsLlIhESBSdahY7BAEbwOOH0Zv3+2v/QQBpUqa8z9+E5UZP1WUQaAo8DQ4GuwHUi0tWr2TdAf1XtBvwRWOS1faCq9tAQ+wn5cmqfM3kMDRs15owzuySchTDRrFpZWVmUFBc5KrllxUW4WlpZnOP1/KOROis40Sx0AiGep2YMFkbwVMX9Zj3yrgc4eqSMo0eOVFHwADp37hyN01J9gO2qukNVfwReAK72bKCq/1ZVdyLyj4C29TU4z2iasQN7c+vgC+jQuQuLVn6WcBZCSLxUDSkpKYweNZpHp46rouTOnz6RQddcS9Pk5Mp18Xj+nrifp95TUZn1PI5QWHCiWuiAn1Bw4ntqxmBhBE9Vdu3aRYNGjZj+68u5f+wIfjewF08/OJO0zGakZmREs4LXBtjt8b3AXueLUcBbHt8VWC4in4nIGF+dRGSMiKwVkbX79+8PeHDuCtObv9rED6UlPPz3dxkzYxYNk5ISzkIIiZmq4cHZs7j4wvOZMLQfo/r1YNylffl+106uuHEUEP/n78anC4S9vb6MCqGwPTsJnfP8tPcldBR4SlW9rTuAJXSAMQDt27cPaoCe83u+amMY4psHZ89i6rRpTBjaj5OaWuGb6c1bcOv9c4HEETwATy58itbts5n66KIq4eGL77uLfd8XMn/BYzz6yNyomZrywKnKiOPPWUQGYsmafh6rL1TVQhFpCbwjIptVdXW1HVoyaBFYYeLBDrKkpARX8xackt2xyvpETN//4OxZTJueR+6wQdXqmsUjSUlJPDp3LvfNnMm2bduoqKjgmWefY9ovL0uI8w8UBaQejAp1zoMjIr8CBqvqaPv7TUAfVZ3g0HYg8ATQT1UP2uuyPIUOMMFJ6HhSlwJ4dS3THi8kWk4cN2VlZVUEz7PPPht0QclYxl9xwFsHX0C/y39B0d7CWhckDWceHBHpC8xQ1cH299sBVPVPXu26AX8HhqrqVh/7mgGUqarfk6yNrPF3jadcNYCtmzfHZVVtfyRCHhx/JNr5+3vOunPg1OWZG6icCcUUVQHQzuN7W6CajdsWOvnA1W7lBkBVC+2/+7CEUp8QjKlWJJI/zqGSEse8AfGs3IBlNj/nnHPo3bs3C+bPi1qH2nDhz/EzNcPFVb/5XTRPpXwKnC4iHUWkMXAt8KpnAxFpD7wC3OSp3IhIsoikuj8DlwFfhmpgnuUZjMNxdVJSUqLVr6teSPTz9+RQzU1CRigUnKgVOsFi/HESj0QTPP4cP4/98APNT2kTtc7WqloBjAeWAV8BL6nqRhEZKyJj7WZ3A82AJ7wiM1sBa0RkA/AJ8Iaqvl3XMfkqz/DA/fcZh2ODIcKEpFSDiFwOPAo0BJao6v1ugaOqC0UkH7gG+NbuUqGqvUSkE5bVBix/oL+p6v01HS8Qs7GvKZhG/BSL74ln6uhEm64yJBY1lWgo2reX3GGDKNi9K2jFL9FKNdRUnmHPnj2cdnpnZr/8dhWfnLpc41gk0aZoEp1ASjTUxxRV3NaiCsTXxvjjGBIRzxpdjZueTEnxIQZc/StG3XUfJYcOVqufFAyJpOD487VxKy+FhYVcPHgo895eU61/MDW+YhX3vbZ48WLSXJmUFB1i1KhR3P2Hu9i3b59ReBKAcPh71qcPjiFG8U6l3cihdEOi+CQlEu5w5oLdu1j57nJGjxrNR2+/Ru6VA6Iye3G0Ekgiu0TLBeONd0LJh/7+Di8ufYWsNm2rVVw3xCeR9Pc0Ck4C452roILqeQuMT1L8kpKSQteuXXlswfyEc7YOBYEoL4mYC8aNU6mGN55bTPOstix4e03c16YyRJ6EVnB8JgAMU+EvgyFaSTRn61AQqPLimeE4Wmt8hQNvC9fRI0f418vPM3H2/ISoTWWIPAn9mhbvIdEGgyG8BJLIzj0lOPPeGQnlaOtp4bIcsPeQWsOUXjz6IxkH68gRtxYcY50xBIJn/hKDIVg8/ZlqmuJLNCuZt4XL1bI1JYcOJow/kq8UAsbfqP6IWwUnURPZGQLDCB9DKEk05SVQPKfnbr9mMALVasLFqz/StOl5rPjwY6Y98VdmLV1m/I0iQNyGiRtqxjt8LwnL0dibeCzfkJM7lRUffsyvxk8l+8yzOHakrE7h0QaLRAoTNwSOe5qmZcuWzPzjfSxZsiSuS6QUFxfTpl17ECEtsxmlRYe4ePh1XHHjKKb98rKEyX8ULhI+D47B4AsjfMKHUXAMgRDvfikjR43iww1fMvWRp6ol01y3Ynnc5z8KN4HKmfhRmQ2GAMnJzaX9GV2qCZ83nltczdkx3gWxwRAJ3FN68UhZWRkvL13KI6+trBItNmHWPCZdOYCGIlX8jYyMCR9x64NjMDjhFj5u5QZ+Ej7/evl5DhcdIisry/joGAyGWlFYWEi6q5ljtNhJTZvyy1/+DykpKUbG1ANGwTEkFIEKH+8MrMZB0GAwBIK/BJBHy0p5ZO5coHqWZyNjQo9RcAwJRSDCxykDq0lIZggGk34gcfGVAPLx2ycx9ndjycjIMDKmnkgoBce79pKptZR4BCJ8AqkxFAvU9JCNxYewiAwRkS0isl1Eqr3qisV8e/vnItIz0L6hwEw7BI7n/ReL96I/nLJXDzivd2UCyFiXMYE8S6PieeuUKybYBRgCbAG2A3kO2wWYb2//HOgZaF+n5dxzz9XaAKg6LNZlMCQK5eXlOiUnV9MzXNouu5OmZ7h0Sk6ulpeXq6pqaWmppqVnaP7qdbp0c2Hlkr96naZnuLS0tDTCZ+Af9/mlpWdo2+xOmpaeUeX8atpeF4C1GgKZ4rQADYGvgU5AY2AD0NWrzeXAW7bMOR/4ONC+TkuwsmZKTq727Ne/8t7JX71Oe1xwkY6fMLFW1zMe8bz/2nToqCenpGrTk5NDfi9GA6Wlpbply5ZqMiPWZUwgz9JwPm8DlTN1tuCISEPgcWAo0BW4TkS6ejUbCpxuL2OAJ4PoWy8YS46FW+uO58riNWWfjfUCiTXN7cfw3H8fYLuq7lDVH4EXgKu92lwNPGPLwY+ADBE5JcC+dcLXtMP4WfNYtOgpxk+YaCw5VL3/zr14MKf9rEfcFt/0lQAy1mVMrBCKKaqoFjqBYqpmW7grjCdCZXF/2WdjtUBiTXP7e/bsieW5/zbAbo/vBfa6QNoE0rdO+Jt2cLVszXsffhQ3D+7a4nl/NklOSejim7EqY2KJUCg49SJ0RGSMiKwVkbX79++v86ANBn8EU2MomvD3kE1JT+ejjz6K5bl/cVjnnanUV5tA+lo7qKWs8efAfqTkMOPuT4wHtz88789Aim/GM7EqY2rCbe0HyIzwWEKh4NSL0FHVRaraS1V7tWjRIsghGgy1I9ZqDPl7yO7fu4ebbr6ZQwf2c6Dwu2rbY6DgYQHQzuN7W8D7KeirTSB9gdrLGve0w2N5k6pMOyzIm8Sga67llOyOCfHg9kfVCuOtKfVxr8bAvRgyYk3G1EQVi3+ExxIKBadehE4o8FlhPFwHNBjqGV9z+/OnT2TI9bfw6Our6NT1bO79za9jce7/U+B0EekoIo2Ba4FXvdq8Coywo6nOBw6r6vcB9q0zD86eRb9zezDu0vMZd2lfJl81kA5nduXGnDsS7sHthOf9eexIGRcPv4750yfG4r2Y0ATzLK3WJjW1voZZ9ygqrHIPO4CO/BSdcJZXmyuoGtnwSaB9nZbaRlF5eGCbaCof4HktzDWKSSqjVDIyNKN5S01OS9dhI8fqS1/uqozUaJqcrGnpGY5RZHWBMEZR6U9RUluxIqLutNeNBcbanwUrcOFr4Augl7++NS21lTW3jp+gXc/trY+9/UHlNe/Zr79Oycmt1f7iiaKiIv3NyJGalpGhbTt01OTUNG2abEVRhfJeNNQf9f28CFTO1HmyT1UrRGQ8sAwrFHOJqm4UkbH29oXAm7Zw2Q78APzGX9+6jqkmXKmpiIOzbL1qllGEZ1XxJKwnhPuvN4l6jWIJ99z+TTfewNCrhvHYsg9ompxcud3VshXNW7Tin6+8THJyckzVwFHVN7Hkiee6hR6fFbg10L7h4tFH5jJteh53XntltarZiUpFRQXTpuexePFi0lyZ6AnlkkEDeGTuXJKSkkw9JkPIMdXEDYiIo+OTALF4fxgsysrKaNO2HXNfe6+KI2fRvr3kDhsUlqrpppp4VUwhxZ/IyZ3Kqk/WVkbwuaej+vfpxdyH5/jsZ65h9FPfz5BA5UxCZTI2GBIJk2sj8sSbA2ltqU1pApMVOnbw6ZMTYYu/UXAMQRMVKbgNAWFybRiigdqUJojhhJQJx6GSEkcfmEMlJREdl5miMgRtXjRTWrFHfZn5zRSVwYlgp0sjMb1qiB3MFJXBYKjETJUYIkmw06WxXozSEB3EdspEQ0gwUWUGgyHcPDh7FtOm55E7bFCNkWVVEwJWteAkei4hQ+AYC44haudPDcFTVlbG1q1bE7ocgCE6CaY0gdvis2D6BL767BOOHjliHOQjTCz6XhoFx2CIA0zEiSFWCGS6tKKighN6gq0b1jFn0hhGXvAzJl3Rn5/36mkc5COEuxBzLBVgNgqOISjc2rp3OGAjwJWSYqwHEcJEnBjiiWnT83j/0/+w4O01LF6znife+ZDO3XrQoEEDDhw4wLJly9izZ0+kh2mIckwUlSEo/EVQpaVnkOZyUXzoILfcfAuPzH045ivjxgLBRJyEO5rKRFEZ6oqv+/lA4XdMunIAx49XkJKWQVlJMd279+D9VStp0qRJBEecGNQUPeuZEd8TV2pqyN0dTBSVod6xrAcf8Ojrq3j3gw/pe2E/M0VSDwQScWKmsAyxgq/7+Y3nFpN9ZleefOcj8t9fx5PvfMSRE/Dz/gMiM1BDFaJxCssoOIaQ4ZmhdOqji/jii8+ZPCUnwqOKf6yIk4OV4bduivbt5XDxIbKysswUliFm8IygcnP0yBHe/b+/MfXRRVXkTO7chXy+YYOZrjI4YhQcQ1hwtWxFRvOWPP3008Ynpx5o374DD036bZUcIw/njKVz5zMAgk6TbzBECqecOTs3b6RJ05MdrZTJaels2LAhEkNNKNwFmL2XaHZCiOaxGWKYon17OVJymHRXJoWFhXTu3DnSQ4o73NWZ8xfn07DRSRw9UsbEyy8iNbMZRw4Xc+HQYXz41qts27atxiks8/8xRBPeOXOKiw5y7Nixanlxvt/5DWXFRZx22mkRHG1849O3BjiEpeREK8aCYwgKX0XVGjdoUMV6sCBvEuddMpTDRYdIi+I8CbGMe9rpkddWsnjNeh5f9gGndTuHs/v0ZeGKTxkzYxbprkyAaiZ/iK2kaSKSKSLviMg2+6/LoU07EXlPRL4SkY0iMslj2wwR+U5E1tvL5fV7BoZg8M6ZU1hQQPfuPXg4ZyxF+/ZyvKKCRTPyyPnFxaS6Mul5bi/jUxYmfPrWRHRUgVEnBccIncTDKSlgeXk53Xr15tbBFzDu0r5MunIAZYeLWfPGP2iaksoZZ3YxwifE+KrOPHHWPD5+5y3gJwXm9NNPj4eq4nnAv1T1dOBf9ndvKoBcVe0CnA/cKiJdPbY/oqo97OXN8A/ZUFc8c+a8v2olyQ1g3KV9GXlhN77d+hVPLP+QP6/+j/EpiwKisaJ4XS04RugYSEpK4sMP1jB61GiOFBfRqFEjGp10Eo8v/zcL3/vUCJ8w4C9yKiXDxc7NG6soMHFQVfxq4Gn789PAL7wbqOr3qvof+3Mp8BXQpt5GaAgrTZo04dOPP2LTxi/RE8erORwbn7L6x1OBicaM+HVVcIzQMQCWkvPYgvls27qFivIfjfAJM06RJmBZZg7t/Z4Hx91SRYEJJk1+lNJKVb8HS6YALf01FpFs4BzgY4/V40XkcxFZ4mRt9ug7RkTWisja/fv3133khpBy/PhxXM1amI7YbmIAABHhSURBVEKcUUCkFZiaqKuCY4SOoQolJSVkZDY3wqeO7Nmzx2+2Vl/VmR+/fRI3XH893xXsdlRgormquIi8KyJfOixXB7mfFGApMFlV3dL3SeBUoAfwPfCwr/6qukhVe6lqrxYtWtTybAzhwp9yHys+ZZGksUM9KRGhsUSzu3DtqFHBMULHEAw1CZ+0tDS2bt3Knj17TFkHB44dO0bv886nfXY21910M+2zs+l93vkcO3asWlunaacB5/Vm0VNPRaUCUxOqeomqnu2w/BPYKyKnANh/9zntQ0QaYcmZ/1XVVzz2vVdVj6vqCeDPQJ/wn5EhHPhS7j2nZGOxMGR9UU51h2G11zsRjb41gVKnUg0isgUYoKrf20Jnpaqe4dCuEfA6sExV5/rYVzbwuqqeXdNxTfr06CYndyqrPllb6QDrFj4NfjzGls2badi4MUdKLUvPf38oY9So0Tw4e1YsTZeEjd7nnc+RE5A7d2HltXs4ZyzJDeDTjz9y7BPu8gvBEK5SDSLyEHBQVWeJSB6QqarTvNoI1lT5IVWd7LXtFLe1WUSmAOep6rU1HdfImujEnSJhyZIlpGW4KCkuovzIEX4o/7Gyjb+yAolMTSUXYoFA5UxdFRwjdAzVcBI+p3fuzPFGJ9GyXTZ7du1kwqx5VZSf/n16MffhOZEeekTZs2cP7bOzefKdj6rVlBp3aV++3fkNrVu3juAIayaMCk4z4CWgPbAL+JWqHhKRLCBfVS8XkX7A+8AXwAm76x2q+qaIPItlKVZgJ/A7t+zxh5E10Y2ncp+amlr54BaMguMLo+AEfhAjdAw+cQuftLQ0zjizCw+8+AbTf305815fWWNRyERk2bJlXHfTzeS/v67attE/P4fnn32awYMHR2BkgWOKbRoiheeD2yg4vkkkBadOcwKqehC42GF9IXC5/XkNPpIdqupNdTm+IbpxO7Ru3bqVNFcmqidINRl1fdK9e3fKSoqrZWt1Z4Xu3r17BEdnMBgMsYXJZGwIO27HY5EGlJroB5+0bt26SrZW+KmmVLfu3aN+espgMEQ/jXCuKdUokoMKE0bBMYQdd9TDkvvuoN/lV7Mgb1IsZ9QNK57ZWkf//Bx+f8n5NPzvUd564/VID81giBlcOD/EYyHyJ9z86JGEz/N6lEPcRZsZBcdQL7hDmj9861V2bf2KcZf2ZezA3uQMGxhrGXXrRFlZmd/weHe21q+3b2NQ/4to0qQJxaVlnHra6abchcHgB89w5iKv9dGQVTfc1CY03medKYfimrGIUXAM9YI7k+53Bbv56IM1fLvzG9asXMF3u6snpPOnBNSkIEQrFRUV5OROpU3bdlw8eCht2rbzq7DMfeRRvtmzj0deX8m8t9eYchcGQw0EUiognvPjxLuyUhuMgmOoV9yOx61bt66WUdefEhCsghBtuCt/z33tvRoVFl+FNE25C4OhbtSkBMSzApSImMxqhqjBUwnwzJHjVgJ8bYv2/DluhcU9dvhJYckdNoiZ986oouj5K6Rpos0MhvDhVoC8kQS2gsQyxoJjiBie003+rBaLFy8mf3F+1Fg0gp0mC0Rh8dxvWlqaqbVjMNQRJ2sMQGaEx+WPcFmQfO033i0cRsEx1DtO000TJ00iNcPlqAScnJJCmo9t9VnAs7bTZDXV52rZsmWV/Z5xZhc6n3EGj99uos0MhtriczoqoqPyT7j8aHztt4L4jjYzCo6h3nHyR1m/ZTtFBw84KgE/lJVRUlwUcYtGMH40ntRUHHDmH++rtt8TjZvQsPy/VQppJlK0mcFgCI7aFsWsyTE7lqlTqYZIYdKnxy5lZWW0aduuij8KWA/8CUP70bnbOUyYvaBanSrAsYBnfdWw8jfuQMpMONXnGjlyJHf/4S46ZHf0ud/NX22ipKQkKgppBoIp1WCIFvyVJPDGlZrKoZKSiJcxCNfxI31eoaZeSjUYDMHizx+lWYtWnHPm6eQOG0RahovDRYf45S//h7v/cBcpKSlMm55Xuc2tILgtGuGuqF1Xx193mPzMe2dUGae7jIWv/ZaUlBiHYoMhxPh6qLtSUx0dipOg0ofHs228WDriFTNFZahX/PmjlBYXsWD+fHZ+s4NLBg1AUd59bxUdsjsybXoeD86eRcHuXby77C0Kdu+qtNx4+8WMnzCRTZs2hdT52HvcR48cofCbr/l+5zdBTZO5w+TdSlhN/jnGodhgCA9OjrdFpaVVEgO6FaFyLJ8Vl0f/otLSsISRZ1J9msk9XkNwGAXHUK/U5I+SkpLCzD/ex4atX/PIa9WT3HkrCJ5+MXNfX0nfocPIz89nwCWXhTRXjnvcj98+iUUz8vjdwF7cP+ZGcn9xCad37kyTJk3Cdj0MICKZIvKOiGyz/7p8tNspIl+IyHoRWRtsf0P8UJNPSm0ceosc2ocymZ4rNTUsx6itf07M4+RgFOiCpWy+A2yz/7p8tNsJfAGsB9YG2997Offcc9UQu5SXl+uUnFxNz3Bpu+xOmp7h0ik5uVpeXq6lpaWalp6h+avX6dLNhZVL/up1mp7h0tLS0sr9eLcdNnKsdr/gosrv+avXac9+/XVKTm6NYyotLdUtW7ZU2b/TuHv1OU/P7Nm7VseozfWINTx/36FcgAeBPPtzHjDbR7udQPPa9vdejKyJXwBVh8V6LDq3C7SPJ67UVCd9RV2pqXUaVyITqJypk5OxiDwIHFLVWSKSZyso0x3a7QR6qeqB2vT3xjj+xQdOfjNbt27l4sFDmff2mmrtJw/px7vL3qr0SfFse/TIEX43sBfzXl8ZlBNwcXExU3JyWLr0FdIzm1FSdIhRo0bx4OxZVcpHuMdbk6MxUGtfoHD7EdUH4XIyFpEtwABV/V5ETgFWquoZDu124ixrAurvjZE18Uugjree7QSCctbNTEtztLy4sK1BDn38jSsJK7S7yr4S0BcoUDlT1ymqq4Gn7c9PA7+o5/6GGMZ7ugmC80nxbFu0bw+pASTTc+POaZPVpi2vvvkWx1XpOegyHvr7Oz5Dv/05GjduejK3jp9Qp1ISTtfDUEkrVf0ewP7b0kc7BZaLyGciMqYW/RGRMSKyVkTW7t+/P0TDNyQioc7FU+G0L5Nl2Sd1VXCM0DGElGB8UjzbijSgNADFyJ0teEpOLis//pQFb6/hz6v+w/w3VvHt5k288dxinxmS/SpfRYdYvuK9ylw2D7z4BstWrWbylJxwXKa4RETeFZEvHZarg9jNharaExgK3CoiFwU7DlVdpKq9VLVXixYtgu1uiDM8/Vf84StzcqjxdkIGTL0sX9Q0hwW8C3zpsFwNFHu1LfKxjyz7b0tgA3CR/T2g/t6LmRePb4LxSfFs62rewqd/jLtdWnqGZrXP1kYnneTo55OSnqHPfbZN22V30i1btlQ73vgJE6sdo/sFF+nga0doclq6PvPJVzps5FhNTkvX1u06aKOTTtLxEybGpD9NbSB8PjhbgFPsz6cAWwLoMwOYWtv+amRNXBOsb0xNffDlN+NjPT58anwew7NfgvvoBCpnjNAxRC2BOP56tt24caOOnzDRUTGakpOrPfv11/zV63TBW+9r63Ydqig37qV1h4563//+o5pDs5uNGzdqRvMWmpKeoa07dNSU9AwdNnKsvvTlLm3doaMOuubaao7OPS64qE5OyLFEGBWch6jqJPygQ5tkINXj87+BIYH2d1qMrDEESqgUHF/7UqPgVBKonKmrk/FDwEH9yUk4U1WnebVJBhqoaqn9+R1gpqq+HUh/J4zjn8Ef3s663s7B/hySJ105gNPO+hmD+p7nmCHZva8HXnwD1RO4WramaXKy1feK/igw/41Vtcp2HA+E0cm4GfAS0B7YBfxKVQ+JSBaQr6qXi0gn4O92lyTgb6p6v7/+NR3XyBpDoDg5B/tzSvbnHFxTFuZgHJ3jkfpyMv7/7d1bqFzVHcfx74+oIMdLtCHxRMULhGJ9kIYSjYpIY0TOS+oNLBS1lKhIqlX6cFDxyYcoxN5sH8QIEaR9UWvQeEcpItqL5CRKaqNBMZyQxMQafTGCfx9mHzk5mcueM3v2rFnz+8Aw+8zs2fu310z+rOxZs9d6YLWkncDq4m8kLZW0pVhnCfCmpCngn8DzEfFiu9eb9WLuYN25g4OPHxtj1XU/50+TR05mueGuWyGCn668sOWcTzPjfh5/4B6OHzvh+87N7397O2MnnZzEpKA5iogDEbEqIpYV9weLx6cjYqJY3hURFxS382c6N+1eb9ZPp9B6Mst2v3xqdd0aTz3QnZ7aKyIOAKuaPD4NfF90gAu6eb1ZlWYPDp7pfPzi7nvY+MB93L56JYvHxzn0/8+59ppreOvVl1i4cGHb7T304Pqjpo24+Zc3c/jrwzy28bEj9gO+KrHZqDrI/M6stOv89Gvwco7cIbTszf611cxknYcOHmD/7k9Yu3Ytd/x6XVfXnmk1rxTAggULeGTyTtat/8MRk4L6qsRmeWs1j1XVVwuuaz85cAfHRkKzsy4zk3XOvaBfWTNfhc32u4c3tJ0U1MzyVNfF9kbton696GmQ8aB44J/NV11XC87hqsTz0a9BxoPiWmOWnrJ1xmdwbKQ0O+syzPsxM7PmPJu4mZmZZccdHDMzM8uOOzhmZmaWnaEcZCxpP/BJD5tYBHxWUZxhzgBp5EghA6SRI4UMMP8cZ0VENjNU9lBrhv19zC0DpJEjhQyQRo5eMpSqM0PZwemVpH8P+pceKWRIJUcKGVLJkUKGlHIMq1TaL4UcKWRIJUcKGVLJUUcGf0VlZmZm2XEHx8zMzLIzqh2cRwcdgDQyQBo5UsgAaeRIIQOkk2NYpdJ+KeRIIQOkkSOFDJBGjr5nGMkxOGZmZpa3UT2DY2ZmZhlzB8fMzMyyk30HR9L1kt6X9K2klj9Jk3SVpA8kfShpsg85TpX0iqSdxf0pLdb7WNJ2SVslVTLLX6djU8Mfi+e3SVpexX7nkeNySV8Ux75V0v19yPC4pH2S3mvxfF1t0SlHHW1xpqTXJe0o/o3c2WSdWtpj2LnOpFFnXGO6ypF/jYmIrG/AecAPgTeAn7RYZwHwEXAucBwwBfyo4hwPAZPF8iTwYIv1PgYWVbjfjscGTAAvAAIuAt7pw/tQJsflwHN9/jxcBiwH3mvxfN/bomSOOtpiHFheLJ8I/G8Qn40cbq4zg68zrjFd58i+xmR/BicidkTEBx1WWwF8GBG7IuIw8DdgTcVR1gCbiuVNwM8q3n4rZY5tDfBENLwNLJQ0PoAcfRcR/wAOtlmljrYok6PvImJPRLxbLH8J7ABOn7NaLe0x7FxnkqgzrjHd5ei7QdeY7Ds4JZ0OfDrr790c/Sb0aklE7IHGmw4sbrFeAC9L+o+kWyrYb5ljq+P4y+5jpaQpSS9IOr/iDGXU0RZl1dYWks4Gfgy8M+eplNpj2LnO9Pf4XWO6l3WNOaaKjQyapFeB05o8dW9EPFtmE00e6/r38+1ydLGZSyJiWtJi4BVJ/y164vNV5tgqOf4KcrxLY46RryRNAH8HllWco5M62qKM2tpC0gnAU8BvIuLQ3KebvGQkry3hOtM+VpPH6q4zrjHdyb7GZNHBiYgretzEbuDMWX+fAUxXmUPSXknjEbGnOP22r8U2pov7fZKeoXHatZfCU+bYKjn+XnPM/uBHxBZJf5G0KCLqnBSujrboqK62kHQsjcLzZEQ83WSVJNojBa4zbaVQZ1xjujAKNcZfUTX8C1gm6RxJxwE3AJsr3sdm4KZi+SbgqP/xSRqTdOLMMnAl0HQEfBfKHNtm4MZiNPtFwBczp7kr1DGHpNMkqVheQePzeaDiHJ3U0RYd1dEWxfY3Ajsi4uEWqyXRHplwnenvZ8k1pgsjUWOqGq2c6g24mkYP8WtgL/BS8fhSYMus9SZojPD+iMYp56pz/AB4DdhZ3J86NweN0f9Txe39qnI0OzbgNuC2YlnAn4vnt9PiVyA15FhXHPcU8DZwcR8y/BXYA3xTfC5+NaC26JSjjra4lMap4G3A1uI2MYj2GPab60wadcY1pqsc2dcYT9VgZmZm2fFXVGZmZpYdd3DMzMwsO+7gmJmZWXbcwTEzM7PsuINjZmZm2XEHx8zMzLLjDo6ZmZll5zuIOlk+H+nDggAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 3))\n", + "\n", + "\n", + "ax1.scatter(X[y_km == 0, 0], X[y_km == 0, 1],\n", + " edgecolor='black',\n", + " c='lightblue', marker='o', s=40, label='cluster 1')\n", + "ax1.scatter(X[y_km == 1, 0], X[y_km == 1, 1],\n", + " edgecolor='black',\n", + " c='red', marker='s', s=40, label='cluster 2')\n", + "ax1.set_title('K Means Clustering')\n", + "\n", + "\n", + "ax2.scatter(X[y_ac == 0, 0], X[y_ac == 0, 1], c='lightblue',\n", + " edgecolor='black',\n", + " marker='o', s=40, label='cluster 1')\n", + "ax2.scatter(X[y_ac == 1, 0], X[y_ac == 1, 1], c='red',\n", + " edgecolor='black',\n", + " marker='s', s=40, label='cluster 2')\n", + "ax2.set_title('Agglomerative Clustering')\n", + "\n", + "plt.legend()\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Clustering using DBSCAN\n", + "\n", + "Unlike K Means or Agglomerative Clustering, DBSCAN is a density-based approach to spatial clustering. It views clusters as areas of high density separated by areas of low density. This approach has several advantages; whereas K Means focuses on finding centroids and assoicating data points with that centroid in a spherical manner, the DBSCAN algorithm can identify clusters of any convex shape. Additionally, DBSCAN is robust to areas of low density. In the above visualization, we see that Agglomerative Clustering ignores the low density space space between the interleaving circles and instead focuses on finding a clustering hierarchy that minimizes the Euclidean distance. While minimizing Euclidean distance is important for some clustering problems, it is visually apparent to a human that following the density trail of points results in the ideal clustering. \n", + "\n", + "For more information on the DBSCAN algorithm and its implementation in scikit-learn, check out this resource: http://scikit-learn.org/stable/modules/clustering.html#dbscan" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.cluster import DBSCAN\n", + "\n", + "db = DBSCAN(eps=0.2, min_samples=2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, let's fit our model to the data and generate predictions." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "y_db = db.fit_predict(X)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lastly, let's visualize the model applied to our data. We see that the DBSCAN algorithm correctly identifies which half-circle each data point is generated from." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XuUVOWV9/Hv1sbBobulmzs0iCRiRAeQtHhLIsEYlJlAZkbf5SVecQghKggE0KylyCQuUCBqvDAOkDFmJsbXZN6QRMWoUVecJIqxvSDXMAoVjCC00m000LrfP7qaVHfXvU5Vnar6fdaqRVfVqXOeorprn+d59tmPuTsiIiJhc1ixGyAiIhKPApSIiISSApSIiISSApSIiISSApSIiISSApSIiISSApSIiISSApSIiISSApSIiIRSVbEbkEzfvn19+PDhxW6GiIgE6MUXX3zH3ful2i7UAWr48OGsX7++2M0QEZEAmdmb6WynIT4REQklBSgREQklBSgREQmlUM9BiYgU28GDB4lEInz44YfFbkrJ6dmzJw0NDfTo0SOr1ytAiYgkEYlEqKmpYfjw4ZhZsZtTMtydvXv3EolEOOaYY7Lah4b4RESS+PDDD+nTp4+CU4bMjD59+uTU81SAEhFJQcEpO7n+vylASUVpbW1ly5YttLa2FrspIpKCApSEXhBBpa2tjTlz5zGkYShnTTqXIQ1DmTN3Hm1tbQG2VKRdIU6EFi1axLJlyzJ+3bvvvss999yT8/HvuusuPvnJT2JmvPPOOznvLx4FKAmtIIPK/AULeeb59az42a+447Ffs+Jnv+KZ59czf8HCPLRcKlUpnAhlE6DcnY8//rjTY2eccQZPPPEERx99dJDN60QBSkIrqKDS2trK6tWrmXnL7dT1HwBAXf8BzLzldtasWdPpLFdDgJKLfJ0Iff/732f06NGMGTOGSy65pNvzEyZMOFQW7p133qGjhumGDRsYP348Y8eOZfTo0WzdupWFCxfyhz/8gbFjx/KNb3wDgNtuu42TTz6Z0aNHc9NNNwHwxhtvcPzxxzNz5kzGjRvHzp07Ox3zpJNOIt+1UhWgJJQyCSqp7Nq1i9q6+kP76VDXfwC1vevYtWtXSZz5SrgF+Tsba8OGDXz729/mqaee4uWXX+aOO+5I+7UrV65k1qxZNDU1sX79ehoaGliyZAmf+MQnaGpq4rbbbuPxxx9n69atPP/88zQ1NfHiiy/y7LPPArB582YuvfRSXnrppbz2lBJRgJJQSieopGvw4MHsb95H8+63Oz3evPtt9r/bzODBgzUEKDkL8nc21lNPPcV5551H3759Aaivr0/7taeddhq33HILS5cu5c033+TII4/sts3jjz/O448/zkknncS4cePYtGkTW7duBeDoo4/m1FNPzardQVCAklBKJ6ikq7q6mmnTpnHPDbMP7a9599vcc8NsrrzySoC8nPlKZQnydzaWu6dM166qqjo0RxR73dFFF13E2rVrOfLII5k0aRJPPfVU3P1ff/31NDU10dTUxLZt25g2bRoAvXr1yqrNQVGAklBKFVSqq6sz2t+tS5dw5vhG5k6ZyOxzPsPcKRM5c3wjty5dkrcz3yBoTqx0BP072+Gss87ioYceYu/evQDs27ev2zbDhw/nxRdfBODhhx8+9Pj27dsZMWIE1157LVOmTOGVV16hpqaGlpaWQ9tMmjSp04nYH//4R3bv3p1VW4MWSIAyszVmttvMXkvwvJnZnWa2zcxeMbNxQRxXSk+6X7itra1cNe1KTj9pTNygkqmqqipWLF9GZOcOnlj3KJGdO1ixfBlVVVV5O/PNhebESlOyE6FsnXDCCXzzm9/kzDPPZMyYMcyZM6fbNvPmzePee+/l9NNP75Ty/aMf/YgTTzyRsWPHsmnTJi699FL69OnDGWecwYknnsg3vvENvvjFL3LRRRdx2mmn8Xd/93ecd955nQJYInfeeScNDQ1EIhFGjx7NVVddlfV7TMjdc74BnwPGAa8leH4y8ChgwKnA79LZ76c//WmX8nDw4EG/bs5crz2qtzcMH+G1R/X26+bM9YMHD6bc7utXX+MbNmzwlpaWvLXvujlzfdxnzvRVz77kP960y1c9+5KP+8yZft2cuSlf29LS4ps3bw60fbm0R4L1+uuvZ/yafPxOlKp4/3/Aek8ntqSzUVo7guFJAtS/ARfG3N8MDEq1TwWo8pHuF26xvpg7AuNRvet86PARflTvurgBNN5rUgXdTLW0tHjtUb0P/R903FY9+5If1btOX3oFlk2Akr/KJUAVag5qCBCbRB+JPtaNmU03s/Vmtn7Pnj0FaZzEF9T8R7rpt/lI0033PSQbAkwkX5l/YZ4TEymkQgWoeCkoHm9Dd7/P3RvdvbFfv355bpbEE/T8R7pfuEF+MWf7Hqqrqxk5cmTKCe18XfMC+csGEyk1hQpQEWBozP0GQKeBIRV0zyDdL9wgv5jzfV1TPns5+coGEyk1hQpQa4FLo9l8pwLvuftbBTq2pKFjKOxPf/pT4D2DdL9wg/pizmfvpkO+ezn5yAYTKTVBpZn/EPgNcJyZRcxsmpnNMLMZ0U0eAbYD24B/B2YGcVzJXdehsGOPO47DevSgtr5Pp+1y7Rmk+4UbxBdzPno3Xeey8t3LyWZOTKTspJNJUaybsvjyL17W3KfGneyTLrg0Lxlk6abf5pKmG2QWXLJMvWwy/6T0hDWL76abbvLbbrst49c1Nzf73XffnfPxL7roIh85cqSfcMIJfsUVV/iBAwfiblcKWXwSQomGwubdfh9P//T/8tYb/wsE2zNINwkh3e0SvTao3k2yuax4vZzFNy9i+/btnTITVQmictTX1mJm3W71tbXFbtohQS23cfHFF7Np0yZeffVVPvjgA1atWhVkMwGVOqpoO3bs4G/+tleCobB6Fp5/bsnOfwQxVJjuXFZ1dTUjRozgxpsWxWQNNnDyKacyeEiDKkFUkOaWFhy63ZrTqMyQTBiX25g8efKhADx+/HgikUhO7zEeDWhXsHvuXXlooj82SDXvfpsDH/yZrVs2s3//fgYPHlxymWMdvZvFNy9i165dWb2HdOayRo4cCXTuadX1H0Dz7rdZPmcGp0+eyvRFSw714OYvWMiK5ZmvgpqN1tbWrN+7hEfHchvPPfccffv2jVuLL5GO5TYuvvhiDhw4wEcffcSSJUt47bXXaGpqAui03Ia7M2XKFJ599lmGDRvG5s2b+d73vpe0x3Xw4EEeeOCBjJYBSZd6UBWqtbWVBx54gAlfPp/vLpzVaShs2ezpXHLJJQwcODDrYbawyGWoMN1MvUQ9rbkrVvLco2v54P33C1odXXX8ykvYl9uYOXMmn/vc5/jsZz+bwbtKjwJUheroHVx03fX0HTyEWf8wga9POoPZX/o8b+94g6/N+Gqxm1h06c5lJetpVfeuo3n3nw7dL0QlCK1tVV7cw7vcxs0338yePXtYsWJFpm8rLQpQFap///7se2cPX/vCKWx44be4O6MaT+Hb//VTPm5rY9iwYcVuYiikM5eVrKfV+m4zdf0HHrqf70oQhbgGTAorrMttrFq1inXr1vHDH/6Qww7LTyhRgKpQi//1W4wYdSJ3/uIZ7l73HHf+4hne2fVHls68XNUKYqRzPVKintbyOTM449wpHNmrV8EqQaiOX3HV1dRg0O1WV1OT9T7DutzGjBkzePvttznttNMYO3Ysixcvzvo9JmLtKenh1NjY6B2ZKRKc1tZWhjQMPTSh36F599tcc+5n2BWJ0Lt37yK2sPS0tbUxf8FC1qxZQ23vOt57dx8jRx7Hls2bOaqunv3vNnPllVdy69Ileb3YNtlnO3fKRCI7d+jkI0MbN27k+OOPL3YzSla8/z8ze9HdG1O9Vll8FSjZWXbffgPYvXu3AlSGEmUNFjqTLrY31zHMpzp+UqoUoCpQ7JxJ17PsQlfLLrdU6I6swUT3C+HWpUuYv2Ahc6dMpLZ3Xafem0gp0RxUBQpDtWylQueP6vgFL8xTIWGW6/+bAlSFKna1bKVC518u14DJX/Xs2ZO9e/cqSGXI3dm7dy89e/bMeh9KkqgAyYbRijHEpol8KSUHDx4kEol0ur5I0tOzZ08aGhro0aNHp8eVJCGHMstWr15NbV09+5v3MW3atE6ZZMWYI8mkhJBIsfXo0YNjjjmm2M2oSBriK2NhHUbTkuYikg4FqDIV5ooCYUjSkOS0TIiEgQJUmQp7RYFiJ2lIfMqulDBRgCpTYR9GUyp0OIV1WFgqUyAByszOMbPNZrbNzLr9JpvZUWb2MzN72cw2mNkVQRy30iUbhimVYTSlQodHmIeFpTLlHKDM7HDgbuBcYBRwoZmN6rLZ14HX3X0MMAFYbmZH5HrsSpXuMIyG0SQTYR8WlsoTRA9qPLDN3be7+wHgQWBql20cqLH2RU2qgX2ABrWzlO4wjIbRJBNhHxaWyhNEgBoCxC5WH4k+Fusu4HhgF/AqMMvdPw7g2BUnm2EYDaNJOkplWFgqRxABKt5Sj13LU0wCmoDBwFjgLjOrjbszs+lmtt7M1u/ZsyeA5pUXDcNUrkKkfmtYWMIkiAAVAYbG3G+gvacU6wrgJ95uG/C/wKfi7czd73P3Rndv7NevXwDNKy8ahqk8hUz91rCwhEkQAeoF4FgzOyaa+HABsLbLNjuAswDMbABwHLA9gGOXvEzPijUMU3mKkfqtYWEJg0CKxZrZZOB24HBgjbt/28xmALj7SjMbDPwHMIj2IcEl7v6DVPst52Kx6dTJS/XajtVbC7VaqxReroV1y229LSkP6RaLVTXzIpkzdx7PPL++26qnZ45vZMXyZWntQ18+5W/Lli2cNelc7njs192em33OZ3hi3aNxC+vmcgIkkm/pBihVkiiCoC6I1DBM+ct2zlEVIaQcKEAVQUcmXs9e1ez63z/wwfvvA8rEk+6ymXNURQgpF+rrF0H//v3Z984epk/4NLX1fWhp3sdZ513I339lmjLxpJtbly5h/oKFzJ0ysducYzxab0vKhQJUESz+128xYtSJzF52z6H5pzsXXMvNV/wfZeJJNx2p34tvXpTWnGPssGDXxAqdAEkpUZJEgSXLyrrm3M+wKxKhd+/eRWyhlINMknCUbCOFpiSJkEo2/NK33wB2796d0f60sJzEk05FCK39JGGnAFVgQVWC0JeLJJNORQhl+knYKUAVWFCVIPTlIulIdCmCMv2kFChAFUGuBTn15SK5UtFhKQUKUEWQa0FOfblIrlR0WEqBAlQRZVsJQl8ukisVHZZSoABVgvTlIkHQ2k8SdroOqkSporkERddBSaGpmnmF0JeLiJQaXahbIVTRXHKli70lrBSgRCpUqou9Fbik2DRZIVKhYi/2jq3XN2/+fA6zw7TYoRSd5qBEKlCqosUjR5/ENUu/m/VqzyLJaA5KRBJKdrH33xzZi/OvnqcqJVJ0gQQoMzvHzDab2TYzi1sMzswmmFmTmW0ws2eCOK6IZCfZxd5/bt3P8E+d0OlxVSmRYsg5QJnZ4cDdwLnAKOBCMxvVZZvewD3AFHc/ATg/1+OKSPYSXex99/WzqKrqwYfvd+4pqUqJFEMQPajxwDZ33+7uB4AHgaldtrkI+Im77wBw98wWPSphyoSSsIpXSWLCKSfzL1ddpSolEgpBpOQMAXbG3I8Ap3TZZiTQw8yeBmqAO9z9+/F2ZmbTgekAw4YNC6B5xdFR6UGZUBJWiZaS7/jdnTtlYrcqJSKFlHMWn5mdD0xy96ui9y8Bxrv7NTHb3AU0AmcBRwK/Af7e3bck23epZfHFVnW48aZFaS+5LRJGqlIi+ZJuFl8Qp/IRYGjM/Qag60xqBHjH3d8H3jezZ4ExQNIAVSq69pbea97LRx99zHfWPtUtE2rulIksvnmR/uAl9DqqlGRKgU2CEsQc1AvAsWZ2jJkdAVwArO2yzU+Bz5pZlZn9Le1DgBsDOHYodF3d9js/e5phxx3PL36wutN2yoSScpaqMoVIpnIOUO7eBlwNrKM96Dzk7hvMbIaZzYhusxF4DHgFeB5Y5e6v5XrsMEi0uu287/wbTz78Qz54//1D2yoTSspZ1xO1FT/7Fc88v575C+JeeSKSUiDXQbn7I+4+0t0/4e7fjj620t1Xxmxzm7uPcvcT3f32II4bBskveDySNzZtAJQJJeUt0YmaLvCVXCidLEexFzx2LRnzQWsLt868nKPq6pUJJWUt2Ylax7B2NvNZUtlU6ihHyVa3nfHVGfwxspMn1j1KZOcOVixfphRzKUvJKlNoWFuypQAVgGRLZ2u9JqkEyU7UNKwt2VI18wApvVYqWcflFmvWrOl2ga9GDiSWlnwXkaKIPVEDdNIm3Wi5DREpiurqakaMGMGNNy3SNVGSEwUoEQmcromSIGiIT0QClWy13rlTJhLZuUPDfRVOQ3wiUhTpXBMlkg4FKBEJlK6JkqAoQIlIRlItwqlroiQoClBSkuprazGzbrf62tpiN61sZVKtPNnF6yLpUpKElCQzI95vrgFh/p0uZXPmzst4EU5dvC7x6ELdgOgPLJxiA1Q90Bxnm7qaGvbt31/AVpUvZeZJkJTFlyMtvlY6mgGPc2tuaSlms8qKMvOkGBSgEkh0oeF1c+Z2myBONWksUuqUmSfFoCG+OJINZ8w8+1T6DhhE63vvcsUVV4DB99Z8j9q6evY372PatGkqjlkAsUN8BpqPKoBs5qBE4kl3iE/fonEkG86o6z+Q6//tAY7sVc2Nl/wT9QMGHgpkHX+w8xcs1B9sntXV1GAawiuoW5cuYf6ChcydMrFbtXKRfFAPKo5kPajZX/o8K596AYDpEz7Nnb94RpPGBVRfWxt3bqkO2NflMfWg8kOJQ5KrgiZJmNk5ZrbZzLaZWcJqkGZ2spl9ZGbnBXHcfEl0oeF3F85i4j9fwJG9etG8+0/U1vfRpHGBNbe0xE+IoD0gxd7qamqK1cyypkU4pVByDlBmdjhwN3AuMAq40MxGJdhuKbAu12MWQuyFhrPOOYOZZ5/KwGHD+cqcGwCo6z+Q/fv2atI4RNy9000p5iKlLYge1Hhgm7tvd/cDwIPA1DjbXQP8GNgdwDHzrqqqihXLlxHZuYMn1z3Gv/zLdPZE3mT/vr0AfPh+K73r+3D7vJkq5yIikgdBJEkMAXbG3I8Ap8RuYGZDgH8EJgInJ9uZmU0HpgMMGzYsgOblprq6msGDBzPzazPg3pWdJogvv/xyMDRpLCKSB0EEKIvzWNeZ6duBBe7+kVm8zWNe6H4fcB+0J0kE0L6stbW1MX/BQlavXn0ojfzSSy/lazO+yrBhww71kr61eLEmjaWiKXFC8iGIIb4IMDTmfgPQNUOgEXjQzN4AzgPuMbMvB3DsvIp3se7/vPQyq1av6fRHqEnj/IotDFtF92QIJUQUjyquSD7lnGZuZlXAFuAs4I/AC8BF7r4hwfb/Afzc3R9Ote9i1uJT7bHwUGHY8NLFu5KNgqWZu3sbcDXt2XkbgYfcfYOZzTCzGbnuv1hUe0wkudbWVlavXn0oOEH738fMW25nzZo1Kv0lOQukkoS7PwI80uWxlQm2vTyIY+ZbbO2xrj0opZGLpHcSN3LkyCK1TsqBisUmoFVBRZJTAVnJNwWoJLQqqEhiOomTfFMtvjQohba4Etbf04KERddxKcaaNWu6XQuoiv6SiFbUFZGC0UmcZEIr6iagxQVFgqdrASUfKiZA6YJCkcLTCaHkomICVKIl3OcvSLg6iIhkqfMJ4TkMGjKEa66dpRNCyUhFzEGpKoRIYcWrMLFs9nSqqw7jN8/9mg8//FBzVhVMc1AxVBVCpHASVZiYd/t9vPrqK4w/9TQNtUtaKiJA6YLC8hRbRDb2Vl9bW+ymVbT2E8K6uCeE1UfV8d4Hf9FQu6SlIgKULigsTwmXf49zzZQUzuDBg3k30WrTzXuZs2KlavdJWioiQEHyqhDKNBIJTnV1NZdfdjnLZk/vdEJ454Jr+dteNQwafkyn7TXULolUTICKXcL9iXWPEtm5g1uXLmH+goUaDxcJ2HdWLKe66jC+Pul0Zp59GrO/9Hn6DRnKB++3aqhd0lYxAapD7AWFSj0XyY+qqip+89yvuWraVbz/bjNHHdWb9U88yugxYzTULmmriDTzeJR6Xvq0kGFpiC2D1LNnT9Xuk7TTzCv2N0Jr2ZS+upoaLEERWQmPjlGLDiuWL2PxzYt0HZSkVLEBSgsSlj5VMi9dXYOWSDwVNwfVQannIiLhFkiAMrNzzGyzmW0zs24ZBmZ2sZm9Er39j5mNCeK4udKChCIi4ZVzkoSZHQ5sAc4GIsALwIXu/nrMNqcDG9292czOBRa5+ymp9l2o9aASrWWjNW5ERIJXyFp844Ft7r7d3Q8ADwJTYzdw9/9x9+bo3d8CDQEcNzBd17LR0hwiIsUXRIAaAuyMuR+JPpbINODRRE+a2XQzW29m6/fs2ZNz47KpEqHro0REii+IAGVxHos7bmhmn6c9QC1ItDN3v8/dG929sV+/flk3KtteUKJKzKoXJiJSWEEEqAgwNOZ+A9CtqJaZjQZWAVPdfW8Ax00q216QluYQEQmHIALUC8CxZnaMmR0BXACsjd3AzIYBPwEucfctARwzqVx6QVqaQ0QkHHIOUO7eBlwNrAM2Ag+5+wYzm2FmM6Kb3Qj0Ae4xsyYzy2tqXi69IF0fJSISDoFUknD3R4BHujy2Mubnq4CrgjhWOnKtEtFR5XzulInd6oVJ+NTX1sZdA6oHcDDO9nU1NapCIVICyrZY7Jy583jm+fWHhvk6ekFnjm9kxfJlae1D10GVhqRFYxM9HuLfe5Fyl+51UGUboNra2lQ1uUIoQImUlooPUB3UCyp/ClAipUXLbUSparKISGmq2GrmIiISbgpQUvLqamow6HbrEecxQwsahkE2Jcik8ihAScnbt38/7t7tdiDOY+6uFPMiUiFmyYQClIgUjAoxSyYUoESkIFpbW1m1ahX/9LXr6NmrPaNWhZglGQUoEcm7trY2rp01i7/85S/cdcN1fPXzjdx/62I+amtTIWZJSAFKRPJu/oKFNG3exj2//A13r3uOO37+NG9uep0frLhFhZglobK/DkpEiqtjdYEVP/tVp9UFrllyB7P+YQJvbnxNhZglLvWgRCSvkq0u0OOIv+GkTx2bViFmpaZXHgUoEcmrZGusfdx2gO/eeWfS+phKTa9cClAiklfJ1libduW0lEN7Sk2vXGVfLFZEii/b1QVaW1sZ0jC00/wVtAe4uVMmEtm5Q3NXJUjFYtOgSucihVFVVcWK5ctYfPOijP7m0lkdW8Wgy1dFDvFpTFukODpWF0j3hDDZ/JVS08tfRQYojWmLlIZk81dKTU9PfW0tZtbtVl9bm5fXBSmQOSgzOwe4AzgcWOXuS7o8b9HnJwN/Bi5399+n2m8+5qCSjWlf96UJbNm0iYEDBwZ6TAmX+tpamltauj1eV1OjQrIhpNWxc5N0Qc8k3//Zvi7NNqU1B5VzD8rMDgfuBs4FRgEXmtmoLpudCxwbvU0H7s31uNlKNqZddURPPnnsSA33lZmuZ4LNLS04dLvFC1pSfB3zV5GdO3hi3aNEdu5gxfJlCk4VIIghvvHANnff7u4HgAeBqV22mQp839v9FuhtZoMCOHbGko1pHzzwF5Y+/JiG+8pM14AkpSnT+SspfUEEqCHAzpj7kehjmW4DgJlNN7P1ZrZ+z549ATSvs0Rj2t9dOIuJ/3wBg4Yfo+rKIiIhEESAsjiPdT1RTWeb9gfd73P3Rndv7NevX86Ni+fWpUs4c3wj131pAld99iRmf+nzHP2pUXxlzg0Aqq4sIhICQQSoCDA05n4D0PWbPZ1tCqZjTHvLpk0c+ODPLPnRL7hs/o0cHh3TVgqriJSLupoaDLrdekDSLL1Er6urqSlY24MIUC8Ax5rZMWZ2BHABsLbLNmuBS63dqcB77v5WAMfOycCBA5k+fTprvnWDUlgrSB3d/+gK/YcnUij79u/H3bvdDtI9USg2WSjR6wqZ6ZpzGoy7t5nZ1cA62tPM17j7BjObEX1+JfAI7Snm22hPM78i1+MG5dalS5i/YCFzp0zslsIq5aGupgZTWrlIyVEtviiVPRKRSpLP65zSOLZq8WWiI4VVRETCoSJLHYmISPgpQEnZCUMNMZGwC0OWXioa4pOy01E5oqt4iRIilaoUEoTUgxIRkVBSgBIRkVBSgBJJQXNaIsWhOSiRFDSnJVIc6kFJ2SmF7CQRSU09KCk7pZCdJCKpqQclIiKhpAAlIiKhpAAlkoLmtESKQwFKJIUwrIsjko5yuyRCAUokgXL7Y5fy13FJRKJFCEuNsvhEEtD1TyLFpR6UiIiEkgKUSA40DCiSPzkFKDOrN7NfmtnW6L91cbYZama/MrONZrbBzGblckyRMCm3MX+pPGE+ycq1B7UQeNLdjwWejN7vqg2Y6+7HA6cCXzezUTkeV0REusjmkogwn2TlGqCmAvdHf74f+HLXDdz9LXf/ffTnFmAjMCTH44rkna5/EimuXLP4Brj7W9AeiMysf7KNzWw4cBLwuxyPK5J3us5JSk25ZZ6mDFBm9gQwMM5T38zkQGZWDfwYmO3uCf/yzWw6MB1g2LBhmRxCRETKSMoA5e5fSPScmb1tZoOivadBwO4E2/WgPTj9p7v/JMXx7gPuA2hsbIx3MiASGlW0D/vFe1wk3+pra7vNFRlQB+wrSouClesc1FrgsujPlwE/7bqBmRmwGtjo7ityPJ5IqNQkmI9K9LhIkBImOGSwjzDPteYaoJYAZ5vZVuDs6H3MbLCZPRLd5gzgEmCimTVFb5NzPK5IKCSq0weENnVXJFaYa03mNBLh7nuBs+I8vguYHP3518QfBREpW+U2WS2lJ/ZLNwy9oWxoqFxEpAx19ORLmUodiYhIKClAiYiUqDAnOARBQ3wiIiUqDIkM+aQelEgelPuZrWQnzIVZw0g9KJE8KPczW8lOS4IszkSPVzr1oERCQmfX5a+N7hfVevRx6U49KJEiiVemRtdOifyVApRIkXS9mFdXs4t0piE+EREJJQUoEREJJQUoEZEC0eUHmdEclEhI1BF/HkoWsL4pAAAIC0lEQVRfXuVDlx9kRgFKpEjqamriZujV1dToi0wEBSiRolEQEklOc1AiIkWii7OTU4ASEUlDPoJJwiXbdXE2oCE+EZG0aJXkwlMPSkREQimnAGVm9Wb2SzPbGv23Lsm2h5vZS2b281yOKVLpsh1q0nyHlJpce1ALgSfd/Vjgyej9RGYBG3M8nkjFy3beQvMdUmpyDVBTgfujP98PfDneRmbWAPw9sCrH44mIlA1Vlkgu1ySJAe7+FoC7v2Vm/RNsdzswH0j5v25m04HpAMOGDcuxeSIiwUh2YXW2dC1ccil7UGb2hJm9Fuc2NZ0DmNk/ALvd/cV0tnf3+9y90d0b+/Xrl85LRMqG5okyV6j/s3379+Pu3W6Jgow+y9yl7EG5+xcSPWdmb5vZoGjvaRCwO85mZwBTzGwy0BOoNbMfuPtXsm61SJlSKnPmwvp/FtZ2lZJc56DWApdFf74M+GnXDdz9endvcPfhwAXAUwpOItnLdt5C8x1SanKdg1oCPGRm04AdwPkAZjYYWOXuk3Pcv4h0ke28heY7pNTkFKDcfS9wVpzHdwHdgpO7Pw08ncsxRUSkMqiShIiIhJIClEiIFHueqBQzz4r9f1Zq7SolKhYrEiLZzhPV19bGrQiR6eKH8TLP6qOPm3Ve7zcsCyuGoQ3xhLVdpUQ9KJEykM8yRs1x9hvGEkml2PuT5NSDEpGyoOuOyo96UCIiEkoKUCJSFBqSk1QUoETkkHiZZ/lSyOU/FPRKkwKUSBkIKqU5XkHUINOlY3tNxOyrPuM9ZaZr0FPvrTQoSUKkDOQzpTnIfSdMZAhg3wmXw6A9EzGtdiihIlTUgxKRQBWrd9IRSLsOGe5L1V46D2eqNxUe6kGJSKBKrXfScZ1XV2FtbyVRgBKRUOjowagUkHRQgBKRUHCP14/JTD6WZZfiUYASkYLJdwBJN6EjUTskXBSgRKRgwlJANbYdXYvgSngoi09EAlVqy0yUWnsriXpQIhKosPSS0lVq7a0k6kGJiEgo5RSgzKzezH5pZluj/9Yl2K63mT1sZpvMbKOZnZbLcUVEpPzl2oNaCDzp7scCT0bvx3MH8Ji7fwoYA2zM8bgiIlLmcg1QU4H7oz/fD3y56wZmVgt8DlgN4O4H3P3dHI8rIiJlLtcANcDd3wKI/ts/zjYjgD3A98zsJTNbZWa9Eu3QzKab2XozW79nz54cmyciIqUqZYAysyfM7LU4t6lpHqMKGAfc6+4nAe+TeCgQd7/P3RvdvbFfv35pHkJERMpNyjRzd/9CoufM7G0zG+Tub5nZIGB3nM0iQMTdfxe9/zBJApSIiAjkPsS3Frgs+vNlwE+7buDufwJ2mtlx0YfOAl7P8bgiIlLmcg1QS4CzzWwrcHb0PmY22MweidnuGuA/zewVYCxwS47HFRGRMmdBVBDOFzPbA7yZp933Bd7J077DQu+xPOg9lge9x7862t1TJhmEOkDlk5mtd/fGYrcjn/Qey4PeY3nQe8ycSh2JiEgoKUCJiEgoVXKAuq/YDSgAvcfyoPdYHvQeM1Sxc1AiIhJuldyDEhGREFOAEhGRUKqYAGVm55vZBjP72MwSpkGa2TlmttnMtplZSZVkymB9rjfM7FUzazKz9YVuZzZSfS7W7s7o86+Y2bhitDMXabzHCWb2XvRzazKzG4vRzmyZ2Roz221mryV4vhw+w1TvsaQ/QwAzG2pmv4qu7bfBzGbF2SaYz9LdK+IGHA8cBzwNNCbY5nDgD7RXYD8CeBkYVey2Z/AebwUWRn9eCCxNsN0bQN9itzeD95XycwEmA48CBpwK/K7Y7c7De5wA/LzYbc3hPX6O9sLRryV4vqQ/wzTfY0l/htH3MAgYF/25BtiSr7/HiulBuftGd9+cYrPxwDZ33+7uB4AHaV/zqlSkXJ+rRKXzuUwFvu/tfgv0jhYwLhWl/ruXkrs/C+xLskmpf4bpvMeS5+5vufvvoz+30L4A7ZAumwXyWVZMgErTEGBnzP0I3f/jwyyd9bkAHHjczF40s+kFa1320vlcSv2zS7f9p5nZy2b2qJmdUJimFUypf4bpKpvP0MyGAycBv+vyVCCfZcrlNkqJmT0BDIzz1DfdvVul9Xi7iPNYqPLwk73HDHZzhrvvMrP+wC/NbFP0zC+s0vlcQv/ZpZBO+39Pew2zVjObDPw/4Ni8t6xwSv0zTEfZfIZmVg38GJjt7vu7Ph3nJRl/lmUVoDzJ2lVpigBDY+43ALty3Gegkr3HNNfnwt13Rf/dbWb/TfvwUpgDVDqfS+g/uxRStj/2S8DdHzGze8ysr7uXSwHSUv8MUyqXz9DMetAenP7T3X8SZ5NAPksN8XX2AnCsmR1jZkcAF9C+5lWpSLk+l5n1MrOajp+BLwJxM45CJJ3PZS1waTR76FTgvY7hzhKR8j2a2UAzs+jP42n/+91b8JbmT6l/himVw2cYbf9qYKO7r0iwWSCfZVn1oJIxs38Evgv0A35hZk3uPsnMBgOr3H2yu7eZ2dXAOtqzqta4+4YiNjtTS4CHzGwasAM4H9rX5yL6HoEBwH9H/0aqgP9y98eK1N60JPpczGxG9PmVwCO0Zw5tA/4MXFGs9mYjzfd4HvA1M2sDPgAu8GjKVCkwsx/SnsXW18wiwE1ADyiPzxDSeo8l/RlGnQFcArxqZk3Rx24AhkGwn6VKHYmISChpiE9EREJJAUpEREJJAUpEREJJAUpEREJJAUpEREJJAUpEREJJAUpERELp/wMD6aH7VhLRlQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "plt.scatter(X[y_db == 0, 0], X[y_db == 0, 1],\n", + " c='lightblue', marker='o', s=40,\n", + " edgecolor='black', \n", + " label='cluster 1')\n", + "plt.scatter(X[y_db == 1, 0], X[y_db == 1, 1],\n", + " c='red', marker='s', s=40,\n", + " edgecolor='black', \n", + " label='cluster 2')\n", + "plt.legend()\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Accelerating DBSCAN with RAPIDS\n", + "\n", + "While the world’s data doubles each year, CPU computing has hit a brick wall with the end of Moore’s law. For the same reasons, scientific computing and deep learning has turned to NVIDIA GPU acceleration, data analytics and machine learning where GPU acceleration is ideal. \n", + "\n", + "NVIDIA created RAPIDS – an open-source data analytics and machine learning acceleration platform that leverages GPUs to accelerate computations. RAPIDS is based on Python, has pandas-like and Scikit-Learn-like interfaces, is built on Apache Arrow in-memory data format, and can scale from 1 to multi-GPU to multi-nodes. RAPIDS integrates easily into the world’s most popular data science Python-based workflows. RAPIDS accelerates data science end-to-end – from data prep, to machine learning, to deep learning. And through Arrow, Spark users can easily move data into the RAPIDS platform for acceleration.\n", + "\n", + "So how do we use RAPIDS? First, we cast our data to a `pandas.DataFrame` and use that to create a `cudf.DataFrame`. " + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "import cudf\n", + "\n", + "X_df = pd.DataFrame({'fea%d'%i: X[:, i] for i in range(X.shape[1])})\n", + "X_gpu = cudf.DataFrame.from_pandas(X_df)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, we load the `DBSCAN` class from the `cuml` package and instantiate it in the same way we did with the `sklearn.cluster.DBSCAN` class." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "from cuml import DBSCAN as cumlDBSCAN\n", + "\n", + "db_gpu = cumlDBSCAN(eps=0.2, min_samples=2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `DBSCAN` class from `cuml` implements the same API as the `sklearn` version; we can use the `fit` and `fit_predict` methods to fit our `DBSCAN` model to the data and generate predictions." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "y_db_gpu = db_gpu.fit_predict(X_gpu)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lastly, let's visualize our results:" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XuUVOWV9/Hv1sbBobulmzs0iCRiRAeQtHhLIsEYlJlAZkbf5SVecQghKggE0KylyCQuUCBqvDAOkDFmJsbXZN6QRMWoUVecJIqxvSDXMAoVjCC00m000LrfP7qaVHfXvU5Vnar6fdaqRVfVqXOeorprn+d59tmPuTsiIiJhc1ixGyAiIhKPApSIiISSApSIiISSApSIiISSApSIiISSApSIiISSApSIiISSApSIiISSApSIiIRSVbEbkEzfvn19+PDhxW6GiIgE6MUXX3zH3ful2i7UAWr48OGsX7++2M0QEZEAmdmb6WynIT4REQklBSgREQklBSgREQmlUM9BiYgU28GDB4lEInz44YfFbkrJ6dmzJw0NDfTo0SOr1ytAiYgkEYlEqKmpYfjw4ZhZsZtTMtydvXv3EolEOOaYY7Lah4b4RESS+PDDD+nTp4+CU4bMjD59+uTU81SAEhFJQcEpO7n+vylASUVpbW1ly5YttLa2FrspIpKCApSEXhBBpa2tjTlz5zGkYShnTTqXIQ1DmTN3Hm1tbQG2VKRdIU6EFi1axLJlyzJ+3bvvvss999yT8/HvuusuPvnJT2JmvPPOOznvLx4FKAmtIIPK/AULeeb59az42a+447Ffs+Jnv+KZ59czf8HCPLRcKlUpnAhlE6DcnY8//rjTY2eccQZPPPEERx99dJDN60QBSkIrqKDS2trK6tWrmXnL7dT1HwBAXf8BzLzldtasWdPpLFdDgJKLfJ0Iff/732f06NGMGTOGSy65pNvzEyZMOFQW7p133qGjhumGDRsYP348Y8eOZfTo0WzdupWFCxfyhz/8gbFjx/KNb3wDgNtuu42TTz6Z0aNHc9NNNwHwxhtvcPzxxzNz5kzGjRvHzp07Ox3zpJNOIt+1UhWgJJQyCSqp7Nq1i9q6+kP76VDXfwC1vevYtWtXSZz5SrgF+Tsba8OGDXz729/mqaee4uWXX+aOO+5I+7UrV65k1qxZNDU1sX79ehoaGliyZAmf+MQnaGpq4rbbbuPxxx9n69atPP/88zQ1NfHiiy/y7LPPArB582YuvfRSXnrppbz2lBJRgJJQSieopGvw4MHsb95H8+63Oz3evPtt9r/bzODBgzUEKDkL8nc21lNPPcV5551H3759Aaivr0/7taeddhq33HILS5cu5c033+TII4/sts3jjz/O448/zkknncS4cePYtGkTW7duBeDoo4/m1FNPzardQVCAklBKJ6ikq7q6mmnTpnHPDbMP7a9599vcc8NsrrzySoC8nPlKZQnydzaWu6dM166qqjo0RxR73dFFF13E2rVrOfLII5k0aRJPPfVU3P1ff/31NDU10dTUxLZt25g2bRoAvXr1yqrNQVGAklBKFVSqq6sz2t+tS5dw5vhG5k6ZyOxzPsPcKRM5c3wjty5dkrcz3yBoTqx0BP072+Gss87ioYceYu/evQDs27ev2zbDhw/nxRdfBODhhx8+9Pj27dsZMWIE1157LVOmTOGVV16hpqaGlpaWQ9tMmjSp04nYH//4R3bv3p1VW4MWSIAyszVmttvMXkvwvJnZnWa2zcxeMbNxQRxXSk+6X7itra1cNe1KTj9pTNygkqmqqipWLF9GZOcOnlj3KJGdO1ixfBlVVVV5O/PNhebESlOyE6FsnXDCCXzzm9/kzDPPZMyYMcyZM6fbNvPmzePee+/l9NNP75Ty/aMf/YgTTzyRsWPHsmnTJi699FL69OnDGWecwYknnsg3vvENvvjFL3LRRRdx2mmn8Xd/93ecd955nQJYInfeeScNDQ1EIhFGjx7NVVddlfV7TMjdc74BnwPGAa8leH4y8ChgwKnA79LZ76c//WmX8nDw4EG/bs5crz2qtzcMH+G1R/X26+bM9YMHD6bc7utXX+MbNmzwlpaWvLXvujlzfdxnzvRVz77kP960y1c9+5KP+8yZft2cuSlf29LS4ps3bw60fbm0R4L1+uuvZ/yafPxOlKp4/3/Aek8ntqSzUVo7guFJAtS/ARfG3N8MDEq1TwWo8pHuF26xvpg7AuNRvet86PARflTvurgBNN5rUgXdTLW0tHjtUb0P/R903FY9+5If1btOX3oFlk2Akr/KJUAVag5qCBCbRB+JPtaNmU03s/Vmtn7Pnj0FaZzEF9T8R7rpt/lI0033PSQbAkwkX5l/YZ4TEymkQgWoeCkoHm9Dd7/P3RvdvbFfv355bpbEE/T8R7pfuEF+MWf7Hqqrqxk5cmTKCe18XfMC+csGEyk1hQpQEWBozP0GQKeBIRV0zyDdL9wgv5jzfV1TPns5+coGEyk1hQpQa4FLo9l8pwLvuftbBTq2pKFjKOxPf/pT4D2DdL9wg/pizmfvpkO+ezn5yAYTKTVBpZn/EPgNcJyZRcxsmpnNMLMZ0U0eAbYD24B/B2YGcVzJXdehsGOPO47DevSgtr5Pp+1y7Rmk+4UbxBdzPno3Xeey8t3LyWZOTKTspJNJUaybsvjyL17W3KfGneyTLrg0Lxlk6abf5pKmG2QWXLJMvWwy/6T0hDWL76abbvLbbrst49c1Nzf73XffnfPxL7roIh85cqSfcMIJfsUVV/iBAwfiblcKWXwSQomGwubdfh9P//T/8tYb/wsE2zNINwkh3e0SvTao3k2yuax4vZzFNy9i+/btnTITVQmictTX1mJm3W71tbXFbtohQS23cfHFF7Np0yZeffVVPvjgA1atWhVkMwGVOqpoO3bs4G/+tleCobB6Fp5/bsnOfwQxVJjuXFZ1dTUjRozgxpsWxWQNNnDyKacyeEiDKkFUkOaWFhy63ZrTqMyQTBiX25g8efKhADx+/HgikUhO7zEeDWhXsHvuXXlooj82SDXvfpsDH/yZrVs2s3//fgYPHlxymWMdvZvFNy9i165dWb2HdOayRo4cCXTuadX1H0Dz7rdZPmcGp0+eyvRFSw714OYvWMiK5ZmvgpqN1tbWrN+7hEfHchvPPfccffv2jVuLL5GO5TYuvvhiDhw4wEcffcSSJUt47bXXaGpqAui03Ia7M2XKFJ599lmGDRvG5s2b+d73vpe0x3Xw4EEeeOCBjJYBSZd6UBWqtbWVBx54gAlfPp/vLpzVaShs2ezpXHLJJQwcODDrYbawyGWoMN1MvUQ9rbkrVvLco2v54P33C1odXXX8ykvYl9uYOXMmn/vc5/jsZz+bwbtKjwJUheroHVx03fX0HTyEWf8wga9POoPZX/o8b+94g6/N+Gqxm1h06c5lJetpVfeuo3n3nw7dL0QlCK1tVV7cw7vcxs0338yePXtYsWJFpm8rLQpQFap///7se2cPX/vCKWx44be4O6MaT+Hb//VTPm5rY9iwYcVuYiikM5eVrKfV+m4zdf0HHrqf70oQhbgGTAorrMttrFq1inXr1vHDH/6Qww7LTyhRgKpQi//1W4wYdSJ3/uIZ7l73HHf+4hne2fVHls68XNUKYqRzPVKintbyOTM449wpHNmrV8EqQaiOX3HV1dRg0O1WV1OT9T7DutzGjBkzePvttznttNMYO3Ysixcvzvo9JmLtKenh1NjY6B2ZKRKc1tZWhjQMPTSh36F599tcc+5n2BWJ0Lt37yK2sPS0tbUxf8FC1qxZQ23vOt57dx8jRx7Hls2bOaqunv3vNnPllVdy69Ileb3YNtlnO3fKRCI7d+jkI0MbN27k+OOPL3YzSla8/z8ze9HdG1O9Vll8FSjZWXbffgPYvXu3AlSGEmUNFjqTLrY31zHMpzp+UqoUoCpQ7JxJ17PsQlfLLrdU6I6swUT3C+HWpUuYv2Ahc6dMpLZ3Xafem0gp0RxUBQpDtWylQueP6vgFL8xTIWGW6/+bAlSFKna1bKVC518u14DJX/Xs2ZO9e/cqSGXI3dm7dy89e/bMeh9KkqgAyYbRijHEpol8KSUHDx4kEol0ur5I0tOzZ08aGhro0aNHp8eVJCGHMstWr15NbV09+5v3MW3atE6ZZMWYI8mkhJBIsfXo0YNjjjmm2M2oSBriK2NhHUbTkuYikg4FqDIV5ooCYUjSkOS0TIiEgQJUmQp7RYFiJ2lIfMqulDBRgCpTYR9GUyp0OIV1WFgqUyAByszOMbPNZrbNzLr9JpvZUWb2MzN72cw2mNkVQRy30iUbhimVYTSlQodHmIeFpTLlHKDM7HDgbuBcYBRwoZmN6rLZ14HX3X0MMAFYbmZH5HrsSpXuMIyG0SQTYR8WlsoTRA9qPLDN3be7+wHgQWBql20cqLH2RU2qgX2ABrWzlO4wjIbRJBNhHxaWyhNEgBoCxC5WH4k+Fusu4HhgF/AqMMvdPw7g2BUnm2EYDaNJOkplWFgqRxABKt5Sj13LU0wCmoDBwFjgLjOrjbszs+lmtt7M1u/ZsyeA5pUXDcNUrkKkfmtYWMIkiAAVAYbG3G+gvacU6wrgJ95uG/C/wKfi7czd73P3Rndv7NevXwDNKy8ahqk8hUz91rCwhEkQAeoF4FgzOyaa+HABsLbLNjuAswDMbABwHLA9gGOXvEzPijUMU3mKkfqtYWEJg0CKxZrZZOB24HBgjbt/28xmALj7SjMbDPwHMIj2IcEl7v6DVPst52Kx6dTJS/XajtVbC7VaqxReroV1y229LSkP6RaLVTXzIpkzdx7PPL++26qnZ45vZMXyZWntQ18+5W/Lli2cNelc7njs192em33OZ3hi3aNxC+vmcgIkkm/pBihVkiiCoC6I1DBM+ct2zlEVIaQcKEAVQUcmXs9e1ez63z/wwfvvA8rEk+6ymXNURQgpF+rrF0H//v3Z984epk/4NLX1fWhp3sdZ513I339lmjLxpJtbly5h/oKFzJ0ysducYzxab0vKhQJUESz+128xYtSJzF52z6H5pzsXXMvNV/wfZeJJNx2p34tvXpTWnGPssGDXxAqdAEkpUZJEgSXLyrrm3M+wKxKhd+/eRWyhlINMknCUbCOFpiSJkEo2/NK33wB2796d0f60sJzEk05FCK39JGGnAFVgQVWC0JeLJJNORQhl+knYKUAVWFCVIPTlIulIdCmCMv2kFChAFUGuBTn15SK5UtFhKQUKUEWQa0FOfblIrlR0WEqBAlQRZVsJQl8ukisVHZZSoABVgvTlIkHQ2k8SdroOqkSporkERddBSaGpmnmF0JeLiJQaXahbIVTRXHKli70lrBSgRCpUqou9Fbik2DRZIVKhYi/2jq3XN2/+fA6zw7TYoRSd5qBEKlCqosUjR5/ENUu/m/VqzyLJaA5KRBJKdrH33xzZi/OvnqcqJVJ0gQQoMzvHzDab2TYzi1sMzswmmFmTmW0ws2eCOK6IZCfZxd5/bt3P8E+d0OlxVSmRYsg5QJnZ4cDdwLnAKOBCMxvVZZvewD3AFHc/ATg/1+OKSPYSXex99/WzqKrqwYfvd+4pqUqJFEMQPajxwDZ33+7uB4AHgaldtrkI+Im77wBw98wWPSphyoSSsIpXSWLCKSfzL1ddpSolEgpBpOQMAXbG3I8Ap3TZZiTQw8yeBmqAO9z9+/F2ZmbTgekAw4YNC6B5xdFR6UGZUBJWiZaS7/jdnTtlYrcqJSKFlHMWn5mdD0xy96ui9y8Bxrv7NTHb3AU0AmcBRwK/Af7e3bck23epZfHFVnW48aZFaS+5LRJGqlIi+ZJuFl8Qp/IRYGjM/Qag60xqBHjH3d8H3jezZ4ExQNIAVSq69pbea97LRx99zHfWPtUtE2rulIksvnmR/uAl9DqqlGRKgU2CEsQc1AvAsWZ2jJkdAVwArO2yzU+Bz5pZlZn9Le1DgBsDOHYodF3d9js/e5phxx3PL36wutN2yoSScpaqMoVIpnIOUO7eBlwNrKM96Dzk7hvMbIaZzYhusxF4DHgFeB5Y5e6v5XrsMEi0uu287/wbTz78Qz54//1D2yoTSspZ1xO1FT/7Fc88v575C+JeeSKSUiDXQbn7I+4+0t0/4e7fjj620t1Xxmxzm7uPcvcT3f32II4bBskveDySNzZtAJQJJeUt0YmaLvCVXCidLEexFzx2LRnzQWsLt868nKPq6pUJJWUt2Ylax7B2NvNZUtlU6ihHyVa3nfHVGfwxspMn1j1KZOcOVixfphRzKUvJKlNoWFuypQAVgGRLZ2u9JqkEyU7UNKwt2VI18wApvVYqWcflFmvWrOl2ga9GDiSWlnwXkaKIPVEDdNIm3Wi5DREpiurqakaMGMGNNy3SNVGSEwUoEQmcromSIGiIT0QClWy13rlTJhLZuUPDfRVOQ3wiUhTpXBMlkg4FKBEJlK6JkqAoQIlIRlItwqlroiQoClBSkuprazGzbrf62tpiN61sZVKtPNnF6yLpUpKElCQzI95vrgFh/p0uZXPmzst4EU5dvC7x6ELdgOgPLJxiA1Q90Bxnm7qaGvbt31/AVpUvZeZJkJTFlyMtvlY6mgGPc2tuaSlms8qKMvOkGBSgEkh0oeF1c+Z2myBONWksUuqUmSfFoCG+OJINZ8w8+1T6DhhE63vvcsUVV4DB99Z8j9q6evY372PatGkqjlkAsUN8BpqPKoBs5qBE4kl3iE/fonEkG86o6z+Q6//tAY7sVc2Nl/wT9QMGHgpkHX+w8xcs1B9sntXV1GAawiuoW5cuYf6ChcydMrFbtXKRfFAPKo5kPajZX/o8K596AYDpEz7Nnb94RpPGBVRfWxt3bqkO2NflMfWg8kOJQ5KrgiZJmNk5ZrbZzLaZWcJqkGZ2spl9ZGbnBXHcfEl0oeF3F85i4j9fwJG9etG8+0/U1vfRpHGBNbe0xE+IoD0gxd7qamqK1cyypkU4pVByDlBmdjhwN3AuMAq40MxGJdhuKbAu12MWQuyFhrPOOYOZZ5/KwGHD+cqcGwCo6z+Q/fv2atI4RNy9000p5iKlLYge1Hhgm7tvd/cDwIPA1DjbXQP8GNgdwDHzrqqqihXLlxHZuYMn1z3Gv/zLdPZE3mT/vr0AfPh+K73r+3D7vJkq5yIikgdBJEkMAXbG3I8Ap8RuYGZDgH8EJgInJ9uZmU0HpgMMGzYsgOblprq6msGDBzPzazPg3pWdJogvv/xyMDRpLCKSB0EEKIvzWNeZ6duBBe7+kVm8zWNe6H4fcB+0J0kE0L6stbW1MX/BQlavXn0ojfzSSy/lazO+yrBhww71kr61eLEmjaWiKXFC8iGIIb4IMDTmfgPQNUOgEXjQzN4AzgPuMbMvB3DsvIp3se7/vPQyq1av6fRHqEnj/IotDFtF92QIJUQUjyquSD7lnGZuZlXAFuAs4I/AC8BF7r4hwfb/Afzc3R9Ote9i1uJT7bHwUGHY8NLFu5KNgqWZu3sbcDXt2XkbgYfcfYOZzTCzGbnuv1hUe0wkudbWVlavXn0oOEH738fMW25nzZo1Kv0lOQukkoS7PwI80uWxlQm2vTyIY+ZbbO2xrj0opZGLpHcSN3LkyCK1TsqBisUmoFVBRZJTAVnJNwWoJLQqqEhiOomTfFMtvjQohba4Etbf04KERddxKcaaNWu6XQuoiv6SiFbUFZGC0UmcZEIr6iagxQVFgqdrASUfKiZA6YJCkcLTCaHkomICVKIl3OcvSLg6iIhkqfMJ4TkMGjKEa66dpRNCyUhFzEGpKoRIYcWrMLFs9nSqqw7jN8/9mg8//FBzVhVMc1AxVBVCpHASVZiYd/t9vPrqK4w/9TQNtUtaKiJA6YLC8hRbRDb2Vl9bW+ymVbT2E8K6uCeE1UfV8d4Hf9FQu6SlIgKULigsTwmXf49zzZQUzuDBg3k30WrTzXuZs2KlavdJWioiQEHyqhDKNBIJTnV1NZdfdjnLZk/vdEJ454Jr+dteNQwafkyn7TXULolUTICKXcL9iXWPEtm5g1uXLmH+goUaDxcJ2HdWLKe66jC+Pul0Zp59GrO/9Hn6DRnKB++3aqhd0lYxAapD7AWFSj0XyY+qqip+89yvuWraVbz/bjNHHdWb9U88yugxYzTULmmriDTzeJR6Xvq0kGFpiC2D1LNnT9Xuk7TTzCv2N0Jr2ZS+upoaLEERWQmPjlGLDiuWL2PxzYt0HZSkVLEBSgsSlj5VMi9dXYOWSDwVNwfVQannIiLhFkiAMrNzzGyzmW0zs24ZBmZ2sZm9Er39j5mNCeK4udKChCIi4ZVzkoSZHQ5sAc4GIsALwIXu/nrMNqcDG9292czOBRa5+ymp9l2o9aASrWWjNW5ERIJXyFp844Ft7r7d3Q8ADwJTYzdw9/9x9+bo3d8CDQEcNzBd17LR0hwiIsUXRIAaAuyMuR+JPpbINODRRE+a2XQzW29m6/fs2ZNz47KpEqHro0REii+IAGVxHos7bmhmn6c9QC1ItDN3v8/dG929sV+/flk3KtteUKJKzKoXJiJSWEEEqAgwNOZ+A9CtqJaZjQZWAVPdfW8Ax00q216QluYQEQmHIALUC8CxZnaMmR0BXACsjd3AzIYBPwEucfctARwzqVx6QVqaQ0QkHHIOUO7eBlwNrAM2Ag+5+wYzm2FmM6Kb3Qj0Ae4xsyYzy2tqXi69IF0fJSISDoFUknD3R4BHujy2Mubnq4CrgjhWOnKtEtFR5XzulInd6oVJ+NTX1sZdA6oHcDDO9nU1NapCIVICyrZY7Jy583jm+fWHhvk6ekFnjm9kxfJlae1D10GVhqRFYxM9HuLfe5Fyl+51UGUboNra2lQ1uUIoQImUlooPUB3UCyp/ClAipUXLbUSparKISGmq2GrmIiISbgpQUvLqamow6HbrEecxQwsahkE2Jcik8ihAScnbt38/7t7tdiDOY+6uFPMiUiFmyYQClIgUjAoxSyYUoESkIFpbW1m1ahX/9LXr6NmrPaNWhZglGQUoEcm7trY2rp01i7/85S/cdcN1fPXzjdx/62I+amtTIWZJSAFKRPJu/oKFNG3exj2//A13r3uOO37+NG9uep0frLhFhZglobK/DkpEiqtjdYEVP/tVp9UFrllyB7P+YQJvbnxNhZglLvWgRCSvkq0u0OOIv+GkTx2bViFmpaZXHgUoEcmrZGusfdx2gO/eeWfS+phKTa9cClAiklfJ1libduW0lEN7Sk2vXGVfLFZEii/b1QVaW1sZ0jC00/wVtAe4uVMmEtm5Q3NXJUjFYtOgSucihVFVVcWK5ctYfPOijP7m0lkdW8Wgy1dFDvFpTFukODpWF0j3hDDZ/JVS08tfRQYojWmLlIZk81dKTU9PfW0tZtbtVl9bm5fXBSmQOSgzOwe4AzgcWOXuS7o8b9HnJwN/Bi5399+n2m8+5qCSjWlf96UJbNm0iYEDBwZ6TAmX+tpamltauj1eV1OjQrIhpNWxc5N0Qc8k3//Zvi7NNqU1B5VzD8rMDgfuBs4FRgEXmtmoLpudCxwbvU0H7s31uNlKNqZddURPPnnsSA33lZmuZ4LNLS04dLvFC1pSfB3zV5GdO3hi3aNEdu5gxfJlCk4VIIghvvHANnff7u4HgAeBqV22mQp839v9FuhtZoMCOHbGko1pHzzwF5Y+/JiG+8pM14AkpSnT+SspfUEEqCHAzpj7kehjmW4DgJlNN7P1ZrZ+z549ATSvs0Rj2t9dOIuJ/3wBg4Yfo+rKIiIhEESAsjiPdT1RTWeb9gfd73P3Rndv7NevX86Ni+fWpUs4c3wj131pAld99iRmf+nzHP2pUXxlzg0Aqq4sIhICQQSoCDA05n4D0PWbPZ1tCqZjTHvLpk0c+ODPLPnRL7hs/o0cHh3TVgqriJSLupoaDLrdekDSLL1Er6urqSlY24MIUC8Ax5rZMWZ2BHABsLbLNmuBS63dqcB77v5WAMfOycCBA5k+fTprvnWDUlgrSB3d/+gK/YcnUij79u/H3bvdDtI9USg2WSjR6wqZ6ZpzGoy7t5nZ1cA62tPM17j7BjObEX1+JfAI7Snm22hPM78i1+MG5dalS5i/YCFzp0zslsIq5aGupgZTWrlIyVEtviiVPRKRSpLP65zSOLZq8WWiI4VVRETCoSJLHYmISPgpQEnZCUMNMZGwC0OWXioa4pOy01E5oqt4iRIilaoUEoTUgxIRkVBSgBIRkVBSgBJJQXNaIsWhOSiRFDSnJVIc6kFJ2SmF7CQRSU09KCk7pZCdJCKpqQclIiKhpAAlIiKhpAAlkoLmtESKQwFKJIUwrIsjko5yuyRCAUokgXL7Y5fy13FJRKJFCEuNsvhEEtD1TyLFpR6UiIiEkgKUSA40DCiSPzkFKDOrN7NfmtnW6L91cbYZama/MrONZrbBzGblckyRMCm3MX+pPGE+ycq1B7UQeNLdjwWejN7vqg2Y6+7HA6cCXzezUTkeV0REusjmkogwn2TlGqCmAvdHf74f+HLXDdz9LXf/ffTnFmAjMCTH44rkna5/EimuXLP4Brj7W9AeiMysf7KNzWw4cBLwuxyPK5J3us5JSk25ZZ6mDFBm9gQwMM5T38zkQGZWDfwYmO3uCf/yzWw6MB1g2LBhmRxCRETKSMoA5e5fSPScmb1tZoOivadBwO4E2/WgPTj9p7v/JMXx7gPuA2hsbIx3MiASGlW0D/vFe1wk3+pra7vNFRlQB+wrSouClesc1FrgsujPlwE/7bqBmRmwGtjo7ityPJ5IqNQkmI9K9LhIkBImOGSwjzDPteYaoJYAZ5vZVuDs6H3MbLCZPRLd5gzgEmCimTVFb5NzPK5IKCSq0weENnVXJFaYa03mNBLh7nuBs+I8vguYHP3518QfBREpW+U2WS2lJ/ZLNwy9oWxoqFxEpAx19ORLmUodiYhIKClAiYiUqDAnOARBQ3wiIiUqDIkM+aQelEgelPuZrWQnzIVZw0g9KJE8KPczW8lOS4IszkSPVzr1oERCQmfX5a+N7hfVevRx6U49KJEiiVemRtdOifyVApRIkXS9mFdXs4t0piE+EREJJQUoEREJJQUoEZEC0eUHmdEclEhI1BF/HkoWsL4pAAAIC0lEQVRfXuVDlx9kRgFKpEjqamriZujV1dToi0wEBSiRolEQEklOc1AiIkWii7OTU4ASEUlDPoJJwiXbdXE2oCE+EZG0aJXkwlMPSkREQimnAGVm9Wb2SzPbGv23Lsm2h5vZS2b281yOKVLpsh1q0nyHlJpce1ALgSfd/Vjgyej9RGYBG3M8nkjFy3beQvMdUmpyDVBTgfujP98PfDneRmbWAPw9sCrH44mIlA1Vlkgu1ySJAe7+FoC7v2Vm/RNsdzswH0j5v25m04HpAMOGDcuxeSIiwUh2YXW2dC1ccil7UGb2hJm9Fuc2NZ0DmNk/ALvd/cV0tnf3+9y90d0b+/Xrl85LRMqG5okyV6j/s3379+Pu3W6Jgow+y9yl7EG5+xcSPWdmb5vZoGjvaRCwO85mZwBTzGwy0BOoNbMfuPtXsm61SJlSKnPmwvp/FtZ2lZJc56DWApdFf74M+GnXDdz9endvcPfhwAXAUwpOItnLdt5C8x1SanKdg1oCPGRm04AdwPkAZjYYWOXuk3Pcv4h0ke28heY7pNTkFKDcfS9wVpzHdwHdgpO7Pw08ncsxRUSkMqiShIiIhJIClEiIFHueqBQzz4r9f1Zq7SolKhYrEiLZzhPV19bGrQiR6eKH8TLP6qOPm3Ve7zcsCyuGoQ3xhLVdpUQ9KJEykM8yRs1x9hvGEkml2PuT5NSDEpGyoOuOyo96UCIiEkoKUCJSFBqSk1QUoETkkHiZZ/lSyOU/FPRKkwKUSBkIKqU5XkHUINOlY3tNxOyrPuM9ZaZr0FPvrTQoSUKkDOQzpTnIfSdMZAhg3wmXw6A9EzGtdiihIlTUgxKRQBWrd9IRSLsOGe5L1V46D2eqNxUe6kGJSKBKrXfScZ1XV2FtbyVRgBKRUOjowagUkHRQgBKRUHCP14/JTD6WZZfiUYASkYLJdwBJN6EjUTskXBSgRKRgwlJANbYdXYvgSngoi09EAlVqy0yUWnsriXpQIhKosPSS0lVq7a0k6kGJiEgo5RSgzKzezH5pZluj/9Yl2K63mT1sZpvMbKOZnZbLcUVEpPzl2oNaCDzp7scCT0bvx3MH8Ji7fwoYA2zM8bgiIlLmcg1QU4H7oz/fD3y56wZmVgt8DlgN4O4H3P3dHI8rIiJlLtcANcDd3wKI/ts/zjYjgD3A98zsJTNbZWa9Eu3QzKab2XozW79nz54cmyciIqUqZYAysyfM7LU4t6lpHqMKGAfc6+4nAe+TeCgQd7/P3RvdvbFfv35pHkJERMpNyjRzd/9CoufM7G0zG+Tub5nZIGB3nM0iQMTdfxe9/zBJApSIiAjkPsS3Frgs+vNlwE+7buDufwJ2mtlx0YfOAl7P8bgiIlLmcg1QS4CzzWwrcHb0PmY22MweidnuGuA/zewVYCxwS47HFRGRMmdBVBDOFzPbA7yZp933Bd7J077DQu+xPOg9lge9x7862t1TJhmEOkDlk5mtd/fGYrcjn/Qey4PeY3nQe8ycSh2JiEgoKUCJiEgoVXKAuq/YDSgAvcfyoPdYHvQeM1Sxc1AiIhJuldyDEhGREFOAEhGRUKqYAGVm55vZBjP72MwSpkGa2TlmttnMtplZSZVkymB9rjfM7FUzazKz9YVuZzZSfS7W7s7o86+Y2bhitDMXabzHCWb2XvRzazKzG4vRzmyZ2Roz221mryV4vhw+w1TvsaQ/QwAzG2pmv4qu7bfBzGbF2SaYz9LdK+IGHA8cBzwNNCbY5nDgD7RXYD8CeBkYVey2Z/AebwUWRn9eCCxNsN0bQN9itzeD95XycwEmA48CBpwK/K7Y7c7De5wA/LzYbc3hPX6O9sLRryV4vqQ/wzTfY0l/htH3MAgYF/25BtiSr7/HiulBuftGd9+cYrPxwDZ33+7uB4AHaV/zqlSkXJ+rRKXzuUwFvu/tfgv0jhYwLhWl/ruXkrs/C+xLskmpf4bpvMeS5+5vufvvoz+30L4A7ZAumwXyWVZMgErTEGBnzP0I3f/jwyyd9bkAHHjczF40s+kFa1320vlcSv2zS7f9p5nZy2b2qJmdUJimFUypf4bpKpvP0MyGAycBv+vyVCCfZcrlNkqJmT0BDIzz1DfdvVul9Xi7iPNYqPLwk73HDHZzhrvvMrP+wC/NbFP0zC+s0vlcQv/ZpZBO+39Pew2zVjObDPw/4Ni8t6xwSv0zTEfZfIZmVg38GJjt7vu7Ph3nJRl/lmUVoDzJ2lVpigBDY+43ALty3Gegkr3HNNfnwt13Rf/dbWb/TfvwUpgDVDqfS+g/uxRStj/2S8DdHzGze8ysr7uXSwHSUv8MUyqXz9DMetAenP7T3X8SZ5NAPksN8XX2AnCsmR1jZkcAF9C+5lWpSLk+l5n1MrOajp+BLwJxM45CJJ3PZS1waTR76FTgvY7hzhKR8j2a2UAzs+jP42n/+91b8JbmT6l/himVw2cYbf9qYKO7r0iwWSCfZVn1oJIxs38Evgv0A35hZk3uPsnMBgOr3H2yu7eZ2dXAOtqzqta4+4YiNjtTS4CHzGwasAM4H9rX5yL6HoEBwH9H/0aqgP9y98eK1N60JPpczGxG9PmVwCO0Zw5tA/4MXFGs9mYjzfd4HvA1M2sDPgAu8GjKVCkwsx/SnsXW18wiwE1ADyiPzxDSeo8l/RlGnQFcArxqZk3Rx24AhkGwn6VKHYmISChpiE9EREJJAUpEREJJAUpEREJJAUpEREJJAUpEREJJAUpEREJJAUpERELp/wMD6aH7VhLRlQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "plt.scatter(X[y_db_gpu == 0, 0], X[y_db_gpu == 0, 1],\n", + " c='lightblue', marker='o', s=40,\n", + " edgecolor='black', \n", + " label='cluster 1')\n", + "plt.scatter(X[y_db_gpu == 1, 0], X[y_db_gpu == 1, 1],\n", + " c='red', marker='s', s=40,\n", + " edgecolor='black', \n", + " label='cluster 2')\n", + "plt.legend()\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Benchmarking: Comparing GPU and CPU\n", + "\n", + "RAPIDS uses GPUs to parallelize operations and accelerate computations. We saw porting an example from the traditional scikit-learn interface to cuML was trivial. So how much speedup do we get from using RAPIDS? \n", + "\n", + "The answer to this question varies depending on the size and shape of the data. In the below example, we generate a matrix of 10,000 rows by 128 columns and show we were able to reduce computational time from ~45 seconds to ~5 seconds - almost a 9x speedup. Feel free to change the number of rows and columns to see how this speedup might change depending on the size and shape of the data.\n", + "\n", + "As a good rule of thumb, larger datasets will benefit from RAPIDS. There is overhead associated with using a GPU; data has to be transferred from the CPU to the GPU, computations have to take place on the GPU, and the results need to be transferred back from the GPU to the CPU. However, the transactional overhead of moving data back and forth from the CPU to the GPU can quickly become negligible due to the performance speedup from computing on a GPU instead of a CPU." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(10000, 128)\n" + ] + } + ], + "source": [ + "import numpy as np\n", + "\n", + "n_rows, n_cols = 10000, 128\n", + "X = np.random.rand(n_rows, n_cols)\n", + "print(X.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### GPU" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [], + "source": [ + "X_df = pd.DataFrame({'fea%d'%i: X[:, i] for i in range(X.shape[1])})\n", + "X_gpu = cudf.DataFrame.from_pandas(X_df)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "db_gpu = cumlDBSCAN(eps=3, min_samples=2)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 181 ms, sys: 39.1 ms, total: 220 ms\n", + "Wall time: 219 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "\n", + "y_db_gpu = db_gpu.fit_predict(X_gpu)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### CPU" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "db = DBSCAN(eps=3, min_samples=2)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 26.1 s, sys: 5.04 ms, total: 26.1 s\n", + "Wall time: 26.1 s\n" + ] + } + ], + "source": [ + "%%time\n", + "\n", + "y_db = db.fit_predict(X)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion\n", + "\n", + "In conclusion, there are certain cases the DBSCAN algorithm can do a better job of clustering than traditional algorithms such as K Means or Agglomerative Clustering. Additionally, porting DBSCAN from CPU to GPU using RAPIDS is a trivial exercise and can yield massive performance gains.\n", + "\n", + "To learn more about RAPIDS, be sure to check out: \n", + "\n", + "* [Open Source Website](http://rapids.ai)\n", + "* [GitHub](https://github.com/rapidsai/)\n", + "* [Press Release](https://nvidianews.nvidia.com/news/nvidia-introduces-rapids-open-source-gpu-acceleration-platform-for-large-scale-data-analytics-and-machine-learning)\n", + "* [NVIDIA Blog](https://blogs.nvidia.com/blog/2018/10/10/rapids-data-science-open-source-community/)\n", + "* [Developer Blog](https://devblogs.nvidia.com/gpu-accelerated-analytics-rapids/)\n", + "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Disclaimer: The above examples borrowed code snippets from *Python Machine Learning, 2nd Ed.* by Sebastian Raschka and Vahid Mirjalili. For a great deep dive into these concepts, the curious reader is strongly encouraged to explore that fantastic resource." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/xgboost/XGBoost_Demo.ipynb b/xgboost/XGBoost_Demo.ipynb new file mode 100644 index 00000000..86eacf86 --- /dev/null +++ b/xgboost/XGBoost_Demo.ipynb @@ -0,0 +1,505 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Introduction to XGBoost with RAPIDS\n", + "#### By Paul Hendricks\n", + "-------\n", + "\n", + "While the world’s data doubles each year, CPU computing has hit a brick wall with the end of Moore’s law. For the same reasons, scientific computing and deep learning has turned to NVIDIA GPU acceleration, data analytics and machine learning where GPU acceleration is ideal. \n", + "\n", + "NVIDIA created RAPIDS – an open-source data analytics and machine learning acceleration platform that leverages GPUs to accelerate computations. RAPIDS is based on Python, has pandas-like and Scikit-Learn-like interfaces, is built on Apache Arrow in-memory data format, and can scale from 1 to multi-GPU to multi-nodes. RAPIDS integrates easily into the world’s most popular data science Python-based workflows. RAPIDS accelerates data science end-to-end – from data prep, to machine learning, to deep learning. And through Arrow, Spark users can easily move data into the RAPIDS platform for acceleration.\n", + "\n", + "In this notebook, we'll show the acceleration one can gain by using GPUs with XGBoost in RAPIDS.\n", + "\n", + "**Table of Contents**\n", + "\n", + "* Setup\n", + "* Load Libraries\n", + "* Load/Simulate Data\n", + " * Load Data\n", + " * Simulate Data\n", + " * Split Data\n", + " * Check Dimensions\n", + "* Convert NumPy data to DMatrix format\n", + "* Set Parameters\n", + "* Train Model\n", + "* Conclusion" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "This notebook was tested using the `nvcr.io/nvidia/rapidsai/rapidsai:0.5-cuda10.0-runtime-ubuntu18.04-gcc7-py3.7` Docker container from [NVIDIA GPU Cloud](https://ngc.nvidia.com) and run on the NVIDIA Tesla V100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. \n", + "\n", + "If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks/issues\n", + "\n", + "To start, let's see what hardware we're working with." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "ExecuteTime": { + "end_time": "2018-11-06T21:03:38.237293Z", + "start_time": "2018-11-06T21:03:37.388285Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Tue May 7 00:32:32 2019 \n", + "+-----------------------------------------------------------------------------+\n", + "| NVIDIA-SMI 418.39 Driver Version: 418.39 CUDA Version: 10.1 |\n", + "|-------------------------------+----------------------+----------------------+\n", + "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", + "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", + "|===============================+======================+======================|\n", + "| 0 Quadro GV100 Off | 00000000:15:00.0 Off | Off |\n", + "| 29% 39C P2 26W / 250W | 8902MiB / 32478MiB | 0% Default |\n", + "+-------------------------------+----------------------+----------------------+\n", + "| 1 Quadro GV100 Off | 00000000:2D:00.0 On | Off |\n", + "| 33% 46C P0 29W / 250W | 260MiB / 32470MiB | 27% Default |\n", + "+-------------------------------+----------------------+----------------------+\n", + " \n", + "+-----------------------------------------------------------------------------+\n", + "| Processes: GPU Memory |\n", + "| GPU PID Type Process name Usage |\n", + "|=============================================================================|\n", + "+-----------------------------------------------------------------------------+\n" + ] + } + ], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, let's see what CUDA version we have." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "ExecuteTime": { + "end_time": "2018-11-06T21:03:39.490984Z", + "start_time": "2018-11-06T21:03:39.134608Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "nvcc: NVIDIA (R) Cuda compiler driver\n", + "Copyright (c) 2005-2018 NVIDIA Corporation\n", + "Built on Sat_Aug_25_21:08:01_CDT_2018\n", + "Cuda compilation tools, release 10.0, V10.0.130\n" + ] + } + ], + "source": [ + "!nvcc --version" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load Libraries\n", + "\n", + "Let's load some of the libraries within the RAPIDs ecosystem and see which versions we have." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "ExecuteTime": { + "end_time": "2018-11-06T21:03:41.067879Z", + "start_time": "2018-11-06T21:03:40.256654Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "numpy Version: 1.16.2\n", + "pandas Version: 0.23.4\n", + "XGBoost Version: 0.83.dev0\n" + ] + } + ], + "source": [ + "import numpy as np; print('numpy Version:', np.__version__)\n", + "import pandas as pd; print('pandas Version:', pd.__version__)\n", + "import xgboost as xgb; print('XGBoost Version:', xgb.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load/Simulate data\n", + "\n", + "### Load Data\n", + "\n", + "We can load the data using `pandas.read_csv`.\n", + "\n", + "### Simulate Data\n", + "\n", + "Alternatively, we can simulate data for our train and validation datasets. The features will be tabular with `n_rows` and `n_columns` in the training dataset, where each value is either of type `np.float32` if the data is numerical or `np.uint8` if the data is categorical. Both numerical and categorical data can also be combined; for this experiment, we have ignored this combination." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# helper function for simulating data\n", + "def simulate_data(m, n, k=2, numerical=False):\n", + " if numerical:\n", + " features = np.random.rand(m, n)\n", + " else:\n", + " features = np.random.randint(2, size=(m, n))\n", + " labels = np.random.randint(k, size=m)\n", + " return np.c_[labels, features].astype(np.float32)\n", + "\n", + "\n", + "# helper function for loading data\n", + "def load_data(filename, n_rows):\n", + " if n_rows >= 1e9:\n", + " df = pd.read_csv(filename)\n", + " else:\n", + " df = pd.read_csv(filename, nrows=n_rows)\n", + " return df.values.astype(np.float32)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "# settings\n", + "LOAD = False\n", + "n_rows = int(1e5)\n", + "n_columns = int(100)\n", + "n_categories = 2" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(100000, 101)\n", + "CPU times: user 62.6 ms, sys: 72.5 ms, total: 135 ms\n", + "Wall time: 134 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "\n", + "if LOAD:\n", + " dataset = load_data('/tmp', n_rows)\n", + "else:\n", + " dataset = simulate_data(n_rows, n_columns, n_categories)\n", + "print(dataset.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Split Data\n", + "\n", + "We'll split our dataset into a 80% training dataset and a 20% validation dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "# identify shape and indices\n", + "n_rows, n_columns = dataset.shape\n", + "train_size = 0.80\n", + "train_index = int(n_rows * train_size)\n", + "\n", + "# split X, y\n", + "X, y = dataset[:, 1:], dataset[:, 0]\n", + "del dataset\n", + "\n", + "# split train data\n", + "X_train, y_train = X[:train_index, :], y[:train_index]\n", + "\n", + "# split validation data\n", + "X_validation, y_validation = X[train_index:, :], y[train_index:]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Check Dimensions\n", + "\n", + "We can check the dimensions and proportions of our training and validation dataets." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "X_train: (80000, 100) float32 y_train: (80000,) float32\n", + "X_validation (20000, 100) float32 y_validation: (20000,) float32\n", + "X_train proportion: 0.8\n", + "X_validation proportion: 0.2\n" + ] + } + ], + "source": [ + "# check dimensions\n", + "print('X_train: ', X_train.shape, X_train.dtype, 'y_train: ', y_train.shape, y_train.dtype)\n", + "print('X_validation', X_validation.shape, X_validation.dtype, 'y_validation: ', y_validation.shape, y_validation.dtype)\n", + "\n", + "# check the proportions\n", + "total = X_train.shape[0] + X_validation.shape[0]\n", + "print('X_train proportion:', X_train.shape[0] / total)\n", + "print('X_validation proportion:', X_validation.shape[0] / total)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Convert NumPy data to DMatrix format\n", + "\n", + "With out data simulated and formatted as NumPy arrays, our next step is to convert this to a `DMatrix` object that XGBoost can work with. We can instantiate an object of the `xgboost.DMatrix` by passing in the feature matrix as the first argument followed by the label vector using the `label=` keyword argument. To learn more about XGBoost's support for data structures other than NumPy arrays, see the documentation for the Data Interface:\n", + "\n", + "\n", + "https://xgboost.readthedocs.io/en/latest/python/python_intro.html#data-interface\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "ExecuteTime": { + "end_time": "2018-11-06T21:03:55.278322Z", + "start_time": "2018-11-06T21:03:54.059643Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 70.8 ms, sys: 43.5 ms, total: 114 ms\n", + "Wall time: 112 ms\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/conda/envs/rapids/lib/python3.7/site-packages/xgboost-0.83.dev0-py3.7.egg/xgboost/core.py:634: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption\n", + " warnings.warn(\"Use subset (sliced data) of np.ndarray is not recommended \" +\n" + ] + } + ], + "source": [ + "%%time\n", + "\n", + "dtrain = xgb.DMatrix(X_train, label=y_train)\n", + "dvalidation = xgb.DMatrix(X_validation, label=y_validation)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Set Parameters\n", + "\n", + "There are a number of parameters that can be set before XGBoost can be run. \n", + "\n", + "* General parameters relate to which booster we are using to do boosting, commonly tree or linear model\n", + "* Booster parameters depend on which booster you have chosen\n", + "* Learning task parameters decide on the learning scenario. For example, regression tasks may use different parameters with ranking tasks.\n", + "\n", + "For more information on the configurable parameters within the XGBoost module, see the documentation here:\n", + "\n", + "\n", + "https://xgboost.readthedocs.io/en/latest/parameter.html" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "ExecuteTime": { + "end_time": "2018-11-06T21:03:57.443698Z", + "start_time": "2018-11-06T21:03:57.438288Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'silent': 1, 'tree_method': 'gpu_hist', 'n_gpus': 1, 'eval_metric': 'auc', 'objective': 'binary:logistic'}\n" + ] + } + ], + "source": [ + "# instantiate params\n", + "params = {}\n", + "\n", + "# general params\n", + "general_params = {'silent': 1}\n", + "params.update(general_params)\n", + "\n", + "# booster params\n", + "n_gpus = 1\n", + "booster_params = {}\n", + "\n", + "if n_gpus != 0:\n", + " booster_params['tree_method'] = 'gpu_hist'\n", + " booster_params['n_gpus'] = n_gpus\n", + "params.update(booster_params)\n", + "\n", + "# learning task params\n", + "learning_task_params = {'eval_metric': 'auc', 'objective': 'binary:logistic'}\n", + "params.update(learning_task_params)\n", + "print(params)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Train Model\n", + "\n", + "Now it's time to train our model! We can use the `xgb.train` function and pass in the parameters, training dataset, the number of boosting iterations, and the list of items to be evaluated during training. For more information on the parameters that can be passed into `xgb.train`, check out the documentation:\n", + "\n", + "\n", + "https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.train" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "# model training settings\n", + "evallist = [(dvalidation, 'validation'), (dtrain, 'train')]\n", + "num_round = 10" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "ExecuteTime": { + "end_time": "2018-11-06T21:04:50.201308Z", + "start_time": "2018-11-06T21:04:00.363740Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[0]\tvalidation-auc:0.504014\ttrain-auc:0.542211\n", + "[1]\tvalidation-auc:0.506166\ttrain-auc:0.559262\n", + "[2]\tvalidation-auc:0.501638\ttrain-auc:0.570375\n", + "[3]\tvalidation-auc:0.50275\ttrain-auc:0.580726\n", + "[4]\tvalidation-auc:0.503445\ttrain-auc:0.589701\n", + "[5]\tvalidation-auc:0.503413\ttrain-auc:0.598342\n", + "[6]\tvalidation-auc:0.504258\ttrain-auc:0.605253\n", + "[7]\tvalidation-auc:0.503157\ttrain-auc:0.611937\n", + "[8]\tvalidation-auc:0.502372\ttrain-auc:0.617561\n", + "[9]\tvalidation-auc:0.501949\ttrain-auc:0.62333\n", + "CPU times: user 1.12 s, sys: 195 ms, total: 1.31 s\n", + "Wall time: 360 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "\n", + "bst = xgb.train(params, dtrain, num_round, evallist)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion\n", + "\n", + "To learn more about RAPIDS, be sure to check out: \n", + "\n", + "* [Open Source Website](http://rapids.ai)\n", + "* [GitHub](https://github.com/rapidsai/)\n", + "* [Press Release](https://nvidianews.nvidia.com/news/nvidia-introduces-rapids-open-source-gpu-acceleration-platform-for-large-scale-data-analytics-and-machine-learning)\n", + "* [NVIDIA Blog](https://blogs.nvidia.com/blog/2018/10/10/rapids-data-science-open-source-community/)\n", + "* [Developer Blog](https://devblogs.nvidia.com/gpu-accelerated-analytics-rapids/)\n", + "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}