Merge branch 'master' into gold/2021

Author: PokhodenkoSA

CHANGELOG.md

@@ -1,17 +1,40 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+
 ## [Unreleased]
+
+## [0.5.0] - 2020-12-17
 ### Added
+- `_Memory.get_pointer_type` static method which returns kind of USM pointer.
+- Utility functions to transform string to device type and back.
+- New `dpctl.dptensor.numpy_usm_shared` module containing USM array. USM array
+extends NumPy ndarray.
+- A lot of new examples. Including examples of building Cython extensions with DPC++ compiler that interoperate with dpCtl.
+- Mechanism for registering a callback function to look and see if the object
+supports USM.
+
+### Changed
 - setup.py builds C++ backend for develop and install commands.
+- Building wheels.
+- Use DPC++ runtime from package `dpcpp_cpp_rt`.
+- All usage of `DPPL` in C-API functions was changed to `DPCTL`, _e.g._, `DPPLQueueMgr_GetCurrentQueue` to `DPCTLQueueMgr_GetCurrentQueue`.
+- Renamed the C-API directory is now called `dpctl-capi` instead of `backends`.
+- Refactoring the `dpctl-capi` functions to prepare for changes to add Level Zero program creation.
+- `SyclProgram` and `SyclKernel` classes were moved out of `dpctl` into the `dpctl.program` sub-module.
+
+### Fixed
+- Klockwork static code analysis warnings.
 
 ## [0.4.0] - 2020-11-04
 ### Added
 - Device descriptors "max_compute_units", "max_work_item_dimensions", "max_work_item_sizes", "max_work_group_size", "max_num_sub_groups" and "aspects" for int64 atomics inside dpctl C API and inside the dpctl.SyclDevice class.
 - MemoryUSM* classes moved to `dpctl.memory` module, added support for aligned allocation, added support for `prefetch` and `mem_advise` (sychronous) methods, implemented `copy_to_host`, `copy_from_host` and `copy_from_device` methods, pickling support, and zero-copy interoperability with Python objects which implement `__sycl_usm_array_inerface__` protocol.
 - Helper scripts to generate API documentation for both C API and Python.
 
-
 ### Fixed
 - Compiler warnings when building libDPPLSyclInterface and the Cython extensions.
 

README.md

@@ -42,6 +42,18 @@ conda install dpctl
 Build and Install with setuptools
 =================================
 dpCtl relies on DPC++ runtime. With Intel oneAPI installed you should activate it.
+`setup.py` requires environment variable `ONEAPI_ROOT` and following packages
+installed:
+- `cython`
+- `numpy`
+- `cmake` - for building C API
+- `ninja` - only on Windows
+
+Activate DPC++ compiler:
+```bash
+export ONEAPI_ROOT=/opt/intel/oneapi
+source ${ONEAPI_ROOT}/compiler/latest/env/vars.sh
+```
 
 For install:
 ```cmd
@@ -81,10 +93,11 @@ Run python examples:
 for script in `ls examples/python/`; do echo "executing ${script}"; python examples/python/${script}; done
 ```
 
-Examples of building Cython extensions with DPC++ compiler, that interoperate with dpCtl can be found in 
-folder `cython`.
+Examples of building Cython extensions with DPC++ compiler, that interoperate
+with dpCtl can be found in folder `cython`.
 
-Each example in `cython` folder can be built using `CC=clang CXX=dpcpp python setup.py build_ext --inplace`.
+Each example in `cython` folder can be built using
+`CC=clang CXX=dpcpp python setup.py build_ext --inplace`.
 Please refer to `run.py` script in respective folders to execute extensions.
 
 Tests

docs/CMakeLists.txt

@@ -23,6 +23,16 @@ if (GIT_FOUND)
         OUTPUT_VARIABLE CURRENT_RELEASE
         OUTPUT_STRIP_TRAILING_WHITESPACE
     )
+    set(CURRENT_COMMIT "")
+    execute_process(
+        COMMAND ${GIT_EXECUTABLE} describe --tags
+        RESULT_VARIABLE result
+        OUTPUT_VARIABLE CURRENT_COMMIT
+        OUTPUT_STRIP_TRAILING_WHITESPACE
+    )
+    if (NOT "${CURRENT_RELEASE}" STREQUAL "${CURRENT_COMMIT}")
+        set(CURRENT_RELEASE "master")
+    endif ()
 endif (GIT_FOUND)
 
 set(DOXYGEN_INPUT_DIR ../dpctl-capi)

docs/README.md

@@ -3,11 +3,13 @@ What?
 
 Generator scripts for dpCtl API documentation. To run these scripts, follow
 the following steps:
-
-`mkdir build`
-`cd build`
-`cmake -DDPCTL_DOCGEN_PREFIX=<WHERE_YOU_WANT_DOCS_TO_BE_GENERATED>`
-`make Sphinx`
+```bash
+cd dpctl/docs
+mkdir build
+cd build
+cmake -DDPCTL_DOCGEN_PREFIX=<WHERE_YOU_WANT_DOCS_TO_BE_GENERATED> ..
+make Sphinx
+```
 
 The `DPCTL_DOCGEN_PREFIX` flag is optional and can be omitted to generate the
 documents in the current source directory in a sub-directory called
@@ -25,4 +27,4 @@ packages installed:
 - sphinx
 - doxygen
 - breathe
-- exhale
+- exhale

docs/dpCtl.dptensor_api.rst

@@ -0,0 +1,8 @@
+.. _dpCtl.dptensor_api:
+
+#########################
+dpCtl dptensor Python API
+#########################
+
+.. automodule:: dpctl.dptensor
+    :members:

docs/dpCtl.program_api.rst

@@ -0,0 +1,8 @@
+.. _dpCtl.program_api:
+
+########################
+dpCtl Program Python API
+########################
+
+.. automodule:: dpctl.program
+    :members:

docs/index.rst

@@ -19,10 +19,9 @@ Indices and tables
 * :ref:`search`
 
 .. toctree::
-    :maxdepth: 2
+    :maxdepth: 3
     :caption: Contents:
 
     self
-    dpCtl Python API <dpCtl_api>
-    dpctl.memory Python API <dpCtl.memory_api>
+    toc_pyapi
     api/dpCtl-CAPI_root

docs/toc_pyapi.rst

@@ -0,0 +1,10 @@
+Python API
+================
+
+.. toctree::
+    :maxdepth: 1
+
+    dpctl - SYCL runtime wrapper classes and queue manager <dpCtl_api>
+    dpctl.memory - USM memory manager <dpCtl.memory_api>
+    dpctl.dptensor - Data-parallel tensor containers <dpCtl.dptensor_api>
+    dpctl.program - Program manager <dpCtl.program_api>

dpctl/__init__.py

@@ -1,5 +1,4 @@
-# ===---------------- __init__.py - dpctl module -------*- Cython -*--------===#
-#
+# ===-----------------------------------------------------------------------===#
 #                      Data Parallel Control (dpCtl)
 #
 # Copyright 2020 Intel Corporation
@@ -17,11 +16,7 @@
 # limitations under the License.
 #
 # ===-----------------------------------------------------------------------===#
-#
-# \file
-# The top-level dpctl module.
-#
-# ===-----------------------------------------------------------------------===#
+
 """
     **Data Parallel Control (dpCtl)**
 
@@ -31,8 +26,14 @@
     a common runtime to manage specific SYCL resources, such as devices
     and USM memory, for SYCL-based Python packages and extension modules.
 
-    Currently, dpCtl has two main features: a global SYCL queue manager
-    and a USM memory manager.
+    The main features presently provided by dpCtl are:
+
+    * A SYCL queue manager exposed directly inside the top-level `dpctl`
+      module.
+    * A USM memory manager (`dpctl.memory`) that provides Python objects
+      implementing the Python buffer protocol using USM shared and USM host
+      allocators. The memory manager also exposes various utility functions
+      to wrap SYCL's USM allocators, deallocators, `memcpy` functions, *etc.*
 """
 __author__ = "Intel Corp."
 

dpctl/_sycl_core.pyx

@@ -303,7 +303,7 @@ cdef class SyclQueue:
     """
 
     @staticmethod
-    cdef SyclQueue _create (DPCTLSyclQueueRef qref):
+    cdef SyclQueue _create(DPCTLSyclQueueRef qref):
         if qref is NULL:
             raise SyclQueueCreationError("Queue creation failed.")
         cdef SyclQueue ret = SyclQueue.__new__(SyclQueue)
@@ -605,7 +605,7 @@ cdef class _SyclRTManager:
     cdef dict _backend_enum_ty_dict
     cdef dict _device_enum_ty_dict
 
-    def __cinit__ (self):
+    def __cinit__(self):
 
         self._backend_str_ty_dict = {
             "opencl" : _backend_type._OPENCL,
@@ -627,7 +627,7 @@ cdef class _SyclRTManager:
             device_type.gpu : _device_type._GPU,
         }
 
-    def _set_as_current_queue (self, backend_ty, device_ty, device_id):
+    def _set_as_current_queue(self, backend_ty, device_ty, device_id):
         cdef DPCTLSyclQueueRef queue_ref
 
         try :
@@ -642,45 +642,47 @@ cdef class _SyclRTManager:
             raise UnsupportedBackendError("Backend can only be opencl or "
                                           "level-0")
 
-    def _remove_current_queue (self):
+    def _remove_current_queue(self):
         DPCTLQueueMgr_PopQueue()
 
-    def dump (self):
+    def dump(self):
         """ Prints information about the Runtime object.
         """
         DPCTLPlatform_DumpInfo()
 
-    def print_available_backends (self):
-        """ Prints the available backends.
+    def print_available_backends(self):
+        """ Prints the available SYCL backends.
         """
         print(self._backend_str_ty_dict.keys())
 
-    cpdef get_current_backend (self):
-        """ Returns the backend for the current queue as `backend_type` enum
+    cpdef get_current_backend(self):
+        """ Returns the backend for the current queue as a `backend_type` enum
         """
         return self.get_current_queue().get_sycl_backend()
 
-    cpdef get_current_device_type (self):
-        """ Returns current device type as `device_type` enum
+    cpdef get_current_device_type(self):
+        """ Returns current device type as a `device_type` enum
         """
         return self.get_current_queue().get_sycl_device().get_device_type()
 
-    cpdef SyclQueue get_current_queue (self):
-        """ Returns the activated SYCL queue as a PyCapsule.
+    cpdef SyclQueue get_current_queue(self):
+        """ Returns the currently activate SYCL queue as a new SyclQueue object.
+        If there are no active queues then a SyclQueueCreationError exception is
+        raised.
         """
         return SyclQueue._create(DPCTLQueueMgr_GetCurrentQueue())
 
-    def get_num_activated_queues (self):
-        """ Return the number of currently activated queues for this thread.
+    def get_num_activated_queues(self):
+        """ Returns the number of currently activated queues for this thread.
         """
         return DPCTLQueueMgr_GetNumActivatedQueues()
 
-    def get_num_platforms (self):
+    def get_num_platforms(self):
         """ Returns the number of available non-host SYCL platforms.
         """
         return DPCTLPlatform_GetNumNonHostPlatforms()
 
-    def get_num_queues (self, backend_ty, device_ty):
+    def get_num_queues(self, backend_ty, device_ty):
         cdef size_t num = 0
         try :
             beTy = self._backend_enum_ty_dict[backend_ty]
@@ -699,7 +701,7 @@ cdef class _SyclRTManager:
 
         return num
 
-    def has_gpu_queues (self, backend_ty=backend_type.opencl):
+    def has_gpu_queues(self, backend_ty=backend_type.opencl):
         cdef size_t num = 0
         try :
             beTy = self._backend_enum_ty_dict[backend_ty]
@@ -714,7 +716,7 @@ cdef class _SyclRTManager:
         else:
             return False
 
-    def has_cpu_queues (self, backend_ty=backend_type.opencl):
+    def has_cpu_queues(self, backend_ty=backend_type.opencl):
         cdef size_t num = 0
         try :
             beTy = self._backend_enum_ty_dict[backend_ty]
@@ -729,21 +731,21 @@ cdef class _SyclRTManager:
         else:
             return False
 
-    def has_sycl_platforms (self):
+    def has_sycl_platforms(self):
         cdef size_t num_platforms = DPCTLPlatform_GetNumNonHostPlatforms()
         if num_platforms:
             return True
         else:
             return False
 
-    def is_in_device_context (self):
+    def is_in_device_context(self):
         cdef size_t num = DPCTLQueueMgr_GetNumActivatedQueues()
         if num:
             return True
         else:
             return False
 
-    def set_default_queue (self, backend_ty, device_ty, device_id):
+    def set_default_queue(self, backend_ty, device_ty, device_id):
         cdef DPCTLSyclQueueRef ret
         try :
             if isinstance(backend_ty, str):
@@ -785,8 +787,17 @@ set_default_queue        = _mgr.set_default_queue
 is_in_device_context     = _mgr.is_in_device_context
 
 cpdef SyclQueue get_current_queue():
-    """
-        Obtain current Sycl Queue from Data Parallel Control package.
+    """ Returns the currently activate SYCL queue as a new SyclQueue object.
+
+    Returns:
+        SyclQueue: If there is a currently active SYCL queue that queue
+        is returned wrapped in a SyclQueue object. The SyclQueue object
+        owns a copy of the currently active SYCL queue as an opaque
+        `DPCTLSyclQueueRef` pointer. The pointer is freed when the SyclQueue
+        is garbage collected.
+
+    Raises:
+        SyclQueueCreationError: If no currently active SYCL queue found.
     """
     return _mgr.get_current_queue()
 
@@ -805,7 +816,7 @@ cpdef get_current_backend():
 from contextlib import contextmanager
 
 @contextmanager
-def device_context (str queue_str="opencl:gpu:0"):
+def device_context(str queue_str="opencl:gpu:0"):
     """
         The SYCL queue defined by the "backend:device type:device id" tuple is
         set as the currently active queue, *i.e.*, a subsequent call to

dpctl/dptensor/__init__.py

@@ -1 +1,32 @@
+# ===-----------------------------------------------------------------------===#
+#                      Data Parallel Control (dpCtl)
+#
+# Copyright 2020 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ===-----------------------------------------------------------------------===#
+"""
+    **Data Parallel Tensor Collection**
+
+    `dpctl.dptensor` is an experimental collection of tensor implementations
+    that will implement future Python data API (https://data-apis.github.io/array-api/latest/).
+
+    Available tensor implementations:
+
+    * `numpy_usm_shared`: Provides a `numpy.ndarray` sub-class whose \
+    underlying memory buffer is allocated with a USM shared memory allocator.
+
+"""
+
 import dpctl.dptensor.numpy_usm_shared