This module is part of the ISS MTech Graduate Certificate series. The module is common for both Graduate Certificate in Big Data Analytics as well as Graduate Certificate in Engineering Big Data series offered by NUS-ISS.
Participants will learn various aspects of data engineering while building data intensive applications. Participants will learn to apply key practices, identify multiple data sources appraised against their business value, design the right storage, and implement proper access model(s). Finally, participants will build a scalable data pipeline solution composed of pluggable hadoop component architecture, based on the combination of requirements in a vendor/technology agnostic manner. Participants will work with Spark framework and special processing libraries.
Upon effective completion of the course, participants will be able to: Understand the growth of big data and need for a scalable processing framework. Understand the fundamental characteristics, storage, analysis techniques and the relevant distributions Understand the distributed storage essentials, storage needs, and relevant architectural mechanism in processing large amounts of structured, semi-structured and unstructured data. Gain expertise with the fault-tolerant computing framework (E.g. Hadoop or Kubernetes) by setting up pseudo cluster nodes or cloud based nodes for processing big data. . Construct configurable and executable tasks using the In Memory Processing frameworks (E.g. Spark Core). Understand the nuances of writing functional programs and use the core libraries to manipulate the large corpse of unstructured data residing as Resilient Distributed Datasets. Organize, store and manipulate the collected data using processing libraries. For example, using special statistical operation and stream processing data tools (E.g. Spark Special Libraries). Understand various data processing, querying and persistence (E.g. Spark QL APIs) available for usage in RDD’s context. Perform tasks such as filtering, selection and categorization.