examples/usage-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "97347d32",
   "metadata": {},
   "source": [
    "# Disk Health Predictor Usage Demo\n",
    "\n",
    "In this notebook, we will show how you can get started with the [`disk-health-predictor`](https://github.com/aicoe-aiops/disk-health-predictor) python module with just a few lines of code! First we will install it as a package, and then we will pass to it a [`smartctl`](https://linux.die.net/man/8/smartctl) json collected from a real hard disk, to predict that hard disk's health and estimate its time-to-failure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9447c7c6",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "To install Disk Health Predictor using pip, you can run the following command:\n",
    "```bash\n",
    "pip install git+https://github.com/aicoe-aiops/disk-health-predictor.git\n",
    "```\n",
    "Alternatively, you can also add it to your Pipfile like this:\n",
    "```\n",
    "[[source]]\n",
    "name = \"pypi\"\n",
    "url = \"https://pypi.org/simple\"\n",
    "verify_ssl = true\n",
    "\n",
    "[dev-packages]\n",
    "some-packege = \"*\"\n",
    "\n",
    "[packages]\n",
    "disk-health-predictor = {git = \"https://github.com/aicoe-aiops/disk-health-predictor.git\", ref=\"master\"}\n",
    "another-pkg = \"*\"\n",
    "\n",
    "[requires]\n",
    "python_version = \"3.8\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e042aa0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets install using the command line for now\n",
    "# !pip install git+https://github.com/aicoe-aiops/disk-health-predictor.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cf30968",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Usage\n",
    "\n",
    "To collect your disk's SMART data using the `smartctl` tool, you can run the following command (replacing `/dev/nvme0` with the device you want)\n",
    "```bash\n",
    "smartctl --json --xall /dev/nvme0\n",
    "```\n",
    "\n",
    "To collect and share (anonymously) this data for the hard disks in your Ceph cluster, you can enable the [Telemetry](https://docs.ceph.com/en/latest/mgr/telemetry/) module. We will use one of the smartctl jsons collected via Ceph telemetry as sample input data in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "820c0432",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5546decf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keys in the input JSON:\n",
      "dict_keys(['2020-04-01 00:08:56', '2020-04-02 00:00:59', '2020-04-02 09:38:43', '2020-04-03 00:03:17', '2020-04-04 00:05:22', '2020-04-05 00:07:31', '2020-04-06 00:09:38', '2020-04-07 00:00:20'])\n",
      "\n",
      "Value for Key 2020-04-01 00:08:56 in the input JSON:\n",
      "{'ata_sct_capabilities': {'data_table_supported': True,\n",
      "                          'error_recovery_control_supported': True,\n",
      "                          'feature_control_supported': True,\n",
      "                          'value': 28861},\n",
      " 'ata_smart_attributes': {'revision': 10,\n",
      "                          'table': [{'flags': {...},\n",
      "                                     'id': 1,\n",
      "                                     'name': 'Raw_Read_Error_Rate',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 44,\n",
      "                                     'value': 80,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 64},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 3,\n",
      "                                     'name': 'Spin_Up_Time',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 93,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 92},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 4,\n",
      "                                     'name': 'Start_Stop_Count',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 20,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 5,\n",
      "                                     'name': 'Reallocated_Sector_Ct',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 10,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 7,\n",
      "                                     'name': 'Seek_Error_Rate',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 45,\n",
      "                                     'value': 93,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 60},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 9,\n",
      "                                     'name': 'Power_On_Hours',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 90,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 90},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 10,\n",
      "                                     'name': 'Spin_Retry_Count',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 97,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 12,\n",
      "                                     'name': 'Power_Cycle_Count',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 20,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 184,\n",
      "                                     'name': 'End-to-End_Error',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 99,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 187,\n",
      "                                     'name': 'Reported_Uncorrect',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 188,\n",
      "                                     'name': 'Command_Timeout',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 189,\n",
      "                                     'name': 'High_Fly_Writes',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 190,\n",
      "                                     'name': 'Airflow_Temperature_Cel',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 40,\n",
      "                                     'value': 67,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 60},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 191,\n",
      "                                     'name': 'G-Sense_Error_Rate',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 99,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 99},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 192,\n",
      "                                     'name': 'Power-Off_Retract_Count',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 193,\n",
      "                                     'name': 'Load_Cycle_Count',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 81,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 81},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 194,\n",
      "                                     'name': 'Temperature_Celsius',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 33,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 40},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 195,\n",
      "                                     'name': 'Hardware_ECC_Recovered',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 9,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 2},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 197,\n",
      "                                     'name': 'Current_Pending_Sector',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 198,\n",
      "                                     'name': 'Offline_Uncorrectable',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 100},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 199,\n",
      "                                     'name': 'UDMA_CRC_Error_Count',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 200,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 200},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 240,\n",
      "                                     'name': 'Head_Flying_Hours',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 253},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 241,\n",
      "                                     'name': 'Total_LBAs_Written',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 253},\n",
      "                                    {'flags': {...},\n",
      "                                     'id': 242,\n",
      "                                     'name': 'Total_LBAs_Read',\n",
      "                                     'raw': {...},\n",
      "                                     'thresh': 0,\n",
      "                                     'value': 100,\n",
      "                                     'when_failed': '',\n",
      "                                     'worst': 253}]},\n",
      " 'ata_smart_data': {'capabilities': {'attribute_autosave_enabled': True,\n",
      "                                     'conveyance_self_test_supported': True,\n",
      "                                     'error_logging_supported': True,\n",
      "                                     'exec_offline_immediate_supported': True,\n",
      "                                     'gp_logging_supported': True,\n",
      "                                     'offline_is_aborted_upon_new_cmd': False,\n",
      "                                     'offline_surface_scan_supported': True,\n",
      "                                     'selective_self_test_supported': True,\n",
      "                                     'self_tests_supported': True,\n",
      "                                     'values': [123, 3]},\n",
      "                    'offline_data_collection': {'completion_seconds': 567,\n",
      "                                                'status': {'passed': True,\n",
      "                                                           'string': 'was '\n",
      "                                                                     'completed '\n",
      "                                                                     'without '\n",
      "                                                                     'error',\n",
      "                                                           'value': 130}},\n",
      "                    'self_test': {'polling_minutes': {'conveyance': 2,\n",
      "                                                      'extended': 768,\n",
      "                                                      'short': 1},\n",
      "                                  'status': {'passed': True,\n",
      "                                             'string': 'completed without '\n",
      "                                                       'error',\n",
      "                                             'value': 0}}},\n",
      " 'ata_smart_error_log': {'summary': {'count': 0, 'revision': 1}},\n",
      " 'ata_smart_selective_self_test_log': {'flags': {'remainder_scan_enabled': False,\n",
      "                                                 'value': 0},\n",
      "                                       'power_up_scan_resume_minutes': 0,\n",
      "                                       'revision': 1,\n",
      "                                       'table': [{'lba_max': 0,\n",
      "                                                  'lba_min': 0,\n",
      "                                                  'status': {...}},\n",
      "                                                 {'lba_max': 0,\n",
      "                                                  'lba_min': 0,\n",
      "                                                  'status': {...}},\n",
      "                                                 {'lba_max': 0,\n",
      "                                                  'lba_min': 0,\n",
      "                                                  'status': {...}},\n",
      "                                                 {'lba_max': 0,\n",
      "                                                  'lba_min': 0,\n",
      "                                                  'status': {...}},\n",
      "                                                 {'lba_max': 0,\n",
      "                                                  'lba_min': 0,\n",
      "                                                  'status': {...}}]},\n",
      " 'ata_smart_self_test_log': {'standard': {'count': 0, 'revision': 1}},\n",
      " 'ata_version': {'major_value': 2032,\n",
      "                 'minor_value': 109,\n",
      "                 'string': 'ACS-3 T13/2161-D revision 5'},\n",
      " 'device': {'info_name': '/dev/sdb [SAT]',\n",
      "            'name': '/dev/sdb',\n",
      "            'protocol': 'ATA',\n",
      "            'type': 'sat'},\n",
      " 'firmware_version': 'NN04',\n",
      " 'form_factor': {'ata_value': 2, 'name': '3.5 inches'},\n",
      " 'host_id': '5c5dce96-73fe-11ea-a3db-02010001002d',\n",
      " 'in_smartctl_database': True,\n",
      " 'interface_speed': {'current': {'bits_per_unit': 100000000,\n",
      "                                 'sata_value': 3,\n",
      "                                 'string': '6.0 Gb/s',\n",
      "                                 'units_per_second': 60},\n",
      "                     'max': {'bits_per_unit': 100000000,\n",
      "                             'sata_value': 14,\n",
      "                             'string': '6.0 Gb/s',\n",
      "                             'units_per_second': 60}},\n",
      " 'json_format_version': [1, 0],\n",
      " 'local_time': {'asctime': 'Wed Apr  1 00:08:56 2020 UTC',\n",
      "                'time_t': 1585699736},\n",
      " 'logical_block_size': 4096,\n",
      " 'model_family': 'Seagate Enterprise Capacity 3.5 HDD',\n",
      " 'model_name': 'ST8000NM0045-1RL112',\n",
      " 'nvme_vendor': 'lvm',\n",
      " 'physical_block_size': 4096,\n",
      " 'power_cycle_count': 29,\n",
      " 'power_on_time': {'hours': 9416},\n",
      " 'rotation_rate': 7200,\n",
      " 'sata_version': {'string': 'SATA 3.1', 'value': 127},\n",
      " 'smart_status': {'passed': True},\n",
      " 'smartctl': {'argv': ['smartctl', '-a', '--json=o', '/dev/sdb'],\n",
      "              'build_info': '(local build)',\n",
      "              'exit_status': 0,\n",
      "              'output': ['smartctl 7.1 2019-12-30 r5022 '\n",
      "                         '[x86_64-linux-5.4.0-0.bpo.4-amd64] (local build)',\n",
      "                         'Copyright (C) 2002-19, Bruce Allen, Christian '\n",
      "                         'Franke, www.smartmontools.org',\n",
      "                         '',\n",
      "                         '=== START OF INFORMATION SECTION ===',\n",
      "                         'Model Family:     Seagate Enterprise Capacity 3.5 '\n",
      "                         'HDD',\n",
      "                         'Device Model:     ST8000NM0045-1RL112',\n",
      "                         'Firmware Version: NN04',\n",
      "                         'User Capacity:    8,001,563,222,016 bytes [8.00 TB]',\n",
      "                         'Sector Size:      4096 bytes logical/physical',\n",
      "                         'Rotation Rate:    7200 rpm',\n",
      "                         'Form Factor:      3.5 inches',\n",
      "                         'Device is:        In smartctl database [for details '\n",
      "                         'use: -P show]',\n",
      "                         'ATA Version is:   ACS-3 T13/2161-D revision 5',\n",
      "                         'SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 '\n",
      "                         'Gb/s)',\n",
      "                         'Local Time is:    Wed Apr  1 00:08:56 2020 UTC',\n",
      "                         'SMART support is: Available - device has SMART '\n",
      "                         'capability.',\n",
      "                         'SMART support is: Enabled',\n",
      "                         '',\n",
      "                         '=== START OF READ SMART DATA SECTION ===',\n",
      "                         'SMART overall-health self-assessment test result: '\n",
      "                         'PASSED',\n",
      "                         '',\n",
      "                         'General SMART Values:',\n",
      "                         'Offline data collection status:  (0x82)\\tOffline '\n",
      "                         'data collection activity',\n",
      "                         '\\t\\t\\t\\t\\twas completed without error.',\n",
      "                         '\\t\\t\\t\\t\\tAuto Offline Data Collection: Enabled.',\n",
      "                         'Self-test execution status:      (   0)\\tThe '\n",
      "                         'previous self-test routine completed',\n",
      "                         '\\t\\t\\t\\t\\twithout error or no self-test has ever ',\n",
      "                         '\\t\\t\\t\\t\\tbeen run.',\n",
      "                         'Total time to complete Offline ',\n",
      "                         'data collection: \\t\\t(  567) seconds.',\n",
      "                         'Offline data collection',\n",
      "                         'capabilities: \\t\\t\\t (0x7b) SMART execute Offline '\n",
      "                         'immediate.',\n",
      "                         '\\t\\t\\t\\t\\tAuto Offline data collection on/off '\n",
      "                         'support.',\n",
      "                         '\\t\\t\\t\\t\\tSuspend Offline collection upon new',\n",
      "                         '\\t\\t\\t\\t\\tcommand.',\n",
      "                         '\\t\\t\\t\\t\\tOffline surface scan supported.',\n",
      "                         '\\t\\t\\t\\t\\tSelf-test supported.',\n",
      "                         '\\t\\t\\t\\t\\tConveyance Self-test supported.',\n",
      "                         '\\t\\t\\t\\t\\tSelective Self-test supported.',\n",
      "                         'SMART capabilities:            (0x0003)\\tSaves SMART '\n",
      "                         'data before entering',\n",
      "                         '\\t\\t\\t\\t\\tpower-saving mode.',\n",
      "                         '\\t\\t\\t\\t\\tSupports SMART auto save timer.',\n",
      "                         'Error logging capability:        (0x01)\\tError '\n",
      "                         'logging supported.',\n",
      "                         '\\t\\t\\t\\t\\tGeneral Purpose Logging supported.',\n",
      "                         'Short self-test routine ',\n",
      "                         'recommended polling time: \\t (   1) minutes.',\n",
      "                         'Extended self-test routine',\n",
      "                         'recommended polling time: \\t ( 768) minutes.',\n",
      "                         'Conveyance self-test routine',\n",
      "                         'recommended polling time: \\t (   2) minutes.',\n",
      "                         'SCT capabilities: \\t       (0x70bd)\\tSCT Status '\n",
      "                         'supported.',\n",
      "                         '\\t\\t\\t\\t\\tSCT Error Recovery Control supported.',\n",
      "                         '\\t\\t\\t\\t\\tSCT Feature Control supported.',\n",
      "                         '\\t\\t\\t\\t\\tSCT Data Table supported.',\n",
      "                         '',\n",
      "                         'SMART Attributes Data Structure revision number: 10',\n",
      "                         'Vendor Specific SMART Attributes with Thresholds:',\n",
      "                         'ID# ATTRIBUTE_NAME          FLAG     VALUE WORST '\n",
      "                         'THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE',\n",
      "                         '  1 Raw_Read_Error_Rate     0x000f   080   064   '\n",
      "                         '044    Pre-fail  Always       -       13618832',\n",
      "                         '  3 Spin_Up_Time            0x0003   093   092   '\n",
      "                         '000    Pre-fail  Always       -       0',\n",
      "                         '  4 Start_Stop_Count        0x0032   100   100   '\n",
      "                         '020    Old_age   Always       -       29',\n",
      "                         '  5 Reallocated_Sector_Ct   0x0033   100   100   '\n",
      "                         '010    Pre-fail  Always       -       0',\n",
      "                         '  7 Seek_Error_Rate         0x000f   093   060   '\n",
      "                         '045    Pre-fail  Always       -       1872145368',\n",
      "                         '  9 Power_On_Hours          0x0032   090   090   '\n",
      "                         '000    Old_age   Always       -       9416',\n",
      "                         ' 10 Spin_Retry_Count        0x0013   100   100   '\n",
      "                         '097    Pre-fail  Always       -       0',\n",
      "                         ' 12 Power_Cycle_Count       0x0032   100   100   '\n",
      "                         '020    Old_age   Always       -       29',\n",
      "                         '184 End-to-End_Error        0x0032   100   100   '\n",
      "                         '099    Old_age   Always       -       0',\n",
      "                         '187 Reported_Uncorrect      0x0032   100   100   '\n",
      "                         '000    Old_age   Always       -       0',\n",
      "                         '188 Command_Timeout         0x0032   100   100   '\n",
      "                         '000    Old_age   Always       -       0 0 0',\n",
      "                         '189 High_Fly_Writes         0x003a   100   100   '\n",
      "                         '000    Old_age   Always       -       0',\n",
      "                         '190 Airflow_Temperature_Cel 0x0022   067   060   '\n",
      "                         '040    Old_age   Always       -       33 (Min/Max '\n",
      "                         '32/34)',\n",
      "                         '191 G-Sense_Error_Rate      0x0032   099   099   '\n",
      "                         '000    Old_age   Always       -       2884',\n",
      "                         '192 Power-Off_Retract_Count 0x0032   100   100   '\n",
      "                         '000    Old_age   Always       -       129',\n",
      "                         '193 Load_Cycle_Count        0x0032   081   081   '\n",
      "                         '000    Old_age   Always       -       39438',\n",
      "                         '194 Temperature_Celsius     0x0022   033   040   '\n",
      "                         '000    Old_age   Always       -       33 (0 18 0 0 '\n",
      "                         '0)',\n",
      "                         '195 Hardware_ECC_Recovered  0x001a   009   002   '\n",
      "                         '000    Old_age   Always       -       13618832',\n",
      "                         '197 Current_Pending_Sector  0x0012   100   100   '\n",
      "                         '000    Old_age   Always       -       0',\n",
      "                         '198 Offline_Uncorrectable   0x0010   100   100   '\n",
      "                         '000    Old_age   Offline      -       0',\n",
      "                         '199 UDMA_CRC_Error_Count    0x003e   200   200   '\n",
      "                         '000    Old_age   Always       -       0',\n",
      "                         '240 Head_Flying_Hours       0x0000   100   253   '\n",
      "                         '000    Old_age   Offline      -       '\n",
      "                         '7019h+59m+24.383s',\n",
      "                         '241 Total_LBAs_Written      0x0000   100   253   '\n",
      "                         '000    Old_age   Offline      -       299321746',\n",
      "                         '242 Total_LBAs_Read         0x0000   100   253   '\n",
      "                         '000    Old_age   Offline      -       263622126',\n",
      "                         '',\n",
      "                         'SMART Error Log Version: 1',\n",
      "                         'No Errors Logged',\n",
      "                         '',\n",
      "                         'SMART Self-test log structure revision number 1',\n",
      "                         'No self-tests have been logged.  [To run self-tests, '\n",
      "                         'use: smartctl -t]',\n",
      "                         '',\n",
      "                         'SMART Selective self-test log data structure '\n",
      "                         'revision number 1',\n",
      "                         ' SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS',\n",
      "                         '    1        0        0  Not_testing',\n",
      "                         '    2        0        0  Not_testing',\n",
      "                         '    3        0        0  Not_testing',\n",
      "                         '    4        0        0  Not_testing',\n",
      "                         '    5        0        0  Not_testing',\n",
      "                         'Selective self-test flags (0x0):',\n",
      "                         '  After scanning selected spans, do NOT read-scan '\n",
      "                         'remainder of disk.',\n",
      "                         'If Selective self-test is pending on power-up, '\n",
      "                         'resume after 0 minute delay.',\n",
      "                         ''],\n",
      "              'platform_info': 'x86_64-linux-5.4.0-0.bpo.4-amd64',\n",
      "              'svn_revision': '5022',\n",
      "              'version': [7, 1]},\n",
      " 'temperature': {'current': 33},\n",
      " 'user_capacity': {'blocks': 1953506646, 'bytes': 8001563222016},\n",
      " 'wwn': {'naa': 5, 'oui': 3152}}\n"
     ]
    }
   ],
   "source": [
    "# lets take a look at what the input data looks like\n",
    "with open(\"sample_input_1.json\", \"r\") as f:\n",
    "    sample_smartctl_jsons = json.load(f)\n",
    "\n",
    "# view keys\n",
    "keys = sample_smartctl_jsons.keys()\n",
    "print(\"Keys in the input JSON:\")\n",
    "print(keys, end=\"\\n\\n\")\n",
    "\n",
    "# view value for a key\n",
    "sample_key = list(keys)[0]\n",
    "print(f\"Value for Key {sample_key} in the input JSON:\")\n",
    "pprint(sample_smartctl_jsons[sample_key], depth=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06d58299",
   "metadata": {},
   "source": [
    "As we can see, the keys in the input json are the datetime objects when the `smartctl` output was taken, and values are the raw `smartctl` json output. Note that predictor needs at least 6 days of data to make a valid prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb2ec587",
   "metadata": {},
   "source": [
    "Disk Health Predictor provides several `Classifier`s, such `PSDiskHealthClassifier` and `RHDiskHealthClassifier`, to estimate disk health using `smartctl` data. These classifiers make use of some pretrained models associated with them to predict a disk's health as either\n",
    "- `BAD` (will fail in less than 2 weeks),\n",
    "- `WARNING` (will fail in 2 to 6 weeks),\n",
    "- `GOOD` (will not fail for at least 6 weeks), or\n",
    "- `UNKNOWN` (could not generate a prediction due to insufficient data).\n",
    "\n",
    "To instantiate these classifiers, you can use the factory function `DiskHealthClassifierFactory` as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8893385a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from disk_health_predictor.classifiers import DiskHealthClassifierFactory  # noqa: E402"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "488c6679",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<disk_health_predictor.classifiers._redhat.RHDiskHealthClassifier at 0x7fb3740afa30>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = DiskHealthClassifierFactory(\"redhat\")\n",
    "clf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62a77459",
   "metadata": {},
   "source": [
    "Now, you can classify disk health to estimate when the disk will fail as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fcc79bc9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This hard disk is estimated to be in the 'Good' health category.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/base.py:301: UserWarning: Trying to unpickle estimator RobustScaler from version 0.19.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:30: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  method='lar', copy_X=True, eps=np.finfo(np.float).eps,\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:167: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  method='lar', copy_X=True, eps=np.finfo(np.float).eps,\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:284: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  eps=np.finfo(np.float).eps, copy_Gram=True, verbose=0,\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:862: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:1101: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:1127: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  eps=np.finfo(np.float).eps, positive=False):\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:1362: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:1602: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/linear_model/least_angle.py:1738: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  eps=np.finfo(np.float).eps, copy_X=True, positive=False):\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/decomposition/online_lda.py:29: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  EPS = np.finfo(np.float).eps\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/ensemble/gradient_boosting.py:32: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  from ._gradient_boosting import predict_stages\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/ensemble/gradient_boosting.py:32: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  from ._gradient_boosting import predict_stages\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/base.py:301: UserWarning: Trying to unpickle estimator DecisionTreeClassifier from version 0.19.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/base.py:301: UserWarning: Trying to unpickle estimator RandomForestClassifier from version 0.19.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n",
      "/home/kachau/.local/share/virtualenvs/demo-dhp-U8zZfpVL/lib/python3.8/site-packages/sklearn/ensemble/base.py:158: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  dtype=np.int)\n"
     ]
    }
   ],
   "source": [
    "pred = clf.predict(sample_smartctl_jsons)\n",
    "print(f\"This hard disk is estimated to be in the '{pred}' health category.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf9f8ef9",
   "metadata": {},
   "source": [
    "If you have any questions, comments, or suggestions, please feel free to [open issues](https://github.com/aicoe-aiops/disk-health-predictor/issues/new/choose) or [submit pull requests](https://github.com/aicoe-aiops/disk-health-predictor/compare)!"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "823a9bb50c2e01d1e21b407185430c5a8374d225b00d9f086b70a5bb43d1f993"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}