Add doc ci (OpenPPL#112)

* CI(github): add lint

* update

* update

* update

* docs(ppq): update doc and add ci

Author: tpoisonooo

.github/workflows/lint.yml

@@ -0,0 +1,21 @@
+name: lint
+
+on: [push, pull_request]
+
+jobs:
+  lint:
+    runs-on: ubuntu-18.04
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python 3.7
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.7
+      - name: Install pre-commit hook
+        run: |
+          # markdownlint requires ruby >= 2.7
+          sudo apt-add-repository ppa:brightbox/ruby-ng -y
+          sudo apt-get update
+          sudo apt-get install -y ruby2.7
+          pip install pre-commit
+          pre-commit install

.github/workflows/ppq_simulator.yml

@@ -30,4 +30,4 @@ jobs:
 
       - name: Test
         run: |
-          cd ../../ && ./test_ppq.sh ${{ github.run_id }} 
+          cd ../../ && ./test_ppq.sh ${{ github.run_id }}

.pre-commit-config.yaml

@@ -0,0 +1,30 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.1.0
+    hooks:
+      - id: trailing-whitespace
+      - id: check-yaml
+      - id: end-of-file-fixer
+      - id: requirements-txt-fixer
+      - id: double-quote-string-fixer
+      - id: check-merge-conflict
+      - id: fix-encoding-pragma
+        args: ["--remove"]
+      - id: mixed-line-ending
+        args: ["--fix=lf"]
+  - repo: https://github.com/markdownlint/markdownlint
+    rev: v0.11.0
+    hooks:
+      - id: markdownlint
+        args:
+          [
+            "-r",
+            "~MD002,~MD013,~MD029,~MD033,~MD034",
+            "-t",
+            "allow_different_nesting",
+          ]
+  - repo: https://github.com/myint/docformatter
+    rev: v1.4
+    hooks:
+      - id: docformatter
+        args: ["--in-place", "--wrap-descriptions", "79"]

LICENSE

@@ -198,4 +198,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

MANIFEST.in

@@ -1,3 +1,3 @@
 include ppq/parser/caffe/*
 include ppq/csrc/cuda/*
-include ppq/csrc/build/readme.md
+include ppq/csrc/build/readme.md

ProgramEntrance.py

@@ -1,19 +1,18 @@
-"""
-    这是一个 PPQ 量化的入口脚本，将你的模型和数据按要求进行打包:
+"""这是一个 PPQ 量化的入口脚本，将你的模型和数据按要求进行打包:
 
-    This file will show you how to quantize your network with PPQ
-        You should prepare your model and calibration dataset as follow:
+This file will show you how to quantize your network with PPQ
+    You should prepare your model and calibration dataset as follow:
 
-        ~/working/model.onnx                          <--  your model
-        ~/working/data/*.npy or ~/working/data/*.bin  <--  your dataset
+    ~/working/model.onnx                          <--  your model
+    ~/working/data/*.npy or ~/working/data/*.bin  <--  your dataset
 
-    if you are using caffe model:
-        ~/working/model.caffemdoel  <--  your model
-        ~/working/model.prototext   <--  your model
+if you are using caffe model:
+    ~/working/model.caffemdoel  <--  your model
+    ~/working/model.prototext   <--  your model
 
-    ### MAKE SURE YOUR INPUT LAYOUT IS [N, C, H, W] or [C, H, W] ###
+### MAKE SURE YOUR INPUT LAYOUT IS [N, C, H, W] or [C, H, W] ###
 
-    quantized model will be generated at: ~/working/quantized.onnx
+quantized model will be generated at: ~/working/quantized.onnx
 """
 from ppq import *
 from ppq.IR.processor import GraphCommandProcessor

README.md

@@ -10,23 +10,23 @@ PPL QuantTool 是一个工业级的神经网络量化工具：我们已经准备
 
 PPQ 使用量化计算图(QIR)描述量化细节，即便在网络极度复杂的情况下，我们依然能够保证以正确的方法模拟硬件计算，从而降低模拟误差。我们知晓硬件的运算细节——在所有已知平台上，PPQ 的模拟误差不超过1%，且保证模拟误差不会指数级扩散。PPQ 有着自定义的量化算子库、网络执行器、调度器与异构执行能力，在网络量化与量化训练方面，使用 PPQ 比原生 PyTorch 快3 ~ 50倍。 借助 PPQ, OpenPPL, TensorRT, Tengine等框架，开发者可以将神经网络模型加速5 ~ 10倍，并部署到多种多样的目标终端，我们期待你将人工智慧真正带到千家万户之间。
 
-### Acceptable Framework: 
+### Acceptable Framework:
 PyTorch | TensorFlow | Caffe | ONNX | MMlab
 
 ### Deploy Platform:
 TensorRT | OpenPPL-CUDA | OpenPPL-DSP | SNPE(Qualcomm) | NXP | Metax | Tengine(Developing) | Ncnn(Developing)
- | 
+ |
 # Video Tutorial(Bilibili 视频教程)
 Watch video tutorial series on www.bilibili.com, following are links of PPQ tutorial links(Only Chinese version).
 
-* 安装教程: [https://www.bilibili.com/video/BV1WS4y1N7Kn](https://www.bilibili.com/video/BV1WS4y1N7Kn "PPQ Installation Tutorial") 
-* 使用教程: [https://www.bilibili.com/video/BV1934y147p2](https://www.bilibili.com/video/BV1934y147p2 "PPQ User Guide") 
-* 基础知识：[https://www.bilibili.com/video/BV1gS4y1Y7KR](https://www.bilibili.com/video/BV1gS4y1Y7KR "Basic Theory") 
-* 网络性能分析：[https://www.bilibili.com/video/BV1oT4y1h73e](https://www.bilibili.com/video/BV1oT4y1h73e "Analysing Your Model") 
-* 量化计算原理(Part 1)：[https://www.bilibili.com/video/BV1fB4y1m7fJ](https://www.bilibili.com/video/BV1fB4y1m7fJ "Quantized Computing") 
-* 量化计算原理(Part 2)：[https://www.bilibili.com/video/BV1qA4y1Q7Uh](https://www.bilibili.com/video/BV1qA4y1Q7Uh "Quantized Computing") 
-* 图优化与量化模拟：[https://www.bilibili.com/video/BV1Kr4y1n7cy](https://www.bilibili.com/video/BV1Kr4y1n7cy "Graph Optimization & quantization simulating.") 
-* 图调度与模式匹配：[https://www.bilibili.com/video/BV1xY411A7ea](https://www.bilibili.com/video/BV1xY411A7ea "Graph Dispatching & Pattern Matching.") 
+* 安装教程: [https://www.bilibili.com/video/BV1WS4y1N7Kn](https://www.bilibili.com/video/BV1WS4y1N7Kn "PPQ Installation Tutorial")
+* 使用教程: [https://www.bilibili.com/video/BV1934y147p2](https://www.bilibili.com/video/BV1934y147p2 "PPQ User Guide")
+* 基础知识：[https://www.bilibili.com/video/BV1gS4y1Y7KR](https://www.bilibili.com/video/BV1gS4y1Y7KR "Basic Theory")
+* 网络性能分析：[https://www.bilibili.com/video/BV1oT4y1h73e](https://www.bilibili.com/video/BV1oT4y1h73e "Analysing Your Model")
+* 量化计算原理(Part 1)：[https://www.bilibili.com/video/BV1fB4y1m7fJ](https://www.bilibili.com/video/BV1fB4y1m7fJ "Quantized Computing")
+* 量化计算原理(Part 2)：[https://www.bilibili.com/video/BV1qA4y1Q7Uh](https://www.bilibili.com/video/BV1qA4y1Q7Uh "Quantized Computing")
+* 图优化与量化模拟：[https://www.bilibili.com/video/BV1Kr4y1n7cy](https://www.bilibili.com/video/BV1Kr4y1n7cy "Graph Optimization & quantization simulating.")
+* 图调度与模式匹配：[https://www.bilibili.com/video/BV1xY411A7ea](https://www.bilibili.com/video/BV1xY411A7ea "Graph Dispatching & Pattern Matching.")
 
 * 其他教程: 等待缓慢更新...
 
@@ -47,7 +47,7 @@ ATTENTION: Make sure your Python version is >= 3.6.0. PPL Quantization Tool is w
 
     * For Windows User:
         * Download ninja.exe from [https://github.com/ninja-build/ninja/releases](https://github.com/ninja-build/ninja/releases), add it to Windows PATH Environment
-        * Download Visual Studio from [https://visualstudio.microsoft.com](https://visualstudio.microsoft.com/zh-hans/), if you already got a c++ compiler, you can skip this step. 
+        * Download Visual Studio from [https://visualstudio.microsoft.com](https://visualstudio.microsoft.com/zh-hans/), if you already got a c++ compiler, you can skip this step.
         * Please Update Visual studio to 2019, otherwise there might be some compile error like: CxxFrameHandler4 unresolved, GSHandlerCheck unresolved.
         * Add your c++ compiler to Windows PATH Environment, if you are using Visual Studio, it should be something like "C:\Program Files\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.16.27023\bin\Hostx86\x86"
 
@@ -70,7 +70,7 @@ python setup.py install
 
 ```bash
 python3 -m pip install ppq
-``` 
+```
 
 # Tutorials and Running Examples
 

Util.py

@@ -65,7 +65,7 @@ def load_calibration_dataset(directory: str, input_shape: List[int], batchsize:
     for idx, tensor in enumerate(samples[: 5]):
         print(f'Loaded sample {idx}, shape: {tensor.shape}')
     assert len(batches) > 0, '你送入了空的数据集'
-    
+
     print(f'Batch Shape: {batches[0].shape}')
     return batches
 
@@ -116,7 +116,7 @@ def export(working_directory: str, quantized: BaseGraph, platform: TargetPlatfor
 
 
 def load_from_file(
-    filename: str, dtype: np.dtype = np.float32, 
+    filename: str, dtype: np.dtype = np.float32,
     shape: List[int] = None, format: str = '.npy') -> np.ndarray:
     if not os.path.exists(filename) or os.path.isdir(filename):
         raise FileNotFoundError(f'Can not load data from file: {filename}. '
@@ -135,14 +135,14 @@ def load_from_file(
 
 
 def dump_to_file(
-    filename: str, data: Union[torch.Tensor, np.ndarray], 
+    filename: str, data: Union[torch.Tensor, np.ndarray],
     format: str = '.npy') -> None:
     if os.path.isdir(filename):
         raise FileExistsError(f'Can not dump data to file: {filename}',
                               ' Cause it is a directory.')
     if os.path.exists(filename):
         ppq_warning(f'Overwritting file {filename} ...')
-    
+
     data = convert_any_to_numpy(data)
     if format == '.npy':
         np.save(file=filename, arr=data)
@@ -180,24 +180,24 @@ def compare_cosine_similarity_between_results(
 
     print(f'总计 {len(samples)} 个 Sample 被检出，{len(outputs)} 个待测输出 Variable')
     recorders = {name: MeasureRecorder(measurement=method) for name in outputs}
-    
+
     for sample_name in samples:
         for output in outputs:
 
             ref_file    = os.path.join(ref_dir, sample_name, output)
             target_file = os.path.join(target_dir, sample_name, output)
-            
+
             ref    = load_from_file(filename=ref_file, format='.dat')[:1000]
             target = load_from_file(filename=target_file, format='.dat')[:1000]
 
             recorders[output].update(
-                y_real=convert_any_to_torch_tensor(ref), 
+                y_real=convert_any_to_torch_tensor(ref),
                 y_pred=convert_any_to_torch_tensor(target))
 
     results = {}
     for output, recorder in recorders.items():
         results[output] = recorder.measure
-    
+
     method_str = 'MEASUREMENT'
     if method == 'snr': method_str = 'NOISE:SIGNAL POWER RATIO'
     if method == 'cosine': method_str = 'COSINE SIMILARITY'
@@ -206,7 +206,7 @@ def compare_cosine_similarity_between_results(
 
 
 def dump_internal_results(
-    graph: BaseGraph, dataloader: Iterable, 
+    graph: BaseGraph, dataloader: Iterable,
     dump_dir: str, executing_device: str, sample: bool = True):
 
     i_dir  = os.path.join(dump_dir, 'inputs')
@@ -218,12 +218,12 @@ def dump_internal_results(
     # 找出所有量化点，抽出所有中间结果.
     for var in graph.variables.values():
         if isinstance(var, QuantableVariable):
-            if (var.source_op_config is not None and 
+            if (var.source_op_config is not None and
                 var.source_op_config.state == QuantizationStates.ACTIVATED):
                 graph.outputs[var.name] = var # 直接标记为网络输出
 
     executor = TorchExecutor(graph, device=executing_device)
-    for batch_idx, batch in tqdm(enumerate(dataloader), 
+    for batch_idx, batch in tqdm(enumerate(dataloader),
                                  total=len(dataloader), desc='Dumping Results ...'):
         batch = batch.to(executing_device)
         outputs = executor.forward(batch)
@@ -234,11 +234,11 @@ def dump_internal_results(
             # 如果数字太多就抽样
             if output.numel() > 10000 and sample:
                 output = tensor_random_fetch(
-                    tensor=output, seed=10086, # 保证随机种子一致才能比对结果 
+                    tensor=output, seed=10086, # 保证随机种子一致才能比对结果
                     num_of_fetches=4096)
 
             dump_to_file(
-                filename=os.path.join(o_dir, str(batch_idx), name + '.dat'), 
+                filename=os.path.join(o_dir, str(batch_idx), name + '.dat'),
                 data=output, format='.dat')
 
         dump_to_file(
@@ -249,11 +249,11 @@ def split_result_to_directory(raw_dir: str, to_dir: str):
     print(f'正从 {raw_dir} 分割数据')
     data_files, sample_names = [], set()
     for file_name in os.listdir(raw_dir):
-        if file_name.endswith('.dat'): 
+        if file_name.endswith('.dat'):
             sample_name = file_name.split('_')[0]
             sample_names.add(sample_name)
             data_files.append((sample_name, file_name))
-        
+
     print(f'加载 {len(data_files)} 数据文件，总计 {len(sample_names)} 个样本')
     create_dir(to_dir)
     for sample_name in sample_names: