科研PPT模板Template1

ThiNet A Filter Level Pruning Method for Deep Neural Network Compression Contents Background 1 Motivation 2 n Filter Selection n Greedy Method n Minimize Reconstruction Error Proposed Method 3 Experimental Results 4 Conclusion 5 2 Contents Background 1 Motivation 2 n Filter Selection n Greedy Method n Minimize Reconstruction Error Proposed Method 3 Experimental Results 4 Conclusion 5 3 Background Deep Neural Network DNN is hard to deployed on hardware with the limitation of computation resources storage battery power Figure Performance and model size of different models on ImageNet 4 Model Compression Existing Compression Methods n Quantization convert full precision weights to low precision version e g INQ BWN TWN n Pruning remove less important weights filters from the model e g Deep Compression DNS ThiNet n Design new structure SqueezeNet Distilling ShuffleNet 5 Pruning Methods n Non structured Pruning remove less important weights n Structured Pruning remove less important filters from model 6 Contents Background 1 Motivation 2 n Filter Selection n Greedy Method n Minimize Reconstruction Error Proposed Method 3 Experimental Results 4 Conclusion 5 7 Motivation Problems of Non structured Pruning n Need specialized hardware and software for inference n Ignore cache and memory issues which leads to limited practical acceleration Benefits of Structured Pruning n No change of network structure and can supported by existing deep learning libraries n Reduce the memory and accelerate inference 8 Contents Background 1 Motivation 2 n Filter Selection n Greedy Method n Minimize Reconstruction Error Proposed Method 3 Experimental Results 4 Conclusion 5 9 Proposed Method ThiNet Thin Net a filter level pruning compression framework for model compression Figure Illustration of ThiNet Filter Selection Pruning Fine tuning 10 Proposed Method Framework of ThiNet 11 Filter Selection n Convolution operation can be computed as follows 1 12 Filter Selection n Define n Then 2 3 13 Greedy Method 4 5 14 Greedy Method n Use greedy method to solve the optimization problem A greedy algorithm for minimizing Eq 5 15 Minimize the Reconstruction Error n Minimize the reconstruction error by weighting the channels n Eq 6 can be solved by the ordinary least squares approach 6 7 16 Contents Background 1 Motivation 2 n Filter Selection n Greedy Method n Minimize Reconstruction Error Proposed Method 3 Experimental Results 4 Conclusion 5 17 Pruning Strategy n VGG 16 prune the first 10 convolutional layers and replace the FC layers with a global average pooling layer n ResNet 50 prune the first two convolutional layers 256 d ReLU ReLU 256 dReLU 256 d ReLU ReLU 256 dReLU 18 Comparison of Existing Methods Figure Comparison of different channel selection methods using VGG 16 GAP on CUB 200 19 VGG 16 on ImageNet ModelTop 1Top 5 Param FLOP s f b ms Original68 34 88 44 138 34M30 94B189 92 407 56 ThiNet Conv69 80 89 53 131 44M9 58B76 71 152 05 Train from scratch67 00 87 45 131 44M9 58B76 71 152 05 ThiNet GAP67 34 87 92 8 32M9 34B71 73 145 51 ThiNet Tiny59 34 81 97 1 32M2 01B29 51 55 83 SqueezeNet Han et al 57 67 80 39 1 24M1 72B37 30 68 62 Table Pruning results of VGG 16 on ImageNet n ThiNet Conv prune 50 of the first 10 convolutional layers n ThiNet GAP replace the FC layers with a global average pooling GAP layer based on ThiNet Conv 20 VGG 16 on ImageNet MethodTop 1Top 5 Param FLOPs APoZ 1 Hu et al 2 16 0 84 APoZ 2 Hu et al 1 81 1 25 Taylor 1 Molchanov et al 1 44 Taylor 2 Molchanov et al 3 94 ThiNet WS Li et al 1 01 0 69 ThiNet Conv 1 46 1 09 ThiNet GAP 1 00 0 52 Table Comparison of state of the art methods on VGG 16 n ThiNet WS use the weight sum WS method for pruning 21 ResNet 50 on ImageNet ModelTop 1Top 5 Param FLOPsf b ms Original72 88 91 14 25 56M7 72B188 27 269 32 ThiNet 7072 04 90 67 16 94M4 88B169 38 243 37 ThiNet 5071 01 90 02 12 38M3 41B153 60 212 29 ThiNet 3068 42 88 30 8 66M2 20B144 45 200 67 Table Performance of pruning ResNet 50 on ImageNet 22 Domain Adaptation Ability Table Comparison of different methods on CUB 200 and Indoor 67 FT denotes Fine Tune 23 Data setStategy Param FLOPsTop 1 CUB 200VGG 16135 07M30 93B72 30 FT prune7 91M9 34B66 90 Train from scratch7 91M9 34B44 27 ThiNet Conv128 16M9 58B70 90 ThiNet GAP7 91M9 34B69 43 ThiNet Tiny1 12M2 01B65 45 AlexNet57 68M1 44B57 28 Indoor 67VGG 16134 52M30 93B72 46 FT prune7 84M9 34B64 70 Train from scratch7 84M9 34B38 81 ThiNet Conv127 62M9 58B72 31 ThiNet GAP7 84M9 34B70 22 ThiNet Tiny1 08M2 01B62 84 AlexNet57 68M1 44B59 55 Contents Background 1 Motivation 2 n Filter Selection n Greedy Method n Minimize Reconstruction Error Proposed Method 3 Experimental Results 4 Conclusion 5 24 Conclusion Contributions n Proposed ThiNet a filter pruning framework to accelerate and compress CNN models n Formally establish filter pruning as an optimization problem n VGG 16 model can be pruned into 5 05MB and shows promising generalization ability on transfer learning Future work n Prune the projection