Generic placeholder image

International Journal of Sensors, Wireless Communications and Control

Author(s): Mohammad Khalid Pandit*, Roohie Naaz Mir and Mohammad Ahsan Chishti

DOI: 10.2174/2210327910666191223124630

DownloadDownload PDF Flyer Cite As
Adaptive Deep Neural Networks for the Internet of Things

Page: [570 - 581] Pages: 12

  • * (Excluding Mailing and Handling)

Abstract

Background: Deep neural networks have become the state of the art technology for real- world classification tasks due to their ability to learn better feature representations at each layer. However, the added accuracy that is associated with the deeper layers comes at a huge cost of computation, energy and added latency.

Objective: The implementations of such architectures in resource constraint IoT devices are computationally prohibitive due to its computational and memory requirements. These factors are particularly severe in IoT domain. In this paper, we propose the Adaptive Deep Neural Network (ADNN) which gets split across the compute hierarchical layers i.e. edge, fog and cloud with all splits having one or more exit locations.

Methods: At every location, the data sample adaptively chooses to exit from the NN (based on confidence criteria) or get fed into deeper layers housed across different compute layers. Design of ADNN, an adaptive deep neural network which results in fast and energy- efficient decision making (inference).

Joint optimization of all the exit points in ADNN such that the overall loss is minimized.

Results: Experiments on MNIST dataset show that 41.9% of samples exit at the edge location (correctly classified) and 49.7% of samples exit at fog layer. Similar results are obtained on fashion MNIST dataset with only 19.4% of the samples requiring the entire neural network layers. With this architecture, most of the data samples are locally processed and classified while maintaining the classification accuracy and also keeping in check the communication, energy and latency requirements for time sensitive IoT applications.

Conclusion: We investigated the approach of distributing the layers of the deep neural network across edge, fog and the cloud computing devices wherein data samples adaptively choose the exit points to classify themselves based on the confidence criteria (threshold). The results show that the majority of the data samples are classified within the private network of the user (edge, fog) while only a few samples require the entire layers of ADNN for classification.

Keywords: Deep neural networks, fog computing, Internet of things, ADNN, MNIST, fog layer.

Graphical Abstract

[1]
Weiser M. The computer for the 21st century. Sci Am 1991; 265(3): 94-105.
[http://dx.doi.org/10.1038/scientificamerican0991-94 PMID: 1675486]
[2]
Ibm annual report, 2013. Available from: . https://www. ibm.com/annualreport/2013
[3]
Available from. https://spectrum.ieee.org/tech-talk/telecom/inter net/popular-internet-of-
[4]
Pandit MK, Mir RN, Chihisti MA. Machine learning at the edge of internet of things. CSI Commun 2017; 41(8): 28-30.
[5]
Li H, Ota K, Dong M. Learning IoT in edge: Deep learning for the internet of things with edge computing. IEEE Netw 2018; 32(1): 96-101.
[http://dx.doi.org/10.1109/MNET.2018.1700202]
[6]
Verma S, Kawamoto Y, Fadlullah ZM, Nishiyama H, Kato N. A survey on network methodologies for real time analytics of massive IoT data and open research issues. IEEE Comm Surv Tutor 2017; 19(3): 1457-77.
[http://dx.doi.org/10.1109/COMST.2017.2694469]
[7]
Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. Inception-v4, inception-resnet and the impact of residual connections on learning AAAI 2017; 4: 12..
[8]
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA. 2016.
[9]
Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, 2015..
[10]
Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 2012; 2012: 1097-105.
[11]
Courbariaux M, Hubara I, Soudry D, El-Yaniv R, Bengio Y. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:160202830, 2016..
[12]
Courbariaux M, Bengio Y, David JP. Binaryconnect: Training deep neural networks with binary weights during propagations. Adv Neural Inf Process Syst 2015; 2015: 3123-31.
[13]
De Coninck E, Verbelen T, Vankeirsbilck B, Bohez S, Leroux S, Simoens P. Dianne: Distributed artificial neural networks for the internet of things. Proceedings of the 2nd Workshop on Middleware for Context-Aware Applications in the IoT, 2015..
[14]
Verbelen T, Simoens P, De Turck F, Dhoedt B. Aiolos: Middleware for improving mobile application performance through cyber foraging. J Syst Softw 2012; 85(11): 2629-39.
[http://dx.doi.org/10.1016/j.jss.2012.06.011]
[15]
Teerapittayanon S, McDanel B, Kung H. Distributed deep neural networks over the cloud, the edge and end devices. 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS). Atlanta, GA, USA, 2017.
[http://dx.doi.org/10.1109/ICDCS.2017.226]
[16]
Leroux S, Bohez S, Verbelen T, Vankeirsbilck B, Simoens P, Dhoedt B. Resource-constrained classification using a cascade of neural network layers. 2015 International Joint Conference on Neural Networks (IJCNN). Killarney, Ireland. 2015.
[http://dx.doi.org/10.1109/IJCNN.2015.7280601]
[17]
Teerapittayanon S, McDanel B, Kung H. Branchynet: Fast inference via early exiting from deep neural networks. 2016 23rd International Conference on Pattern Recognition (ICPR). Cancun, Mexico, 2016..
[18]
De Coninck E, Verbelen T, Vankeirsbilck B, et al. Distributed neural networks for internet of things: The big-little approach. Int Internet Things 2015; 170: 484-92.
[19]
Krizhevsky A. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:14045997. 2014.
[20]
Dean J, Corrado G, Monga R, et al. Large scale distributed deep networks. Adv Neural Inf Process Syst 2012; 2012: 1223-31.
[21]
De Coninck E, Verbelen T, Vankeirsbilck B, et al. Distributed neural networks for internet of things: The big-little approach. Int Internet Things 2015; 170: 484-92.
[22]
Leroux S, Bohez S, Verbelen T, Vankeirsbilck B, Simoens P, Dhoedt B. Resource-constrained classification using a cascade of neural network layers. 2015 International Joint Conference on Neural Networks (IJCNN). Killarney, Ireland. 2015.
[23]
Deng L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Process Mag 2012; 29(6): 141-2.
[http://dx.doi.org/10.1109/MSP.2012.2211477]
[24]
Xiao H, Rasul K, Vollgraf R. Fashion-Mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:170807747,. 2017.
[25]
Krizhevsky A, Hinton G. “Learning multiple layers of features from tiny images,” tech rep. Citeseer 2009.
[26]
LeCun Y, Bottou L, Bengio Y, Haffner P. Gradientbased learning applied to document recognition. Proc IEEE 1998; 86(11): 2278-324.
[http://dx.doi.org/10.1109/5.726791]
[27]
Brooks D. David brooks: Taking machine learning to edge devices. 2016.
[28]
Satyanarayanan M, Bahl P, Caceres R, Davies N. The case for VM-based cloudlets in mobile computing. IEEE Pervasive Comput 2009; 8(4): 14-23.
[http://dx.doi.org/10.1109/MPRV.2009.82]
[29]
Bonomi F, Milito R, Zhu J, Addepalli S. Fog computing and its role in the internet of things. Proceedings of the first edition of the MCC workshop on Mobile cloud computing, 2012.