Outlier Detection Based on MapReduce for Analyzing Big Data

Yejin Hong; Eunhee Na; Yonghwan Jung; Yangwoo Kim

Outlier Detection Based on MapReduce for Analyzing Big Data

Yejin Hong, Eunhee Na, Yonghwan Jung, Yangwoo Kim, Journal of Internet Computing and Services, Vol. 18, No. 1, pp. 27-36, Feb. 2017

10.7472/jksii.2017.18.1.27, Full Text:

Keywords: Big data, Outlier, MapReduce, Distributed Processing, Spark

Abstract

In near future, IoT data is expected to be a major portion of Big Data. Moreover, sensor data is expected to be major portion of IoT data, and its' research is actively carried out currently. However, processed results may not be trusted and used if outlier data is included in the processing of sensor data. Therefore, method for detection and deletion of those outlier data before processing is studied in this paper. Moreover, we used Spark which is memory based distributed processing environment for fast processing of big sensor data. The detection and deletion of outlier data consist of four stages, and each stage is implemented with Mapper and Reducer operation. The proposed method is compared in three different processing environments, and it is expected that the outlier detection and deletion performance is best in the distributed Spark environment as data volume is increasing.

Statistics

Show / Hide Statistics

Statistics (Cumulative Counts from November 1st, 2017)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.

Cite this article

[APA Style]

Hong, Y., Na, E., Jung, Y., & Kim, Y. (2017). Outlier Detection Based on MapReduce for Analyzing Big Data. Journal of Internet Computing and Services, 18(1), 27-36. DOI: 10.7472/jksii.2017.18.1.27.

[IEEE Style]

Y. Hong, E. Na, Y. Jung, Y. Kim, "Outlier Detection Based on MapReduce for Analyzing Big Data," Journal of Internet Computing and Services, vol. 18, no. 1, pp. 27-36, 2017. DOI: 10.7472/jksii.2017.18.1.27.

[ACM Style]

Yejin Hong, Eunhee Na, Yonghwan Jung, and Yangwoo Kim. 2017. Outlier Detection Based on MapReduce for Analyzing Big Data. Journal of Internet Computing and Services, 18, 1, (2017), 27-36. DOI: 10.7472/jksii.2017.18.1.27.