基于大数据技术的用电信息采集分析系统的设计与实现.docx

1、基于大数据技术的用电信息采集分析系统的设计与实现硕士专业学位论文基于大数据技术的用电信息采集分析系统的设计与实现Design andImplementation of Electro-Information Collection and Analysis System Based on Big-Data Technology作者: XXXX导师: XXXX北京交通大学2015年12月学位论文版权使用授权书本学位论文作者完全了解北京交通大学有关保留、使用学位论文的规定。特授权北京交通大学可以将学位论文的全部或部分内容编入有关数据库进行检索，提供阅览服务，并采用影印、缩印或扫描等复制手段保存、汇编

2、以供查阅和借阅。同意学校向国家有关部门或机构送交论文的复印件和磁盘。（保密的学位论文在解密后适用本授权说明）学位论文作者签名： 导师签名：签字日期： 年 月 日 签字日期： 年 月 日学校代码：10004 密级:公开 北京交通大学硕士专业学位论文基于大数据技术的用电信息采集分析系统的设计与实现Design andImplementation of Electro-Information Collection and Analysis System Based on Big-Data Technology作者姓名：XXX 学 号：XXXX导师姓名：XXXX 职 称：XXXX工程硕士专业领域：软件

3、工程 学位级别：硕士北京交通大学2021年4月致谢摘要大数据概念兴起于互联网行业，随后在能源行业引起关注，主要特点是数据规模量级高、类型多样化、增长速度高。大数据技术是实现大数据采集、存储、统计分析、展现等过程的新一代数据处理技术。用电信息采集系统，是在电力行业对用电客户的用电信息进行采集、处理和监控的系统，是为实现用电管理和智能用电而建立的辅助系统。本论文论述的是如何运用大数据技术实现新型用电信息采集分析系统过程，不包含智能采集终端等硬件设备的设计与应用，研究目的是实现将来自互联网行业的新兴技术应用于传统能源行业。本论文描述的系统全名为基于大数据技术的用电信息采集分析系统，本系统总共分为五个

4、模块，包括数据接入转发、实时计算、数据入库存储、数据统计分析和可视化展示。本人长期从事软件设计开发工作，负责其中数据接入转发模块、实时计算模块与数据入库存储三个模块的设计与实现：(1) 数据接入转发模块地市级计量自动化系统将计量数据和电能量信息以二进制的形式上传到省级中心系统，上传周期为15分钟，每个周期的数据容量大于10GB。本模块通过分布式处理技术Kafka，在一个上传周期内完成数据接入、存储并转发，同时本模块实现了数据出错重发机制。(2) 实时计算模块本模块使用分布式流计算引擎Storm，在数据接入转发模块转发信息的基础上，实现四分线损统计、供电质量统计、供售电量统计、供电时间统计业务逻

5、辑。本模块能够通过简单操作增加计算节点，扩充系统的处理能力。(3) 数据入库存储模块本模块将数据快速安全地存储于分布式数据库HBase与分布式文件系统HDFS(Hadoop Distributed File System)中，并快速读取。经测试和模拟数据验证，本系统在大数据技术架构基础上，完成用电信息采集系统五项功能。功能之一：通过分布式、数据持久化接入组件，本系统实现了海量用电客户用电数据实时可靠的接入。功能之二：通过分布式流式业务处理组件，本系统实现了阶段性电量计算、四分线损统计等实时计算功能。功能之三：基于HBase分布式数据库和HDFS分布式文件系统，本系统实现了分布式存储，数据根据实

6、际情况选择存储方式。功能之四，结合机器学习算法库与MapReduce分布式计算框架，本系统实现用电客户用电行为分析等离线分析挖掘。功能之五，本系统实现了数据处理过程完全可视化以及系统组件的实时监控。关键词：分布式；用电信息；采集；分析；大数据技术ABSTRACTBig data is a concept thats on the upgrade in the internet industry at the outset, and then caused concern in the energy industry, which main feature is the high-level s

7、cale of the data, the diversifications of the data type, the high growth rate. Big data technology is a new generation of data processing technology, realizing collection, storage, statistical analysis, display of big data and so on. Electro-information collection and analysis system is a system for

8、 collecting, processing and monitoring the electricity information thats from electricity customers, as an assistant system assisting for the power management and intelligent use of electricity. This paper discusses how use big data technology to achieve a new type of electro-information collection

9、and analysis system, does not include the design and application of hardware devices such as intelligent acquisition terminal, which purpose of the research is to apply the new technology in the traditional energy industry. The full name of this system is design andimplementation of electro-informat

10、ion collection and analysis system based on big-data technology, which is divided into five modules, including data access and forwarding, real-time calculation, data storage, data analysis and data visualization display. The author is engaged in the design and development of software for a long tim

11、e, so is responsible for the design and implementation of data access and forwarding module, real-time calculation module and data storage module of three modules:(1) Data access and forwarding moduleThe measurement automation system on the city level would upload measure data and energy information

12、 in binary form to the provincial center system. Data capacity produced in each upload cycle which is 15 minutes is greater than 10GB. In this module, through the distributed processing technology named Kafka, the system completes data access, storage and forwarding in an upload cycle. At the same t

13、ime, the module supply data retransmission function.(2) Real-time calculation moduleThis module uses distributed stream computing engine named Storm after data access forwarding module, according to the forwarding information to complete four loss statistics, statistics of power supply quality, for

14、electricity sales statistics, power supply time of statistical calculation. This module can be used to increase the processing capacity of the system, which can be used to increase the computing nodes.(3) Data storage moduleThis module puts data stored quickly in HBase that is a distributed database

15、 and HDFS (Hadoop Distributed File System) that is a distributed file system, and gets data from them quickly.Proved by test and verification with simulation data, this system based on the big data technology structure, completes five functions of electro-information collection and analysis system.T

16、he first function: Through a distributed, data persistent access component, this system archives the real-time and reliable access from massive user electricity data.The second function: Via a distributed flow type business processing component, this system achieves stage power calculation, four-ins line loss statistics real-time calculation func